docs: add integration guides and incident runbooks (#20)
This commit is contained in:
+193
-68
@@ -1,97 +1,222 @@
|
||||
# Diagnostic TestClient integration guide
|
||||
# Start-to-finish TestClient guide
|
||||
|
||||
Tracking: #25
|
||||
Tracking: #20, #25
|
||||
|
||||
`FinalFactory.Rendezvous.TestClient` is the smallest supported public-SDK consumer.
|
||||
It exists for integration development, CI smoke checks, deployment verification,
|
||||
and operator diagnosis. It is intentionally not a production game client, game
|
||||
server, matchmaking UI, or relay.
|
||||
`FinalFactory.Rendezvous.TestClient` is the supported executable proof that a
|
||||
consumer can publish, browse, authorize, punch, connect, exchange direct traffic,
|
||||
and diagnose a failure using only the public Client and Contracts packages. It is
|
||||
intentionally thin: a polished server browser and player-facing connection UI
|
||||
belong in each game repository.
|
||||
|
||||
The automated scenario matrix, privileged Linux namespace run, and topology
|
||||
limitations are documented in the [deterministic topology harness](topology-harness.md).
|
||||
> **Traversal boundary:** Rendezvous v1 is not a relay and cannot guarantee a
|
||||
> connection through symmetric NAT, carrier-grade NAT, restrictive firewalls,
|
||||
> VPNs, or platform policy. It provides no accounts, social system, skill-based
|
||||
> matchmaking, gameplay server, gameplay authority, or gameplay transport.
|
||||
|
||||
## Prerequisites
|
||||
The automated scenario matrix, privileged Linux namespace run, and simulation
|
||||
limits are in the [deterministic topology harness](topology-harness.md). The
|
||||
[SDK seam guide](sdk-seams.md) covers the few integration details that this
|
||||
executable cannot show.
|
||||
|
||||
Start a configured Rendezvous service and note both its HTTP base URL and UDP
|
||||
mediator endpoint. The host needs a tenant-scoped publisher credential from the
|
||||
deployment secret boundary. Put it in an environment variable and pass only that
|
||||
variable's name when the default is unsuitable:
|
||||
## First local connection from a clean checkout
|
||||
|
||||
Prerequisites are the pinned .NET SDK, Docker with Compose, OpenSSL, Python 3,
|
||||
`curl`, and `jq`. Run these commands from the repository root. The generated key
|
||||
and credential are disposable local fixtures, not production provisioning.
|
||||
|
||||
```bash
|
||||
export RENDEZVOUS_PUBLISHER_CREDENTIAL='<deployment-supplied value>'
|
||||
install -d -m 0700 deploy/compose/secrets
|
||||
umask 077
|
||||
openssl rand -out deploy/compose/secrets/signing-key 32
|
||||
export RENDEZVOUS_UID="$(id -u)"
|
||||
export RENDEZVOUS_GID="$(id -g)"
|
||||
test "$RENDEZVOUS_UID" -ne 0
|
||||
docker compose -f deploy/compose/compose.yaml up --build --detach
|
||||
ready=false
|
||||
for attempt in {1..45}; do
|
||||
if curl --fail --silent http://127.0.0.1:8080/health/ready >/dev/null; then
|
||||
ready=true
|
||||
break
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
test "$ready" = true
|
||||
curl --fail http://127.0.0.1:8080/health/live
|
||||
curl --fail http://127.0.0.1:8080/health/ready
|
||||
dotnet build src/FinalFactory.Rendezvous.TestClient --configuration Release
|
||||
```
|
||||
|
||||
Never put the credential in a command argument, URL, checked-in configuration,
|
||||
shell trace, or captured test fixture. The development server's signing material
|
||||
is process-ephemeral; credentials from a prior development process are invalid.
|
||||
|
||||
## Manual three-terminal flow
|
||||
|
||||
Start the host:
|
||||
Only the host terminal needs a publisher credential. Disable shell tracing before
|
||||
capturing it; the helper prints the credential on stdout so command substitution
|
||||
can place it directly in the environment without writing it to disk.
|
||||
|
||||
```bash
|
||||
dotnet run --project src/FinalFactory.Rendezvous.TestClient -- \
|
||||
host --service http://127.0.0.1:5000/ --mediator 127.0.0.1:9050 \
|
||||
--game space-game --environment development --region local --protocol 1
|
||||
set +x
|
||||
export RENDEZVOUS_PUBLISHER_CREDENTIAL="$(./scripts/mint-local-publisher-credential.sh)"
|
||||
```
|
||||
|
||||
Browse from another terminal:
|
||||
The helper accepts no arguments, reads the ignored `0600` local Compose key, and
|
||||
mints only `space-game` / `smoke` / `local` / protocol `1` for ten minutes. It is
|
||||
not a reusable issuer or an example for production. Never put the result in a
|
||||
command argument, URL, shell history, log, screenshot, support ticket, captured
|
||||
fixture, or source file.
|
||||
|
||||
In terminal 1, publish a host. It stays alive for at most 60 seconds and exits
|
||||
after a joining peer completes the authenticated echo exchange:
|
||||
|
||||
```bash
|
||||
dotnet run --project src/FinalFactory.Rendezvous.TestClient -- \
|
||||
browse --service http://127.0.0.1:5000/ \
|
||||
--game space-game --environment development --region local --protocol 1
|
||||
dotnet run --project src/FinalFactory.Rendezvous.TestClient \
|
||||
--configuration Release --no-build -- \
|
||||
host --service http://127.0.0.1:8080/ --mediator 127.0.0.1:9050 \
|
||||
--game space-game --environment smoke --region local --protocol 1 \
|
||||
--display-name "Local diagnostic" --timeout-seconds 60 --run-seconds 60 \
|
||||
--exit-after-echo
|
||||
```
|
||||
|
||||
Join from a third terminal. Omit `--listing` for an interactive choice:
|
||||
Copy the public listing ID printed by the host, or discover it from terminal 2:
|
||||
|
||||
```bash
|
||||
dotnet run --project src/FinalFactory.Rendezvous.TestClient -- \
|
||||
join --service http://127.0.0.1:5000/ --mediator 127.0.0.1:9050 \
|
||||
--game space-game --environment development --region local --protocol 1 \
|
||||
--listing 00000000-0000-0000-0000-000000000000
|
||||
dotnet run --project src/FinalFactory.Rendezvous.TestClient \
|
||||
--configuration Release --no-build -- \
|
||||
browse --service http://127.0.0.1:8080/ \
|
||||
--game space-game --environment smoke --region local --protocol 1
|
||||
```
|
||||
|
||||
Replace the sample UUID with the public listing ID printed by host or browse.
|
||||
Host and join each create one caller-owned LiteNetLib manager. That same socket
|
||||
sends presence/punch traffic, establishes the authenticated direct connection,
|
||||
and carries the ping/echo/ack/completion payload. The final completion confirms
|
||||
that the host received the reliable acknowledgement; none of this traffic passes through the HTTP
|
||||
service or UDP mediator.
|
||||
In terminal 3, either omit `--listing` and select interactively, or provide the
|
||||
copied ID for deterministic selection:
|
||||
|
||||
## CI and deployment smoke flow
|
||||
```bash
|
||||
dotnet run --project src/FinalFactory.Rendezvous.TestClient \
|
||||
--configuration Release --no-build -- \
|
||||
join --service http://127.0.0.1:8080/ --mediator 127.0.0.1:9050 \
|
||||
--game space-game --environment smoke --region local --protocol 1 \
|
||||
--listing REPLACE_WITH_LISTING_UUID --timeout-seconds 30
|
||||
```
|
||||
|
||||
Use `--script --json`, set `--listing` when deterministic selection matters, and
|
||||
check the documented process exit code. `--timeout-seconds` bounds each startup,
|
||||
traversal, or direct-traffic stage; a script host also uses it as its total runtime
|
||||
unless `--run-seconds` is explicit. A host can add `--exit-after-echo` so it
|
||||
terminates after the joining peer acknowledges direct traffic and receives the
|
||||
host's completion confirmation. Every wait is
|
||||
bounded by coordinator state and `--timeout-seconds`; no orchestration should use
|
||||
an unbounded sleep.
|
||||
Success means the joiner prints `join.connected` and verified direct traffic,
|
||||
and the host prints verified direct traffic before deregistering. The host and
|
||||
joiner each create one caller-owned LiteNetLib manager. The same UDP socket sends
|
||||
presence and punch traffic, accepts the authenticated peer, and carries the
|
||||
ping/echo/ack/completion payload; direct traffic does not pass through the HTTP
|
||||
service or mediator.
|
||||
|
||||
The normal test suite contains a real process gate that starts the built Server,
|
||||
host TestClient, and join TestClient, waits for readiness and versioned events,
|
||||
and verifies direct traffic, cleanup, JSON shape, and secret canaries. Process
|
||||
trees are force-terminated in the test cleanup path if normal shutdown fails.
|
||||
Clean up secrets and the disposable service when finished:
|
||||
|
||||
Useful success events are:
|
||||
```bash
|
||||
unset RENDEZVOUS_PUBLISHER_CREDENTIAL
|
||||
docker compose -f deploy/compose/compose.yaml down
|
||||
rm deploy/compose/secrets/signing-key
|
||||
```
|
||||
|
||||
- `host.registered`, `host.ready`, `host.direct-traffic`, and `host.deregistered`;
|
||||
- `browse.completed` and `browse.session`; and
|
||||
- `join.connected`, `join.direct-traffic`, and `join.outcome-report`.
|
||||
## Script and JSON automation
|
||||
|
||||
Failure events preserve stable typed phases and outcomes. When a terminal outcome
|
||||
contains a configured dedicated endpoint, `join.fallback` reports `available`
|
||||
with endpoint type `dedicated`; no raw address is printed and no fallback is
|
||||
started implicitly.
|
||||
`--script` forbids prompts and selects the first compatible listing unless
|
||||
`--listing UUID` fixes the choice. `--json` emits one JSON object per line with
|
||||
`version: 1`. New optional properties may be added, but event names and exit
|
||||
codes are stable automation contracts. Informational events use stdout and
|
||||
failures use stderr.
|
||||
|
||||
## What the proof does and does not establish
|
||||
The deployment smoke performs the full health, publish, join, mediation, direct
|
||||
traffic, outcome-report, and cleanup flow using bounded waits:
|
||||
|
||||
The deterministic loopback test proves the complete service/host/client protocol,
|
||||
ticket admission, and peer-to-peer payload path. Loopback is not evidence that all
|
||||
consumer routers, carrier-grade NATs, symmetric NATs, firewalls, VPNs, IPv6 paths,
|
||||
or platform policies permit hole punching. Same-LAN, separated observed endpoints,
|
||||
network namespaces/containers, mediator restart, and adverse topology coverage
|
||||
belong to the topology harness tracked by #14. Production rollout still requires
|
||||
tests from representative networks and a game-owned fallback policy.
|
||||
```bash
|
||||
dotnet build src/FinalFactory.Rendezvous.TestClient --configuration Release
|
||||
./scripts/smoke-deployment.sh
|
||||
```
|
||||
|
||||
For custom automation, capture JSON and preserve the process status separately:
|
||||
|
||||
```bash
|
||||
set +e
|
||||
dotnet run --project src/FinalFactory.Rendezvous.TestClient \
|
||||
--configuration Release --no-build -- \
|
||||
browse --service http://127.0.0.1:8080/ \
|
||||
--game space-game --environment smoke --region local --protocol 1 \
|
||||
--script --json >browse.jsonl
|
||||
status=$?
|
||||
set -e
|
||||
jq -e 'select(.version == 1 and .event == "browse.completed")' browse.jsonl
|
||||
test "$status" -eq 0
|
||||
```
|
||||
|
||||
Never use an unbounded sleep to orchestrate processes. Wait for versioned events
|
||||
such as `host.ready` and apply a deadline. Useful success events are
|
||||
`host.registered`, `host.ready`, `host.direct-traffic`, `host.deregistered`,
|
||||
`browse.completed`, `browse.session`, `join.connected`, `join.direct-traffic`,
|
||||
and `join.outcome-report`.
|
||||
|
||||
| Exit | Meaning |
|
||||
| ---: | --- |
|
||||
| `0` | Requested diagnostic flow completed successfully |
|
||||
| `2` | Invalid command or options |
|
||||
| `3` | Missing or invalid local configuration |
|
||||
| `10` | HTTP, registration, browser, lease, or socket failure |
|
||||
| `11` | No compatible session was available or selected |
|
||||
| `12` | Authorization or traversal reached a typed terminal failure |
|
||||
| `13` | Direct connection succeeded but the direct traffic proof failed |
|
||||
| `130` | Caller cancellation or Ctrl+C |
|
||||
|
||||
## Observe a safe failure
|
||||
|
||||
Run this after the protocol-1 browse in terminal 2 and before the terminal-3
|
||||
join (or restart terminal 1 first). The preceding browse proves that one
|
||||
protocol-1 host is present. Now browse for deliberately incompatible protocol
|
||||
`999`. The command emits a successful directory response with
|
||||
`browse.completed`, `count: 0`, then exits `11` to distinguish compatibility
|
||||
from a service outage:
|
||||
|
||||
```bash
|
||||
set +e
|
||||
dotnet run --project src/FinalFactory.Rendezvous.TestClient \
|
||||
--configuration Release --no-build -- \
|
||||
browse --service http://127.0.0.1:8080/ \
|
||||
--game space-game --environment smoke --region local --protocol 999 \
|
||||
--script --json >incompatible.jsonl
|
||||
status=$?
|
||||
set -e
|
||||
jq -e 'select(.event == "browse.completed" and .phase == "directory" and .count == 0)' \
|
||||
incompatible.jsonl
|
||||
test "$status" -eq 11
|
||||
```
|
||||
|
||||
This is a diagnostic failure drill, not a bypass: unknown tenant scope and
|
||||
protocols still fail closed, and the local helper cannot mint a credential for
|
||||
them.
|
||||
|
||||
## Diagnose by phase, not by guesswork
|
||||
|
||||
Start with the exit code, then the last versioned event and its `phase`, `status`,
|
||||
and typed `outcome`. Endpoint categories may be
|
||||
reported as `loopback`, `private`, or `public`; raw endpoints, credentials,
|
||||
capabilities, metadata, and player identities are never emitted.
|
||||
|
||||
| Symptom or last event | Distinction | Check next |
|
||||
| --- | --- | --- |
|
||||
| `host.configuration`, exit `3` | Local credential variable is missing or malformed before any request | Confirm the named environment variable exists, tracing is off, and the credential has not expired |
|
||||
| `host.registration`, exit `10` | Publisher authentication, tenant policy, metadata, quota, or HTTP failure | Use the typed status; compare credential scope with game/environment/region and the provisioned policy, then correlate protected server telemetry by operation and time |
|
||||
| `browse.sessions`, exit `10` | Directory request failed | Check HTTP reachability, `/health/ready`, rate limiting, and contract compatibility |
|
||||
| `browse.completed` count `0`, or `join.selection` empty, exit `11` | Healthy directory but no compatible visible listing | Match game, environment, region, and exact gameplay protocol; then confirm a host lease is still active |
|
||||
| Exact `join.selection` failure, exit `10` | Listing disappeared, is hidden, or scope no longer matches | Browse again; do not retry an old listing ID forever |
|
||||
| `join.authorization`, exit `12` | Service rejected the attempt before NAT traversal | Inspect typed category/outcome for policy, capacity, stale host, or active-attempt limits |
|
||||
| `join.punch` / `join.traversal`, exit `12` | Mediation or NAT traversal did not establish a peer | Confirm UDP endpoint/reply path, host presence, clocks, firewall/NAT behavior, and topology; use a game-owned fallback if policy supplies one |
|
||||
| `join.direct-connect`, exit `12` | Introduction occurred but authenticated direct admission failed | Confirm host is polling the same socket, the one-time ticket is current, and game admission did not reject capacity, identity, or bans |
|
||||
| `join.connected` followed by exit `13` | Peer connected but the direct gameplay-like echo did not finish | Inspect the peer lifecycle and caller polling; this is not an HTTP/directory failure |
|
||||
|
||||
Stopping a host without deregistration may leave its listing visible only until
|
||||
the bounded lease expires. During that window, a join can produce a typed stale
|
||||
host or traversal outcome; it must not be interpreted as a healthy host. Restarting
|
||||
the single-active service intentionally loses all ephemeral listings and attempts,
|
||||
so hosts re-register and clients browse again.
|
||||
|
||||
If a terminal outcome reports an authoritative dedicated fallback,
|
||||
`join.fallback` exposes only availability and endpoint type. TestClient never
|
||||
connects to it automatically. The game owns the decision, authentication, and
|
||||
connection policy. If no fallback is present, Rendezvous v1 offers no relay.
|
||||
|
||||
## Production use
|
||||
|
||||
Do not copy a production signing key to a diagnostic host. Supply a short-lived,
|
||||
least-scope publisher credential from the deployment secret boundary and set the
|
||||
external service, mediator, and matching scope variables described in the
|
||||
[secure Linux deployment smoke](../deployment/linux.md#http-and-udp-smoke).
|
||||
Run representative external-network tests; loopback success is not NAT coverage.
|
||||
|
||||
Reference in New Issue
Block a user