docs: add integration guides and incident runbooks (#20)

2026-07-16 18:25:10 +02:00
parent cc5793f935
commit 7fb85059fb
11 changed files with 1082 additions and 104 deletions
@@ -100,6 +100,8 @@ defined in [hostile-input and overload protection](docs/security/abuse-protectio
 Health semantics, bounded telemetry, alerting, audit privacy, and the authenticated
 operator controls are defined in the
 [observability and operator runbook](docs/operations/observability-and-operator-runbook.md).
 Concrete detect/contain/recover/verify procedures are in the
 [incident and change runbooks](docs/operations/incident-runbooks.md).
 The pinned non-root container, production topology, graceful drain, Linux
 hardening, smoke procedure, and recovery lifecycle are documented in
 [secure single-active Linux deployment](docs/deployment/linux.md).
@@ -111,6 +113,9 @@ signing, staged promotion, rollback, and migration are defined in
 [releases and compatibility](docs/releases/README.md).
 The scriptable host/browser/join diagnostic and its stable automation contract are
 documented in the [TestClient integration guide](docs/integration/test-client.md).
 The package, gameplay-socket, host-admission, provisioning, metadata, key rotation,
 versioning, and secure rollout seams are in the
 [game integration guide](docs/integration/sdk-seams.md).
 The always-on three-party scenarios, optional Linux namespace topology, and
 simulation limits are documented in the
 [deterministic topology harness](docs/integration/topology-harness.md).
@@ -177,6 +177,9 @@ dotnet build src/FinalFactory.Rendezvous.TestClient --configuration Release
 For the local Compose profile, the script derives a ten-minute diagnostic
 publisher credential from the ignored local key without printing either secret.
 The fixed-scope helper used by the smoke can also support the manual
 [TestClient local flow](../integration/test-client.md); it is deliberately not a
 production issuer.
 For production, do not copy the signing key to the smoke host. Instead inject a
 short-lived, region-scoped credential through
 `RENDEZVOUS_PUBLISHER_CREDENTIAL`, and set the external endpoints:
@@ -0,0 +1,227 @@
 # Game integration seams
 Tracking: #20
 Use the [TestClient start-to-finish guide](test-client.md) before integrating a
 game. It proves the service and network path without engine or game code. This
 page documents only the seams that the diagnostic cannot choose for a game:
 package/version policy, ownership of the gameplay socket, host admission,
 metadata, credential custody, and deployment compatibility.
 ## Packages and compatibility
 Consume `FinalFactory.Rendezvous.Client` and
 `FinalFactory.Rendezvous.Contracts` from the approved Gitea NuGet source and pin
 both to the same exact released version. Do not use a floating version range.
 The current release matrix is machine-readable in
 [`compatibility.json`](../releases/compatibility.json); the same window is
 available from authenticated `GET /v1/operator/status`.
 The authoritative package feed is
 `https://git.finalfactory.de/api/packages/HeiKyu/nuget/index.json`. Add it to the
 consumer's `NuGet.config` and map only Rendezvous packages to it; retain the
 consumer's existing NuGet.org mapping for other dependencies:
 ```xml
 <packageSources>
  <add key="FinalFactory" value="https://git.finalfactory.de/api/packages/HeiKyu/nuget/index.json" />
 </packageSources>
 <packageSourceMapping>
  <packageSource key="FinalFactory">
    <package pattern="FinalFactory.Rendezvous.*" />
  </packageSource>
  <packageSource key="nuget.org">
    <package pattern="*" />
  </packageSource>
 </packageSourceMapping>
 ```
 When the feed is anonymously readable, no reader credential is needed. If
 registry policy requires authentication, use the platform's NuGet credential
 provider or a protected per-user/CI NuGet configuration populated by the secret
 manager. Never put a registry token in the project file, repository,
 package-source URL, or `dotnet` command argument.
 ```xml
 <ItemGroup>
  <PackageReference Include="FinalFactory.Rendezvous.Client" Version="1.0.0" />
  <PackageReference Include="FinalFactory.Rendezvous.Contracts" Version="1.0.0" />
 </ItemGroup>
 ```
 Run `dotnet restore`, then `dotnet list package --include-transitive` and verify
 that Client and Contracts resolve to the same exact version and LiteNetLib to the
 release matrix version before compiling the game.
 Release 1.0.0 targets `netstandard2.1`, requires LiteNetLib `2.1.4`, speaks HTTP,
 UDP, and connection-ticket contract version `1`, and requires an exact
 tenant-configured gameplay protocol match. A package patch does not silently
 change a wire version. Follow the [release and migration policy](../releases/README.md)
 when changing any dimension, and validate the generated
 [OpenAPI v1 document](../api/rendezvous-v1.json) rather than hand-building HTTP.
 ## One caller-owned gameplay socket
 Create the game's LiteNetLib manager through `RendezvousNetListener`; do not open
 a separate NAT socket. The game owns start, stop, and disposal. A coordinator
 owns polling while it is active, so call its `Poll()` once from the game/network
 thread and do not also call `NetManager.PollEvents()` during that period.
 ```csharp
 RendezvousNetListener networkEvents = new();
 NetManager gameplayNetwork = networkEvents.CreateManager();
 if (!gameplayNetwork.Start(gameplayPort))
 {
    throw new InvalidOperationException("Gameplay UDP socket could not start.");
 }
 using RendezvousHostCoordinator host = new(
    gameplayNetwork,
    networkEvents,
    mediatorEndPoint,
    publishedSession,
    joinClient);
 host.Poll(); // call each game frame while this coordinator owns polling
 ```
 Register normal game callbacks on `networkEvents.GameplayEvents`. Rendezvous
 reserves only its authenticated direct requests and forwards other callbacks.
 The same socket sends host presence, punches through the mediator, establishes
 the peer, and then carries gameplay. A NAT introduction is not success; accept a
 peer only after the coordinator reports the typed `Connected` outcome.
 `Poll()` does not fetch new invitations. Schedule
 `RefreshJoinAttemptsAsync` repeatedly for the entire hosting lifetime using a
 bounded caller-owned timer (the diagnostic uses 250 ms), never allow two refreshes
 to overlap, and inspect each typed result. The refresh performs HTTP work and
 queues a snapshot; it does not call the LiteNetLib manager. Continue calling
 `Poll()` on the manager's owning thread so the queued snapshot, presence traffic,
 and callbacks are processed. Run the lease maintainer concurrently and cancel
 both loops before disposing the coordinator.
 For example, start one sequential refresh loop when hosting begins and await it
 during shutdown:
 ```csharp
 static async Task RefreshInvitationsAsync(
    RendezvousHostCoordinator host,
    CancellationToken cancellationToken)
 {
    using PeriodicTimer timer = new(TimeSpan.FromMilliseconds(250));
    do
    {
        RendezvousClientResult<int> result =
            await host.RefreshJoinAttemptsAsync(cancellationToken);
        if (!result.IsSuccess)
        {
            ObserveBoundedHostRefreshFailure(result.Error);
        }
    }
    while (await timer.WaitForNextTickAsync(cancellationToken));
 }
 ```
 On the joining side, create an attempt through `RendezvousJoinClient`, then give
 the issued attempt to `RendezvousClientCoordinator` using the same manager and
 listener. Cancellation, outcome reporting, bounded deadlines, fallback, and
 lease-maintainer examples are in the packaged
 [`FinalFactory.Rendezvous.Client` README](../../src/FinalFactory.Rendezvous.Client/README.md).
 ## Host admission remains game-owned
 The coordinator privately validates and consumes the signed one-time connection
 ticket before accepting the LiteNetLib transport request. Do not create a second
 `ConnectionTicketValidator` beside it: the coordinator deliberately does not
 expose the expected or presented ticket. A connected transport proves only that
 Rendezvous authorized one attempt; it does not prove player identity,
 entitlement, capacity, ban status, or gameplay compatibility.
 Treat `AttemptCompleted` with a successful outcome and non-null `Peer` as the
 start of game-owned admission. Keep that peer outside authoritative gameplay
 until the game's normal authentication and admission exchange succeeds; disconnect
 it on rejection or timeout:
 ```csharp
 host.AttemptCompleted += (_, completed) =>
 {
    if (!completed.Outcome.IsSuccess || completed.Peer is null)
    {
        return;
    }
    BeginBoundedGameAuthentication(
        completed.Peer,
        onAccepted: AdmitToAuthoritativeGameplay,
        onRejected: peer => peer.Disconnect());
 };
 ```
 Revoke an attempt when the game cancels it. Never log a ticket or capability. A
 successful Rendezvous check must not bypass the game's authentication or
 authoritative server rules. `ConnectionTicketValidator` is a lower-level
 primitive for a custom transport integration that owns the complete request
 acceptance path; it is not an extra gate for `RendezvousHostCoordinator`.
 ## Provision each game and environment
 Provision game/environment scope before issuing credentials. The policy fixes
 enabled regions, exact gameplay protocols, visibility and publisher trust modes,
 metadata schema and byte budgets, quotas, and whether a dedicated fallback may
 be published. Unknown or disabled scope fails closed. Follow
 [game provisioning and signing-key lifecycle](../security/provisioning.md) for
 the complete schema, principal kinds, secret providers, overlap, and revocation.
 Dedicated publisher credentials belong only on trusted hosting infrastructure.
 Never ship one in a player build, repository, image layer, appsettings file, URL,
 argument, log, crash report, or analytics event. Issue a short-lived credential
 scoped to one game/environment and its allowed regions from the trusted
 deployment boundary. Player-host grants are issued to an authenticated player
 session at runtime and are never embedded in the build. Player-host grants,
 dedicated publishers, anonymous unlisted hosts, and operators are separate
 principal kinds; do not interchange them.
 Rotate signing keys with an overlap:
 1. install a new authorized key inside its `NotBefore`/`SignUntil` window;
 2. begin issuing with it while the old key remains verify-only;
 3. wait at least the maximum credential lifetime plus allowed clock skew;
 4. retire the old verifier after `VerifyUntil` and preserve custody records.
 A suspected compromise is not routine rotation: stop issuance, revoke the exact
 key through the protected operator route, remove or replace it in provisioning,
 invalidate affected credentials, and follow the
 [key-compromise runbook](../operations/incident-runbooks.md#signing-key-or-issuer-compromise).
 ## Metadata is public and policy-owned
 Treat listing metadata as untrusted public input. Define a small allowlist in
 each provisioned game's `MetadataValueMaxBytes`, set `RequiredMetadataKeys`, and
 keep `MetadataMaxKeys` and `MetadataMaxBytes` to the smallest useful values. Values
 must be display data only—for example a bounded map or ruleset identifier. Never
 publish player identity, free-form chat, secrets, access tokens, internal
 addresses, world state, or data needed for authoritative gameplay.
 The platform contract caps metadata at 32 keys, 256 UTF-8 bytes per value, and
 4096 encoded bytes total; tenant policy can and should be smaller. Build version
 and display name are separately bounded public fields. Games must escape metadata
 for their UI and must not infer trust from a listing being present.
 ## Local, staging, and production path
 Use the checked-in Compose profile only for the local TestClient guide. For a
 real environment:
 1. provision the game/environment policy and externally held signing keys;
 2. deploy one active service behind the source-preserving HTTPS/UDP topology in
   [secure single-active Linux deployment](../deployment/linux.md);
 3. install matching exact package versions in the game and set its service and
   mediator endpoints through environment-specific configuration;
 4. pass the TestClient health/publish/browse/punch/direct-traffic smoke using a
   short-lived diagnostic credential;
 5. run the topology harness and representative consumer-network trials; and
 6. monitor typed outcomes and bounded metrics before broad rollout.
 Rendezvous v1 has no relay, account system, matchmaking engine, server-browser
 UI, gameplay authority, or durable session database. A game owns player-facing
 recovery and an explicit fallback. Do not describe direct traversal as guaranteed.
@@ -1,97 +1,222 @@
-# Diagnostic TestClient integration guide
+# Start-to-finish TestClient guide
-Tracking: #25
+Tracking: #20, #25
-`FinalFactory.Rendezvous.TestClient` is the smallest supported public-SDK consumer.
+`FinalFactory.Rendezvous.TestClient` is the supported executable proof that a
-It exists for integration development, CI smoke checks, deployment verification,
+consumer can publish, browse, authorize, punch, connect, exchange direct traffic,
-and operator diagnosis. It is intentionally not a production game client, game
+and diagnose a failure using only the public Client and Contracts packages. It is
-server, matchmaking UI, or relay.
+intentionally thin: a polished server browser and player-facing connection UI
 belong in each game repository.
-The automated scenario matrix, privileged Linux namespace run, and topology
+> **Traversal boundary:** Rendezvous v1 is not a relay and cannot guarantee a
-limitations are documented in the [deterministic topology harness](topology-harness.md).
+> connection through symmetric NAT, carrier-grade NAT, restrictive firewalls,
 > VPNs, or platform policy. It provides no accounts, social system, skill-based
 > matchmaking, gameplay server, gameplay authority, or gameplay transport.
-## Prerequisites
+The automated scenario matrix, privileged Linux namespace run, and simulation
 limits are in the [deterministic topology harness](topology-harness.md). The
 [SDK seam guide](sdk-seams.md) covers the few integration details that this
 executable cannot show.
-Start a configured Rendezvous service and note both its HTTP base URL and UDP
+## First local connection from a clean checkout
-mediator endpoint. The host needs a tenant-scoped publisher credential from the
+
-deployment secret boundary. Put it in an environment variable and pass only that
+Prerequisites are the pinned .NET SDK, Docker with Compose, OpenSSL, Python 3,
-variable's name when the default is unsuitable:
+`curl`, and `jq`. Run these commands from the repository root. The generated key
 and credential are disposable local fixtures, not production provisioning.
 ```bash
-export RENDEZVOUS_PUBLISHER_CREDENTIAL='<deployment-supplied value>'
+install -d -m 0700 deploy/compose/secrets
 umask 077
 openssl rand -out deploy/compose/secrets/signing-key 32
 export RENDEZVOUS_UID="$(id -u)"
 export RENDEZVOUS_GID="$(id -g)"
 test "$RENDEZVOUS_UID" -ne 0
 docker compose -f deploy/compose/compose.yaml up --build --detach
 ready=false
 for attempt in {1..45}; do
  if curl --fail --silent http://127.0.0.1:8080/health/ready >/dev/null; then
    ready=true
    break
  fi
  sleep 1
 done
 test "$ready" = true
 curl --fail http://127.0.0.1:8080/health/live
 curl --fail http://127.0.0.1:8080/health/ready
 dotnet build src/FinalFactory.Rendezvous.TestClient --configuration Release
 ```
-Never put the credential in a command argument, URL, checked-in configuration,
+Only the host terminal needs a publisher credential. Disable shell tracing before
-shell trace, or captured test fixture. The development server's signing material
+capturing it; the helper prints the credential on stdout so command substitution
-is process-ephemeral; credentials from a prior development process are invalid.
+can place it directly in the environment without writing it to disk.
 ## Manual three-terminal flow
 Start the host:
 ```bash
-dotnet run --project src/FinalFactory.Rendezvous.TestClient -- \
+set +x
-  host --service http://127.0.0.1:5000/ --mediator 127.0.0.1:9050 \
+export RENDEZVOUS_PUBLISHER_CREDENTIAL="$(./scripts/mint-local-publisher-credential.sh)"
  --game space-game --environment development --region local --protocol 1
 ```
-Browse from another terminal:
+The helper accepts no arguments, reads the ignored `0600` local Compose key, and
 mints only `space-game` / `smoke` / `local` / protocol `1` for ten minutes. It is
 not a reusable issuer or an example for production. Never put the result in a
 command argument, URL, shell history, log, screenshot, support ticket, captured
 fixture, or source file.
 In terminal 1, publish a host. It stays alive for at most 60 seconds and exits
 after a joining peer completes the authenticated echo exchange:
 ```bash
-dotnet run --project src/FinalFactory.Rendezvous.TestClient -- \
+dotnet run --project src/FinalFactory.Rendezvous.TestClient \
-  browse --service http://127.0.0.1:5000/ \
+  --configuration Release --no-build -- \
-  --game space-game --environment development --region local --protocol 1
+  host --service http://127.0.0.1:8080/ --mediator 127.0.0.1:9050 \
  --game space-game --environment smoke --region local --protocol 1 \
  --display-name "Local diagnostic" --timeout-seconds 60 --run-seconds 60 \
  --exit-after-echo
 ```
-Join from a third terminal. Omit `--listing` for an interactive choice:
+Copy the public listing ID printed by the host, or discover it from terminal 2:
 ```bash
-dotnet run --project src/FinalFactory.Rendezvous.TestClient -- \
+dotnet run --project src/FinalFactory.Rendezvous.TestClient \
-  join --service http://127.0.0.1:5000/ --mediator 127.0.0.1:9050 \
+  --configuration Release --no-build -- \
-  --game space-game --environment development --region local --protocol 1 \
+  browse --service http://127.0.0.1:8080/ \
-  --listing 00000000-0000-0000-0000-000000000000
+  --game space-game --environment smoke --region local --protocol 1
 ```
-Replace the sample UUID with the public listing ID printed by host or browse.
+In terminal 3, either omit `--listing` and select interactively, or provide the
-Host and join each create one caller-owned LiteNetLib manager. That same socket
+copied ID for deterministic selection:
 sends presence/punch traffic, establishes the authenticated direct connection,
 and carries the ping/echo/ack/completion payload. The final completion confirms
 that the host received the reliable acknowledgement; none of this traffic passes through the HTTP
 service or UDP mediator.
-## CI and deployment smoke flow
+```bash
 dotnet run --project src/FinalFactory.Rendezvous.TestClient \
  --configuration Release --no-build -- \
  join --service http://127.0.0.1:8080/ --mediator 127.0.0.1:9050 \
  --game space-game --environment smoke --region local --protocol 1 \
  --listing REPLACE_WITH_LISTING_UUID --timeout-seconds 30
 ```
-Use `--script --json`, set `--listing` when deterministic selection matters, and
+Success means the joiner prints `join.connected` and verified direct traffic,
-check the documented process exit code. `--timeout-seconds` bounds each startup,
+and the host prints verified direct traffic before deregistering. The host and
-traversal, or direct-traffic stage; a script host also uses it as its total runtime
+joiner each create one caller-owned LiteNetLib manager. The same UDP socket sends
-unless `--run-seconds` is explicit. A host can add `--exit-after-echo` so it
+presence and punch traffic, accepts the authenticated peer, and carries the
-terminates after the joining peer acknowledges direct traffic and receives the
+ping/echo/ack/completion payload; direct traffic does not pass through the HTTP
-host's completion confirmation. Every wait is
+service or mediator.
 bounded by coordinator state and `--timeout-seconds`; no orchestration should use
 an unbounded sleep.
-The normal test suite contains a real process gate that starts the built Server,
+Clean up secrets and the disposable service when finished:
 host TestClient, and join TestClient, waits for readiness and versioned events,
 and verifies direct traffic, cleanup, JSON shape, and secret canaries. Process
 trees are force-terminated in the test cleanup path if normal shutdown fails.
-Useful success events are:
+```bash
 unset RENDEZVOUS_PUBLISHER_CREDENTIAL
 docker compose -f deploy/compose/compose.yaml down
 rm deploy/compose/secrets/signing-key
 ```
- `host.registered`, `host.ready`, `host.direct-traffic`, and `host.deregistered`;
+## Script and JSON automation
 - `browse.completed` and `browse.session`; and
 - `join.connected`, `join.direct-traffic`, and `join.outcome-report`.
-Failure events preserve stable typed phases and outcomes. When a terminal outcome
+`--script` forbids prompts and selects the first compatible listing unless
-contains a configured dedicated endpoint, `join.fallback` reports `available`
+`--listing UUID` fixes the choice. `--json` emits one JSON object per line with
-with endpoint type `dedicated`; no raw address is printed and no fallback is
+`version: 1`. New optional properties may be added, but event names and exit
-started implicitly.
+codes are stable automation contracts. Informational events use stdout and
 failures use stderr.
-## What the proof does and does not establish
+The deployment smoke performs the full health, publish, join, mediation, direct
 traffic, outcome-report, and cleanup flow using bounded waits:
-The deterministic loopback test proves the complete service/host/client protocol,
+```bash
-ticket admission, and peer-to-peer payload path. Loopback is not evidence that all
+dotnet build src/FinalFactory.Rendezvous.TestClient --configuration Release
-consumer routers, carrier-grade NATs, symmetric NATs, firewalls, VPNs, IPv6 paths,
+./scripts/smoke-deployment.sh
-or platform policies permit hole punching. Same-LAN, separated observed endpoints,
+```
-network namespaces/containers, mediator restart, and adverse topology coverage
+
-belong to the topology harness tracked by #14. Production rollout still requires
+For custom automation, capture JSON and preserve the process status separately:
-tests from representative networks and a game-owned fallback policy.
+
 ```bash
 set +e
 dotnet run --project src/FinalFactory.Rendezvous.TestClient \
  --configuration Release --no-build -- \
  browse --service http://127.0.0.1:8080/ \
  --game space-game --environment smoke --region local --protocol 1 \
  --script --json >browse.jsonl
 status=$?
 set -e
 jq -e 'select(.version == 1 and .event == "browse.completed")' browse.jsonl
 test "$status" -eq 0
 ```
 Never use an unbounded sleep to orchestrate processes. Wait for versioned events
 such as `host.ready` and apply a deadline. Useful success events are
 `host.registered`, `host.ready`, `host.direct-traffic`, `host.deregistered`,
 `browse.completed`, `browse.session`, `join.connected`, `join.direct-traffic`,
 and `join.outcome-report`.
 | Exit | Meaning |
 | ---: | --- |
 | `0` | Requested diagnostic flow completed successfully |
 | `2` | Invalid command or options |
 | `3` | Missing or invalid local configuration |
 | `10` | HTTP, registration, browser, lease, or socket failure |
 | `11` | No compatible session was available or selected |
 | `12` | Authorization or traversal reached a typed terminal failure |
 | `13` | Direct connection succeeded but the direct traffic proof failed |
 | `130` | Caller cancellation or Ctrl+C |
 ## Observe a safe failure
 Run this after the protocol-1 browse in terminal 2 and before the terminal-3
 join (or restart terminal 1 first). The preceding browse proves that one
 protocol-1 host is present. Now browse for deliberately incompatible protocol
 `999`. The command emits a successful directory response with
 `browse.completed`, `count: 0`, then exits `11` to distinguish compatibility
 from a service outage:
 ```bash
 set +e
 dotnet run --project src/FinalFactory.Rendezvous.TestClient \
  --configuration Release --no-build -- \
  browse --service http://127.0.0.1:8080/ \
  --game space-game --environment smoke --region local --protocol 999 \
  --script --json >incompatible.jsonl
 status=$?
 set -e
 jq -e 'select(.event == "browse.completed" and .phase == "directory" and .count == 0)' \
  incompatible.jsonl
 test "$status" -eq 11
 ```
 This is a diagnostic failure drill, not a bypass: unknown tenant scope and
 protocols still fail closed, and the local helper cannot mint a credential for
 them.
 ## Diagnose by phase, not by guesswork
 Start with the exit code, then the last versioned event and its `phase`, `status`,
 and typed `outcome`. Endpoint categories may be
 reported as `loopback`, `private`, or `public`; raw endpoints, credentials,
 capabilities, metadata, and player identities are never emitted.
 | Symptom or last event | Distinction | Check next |
 | --- | --- | --- |
 | `host.configuration`, exit `3` | Local credential variable is missing or malformed before any request | Confirm the named environment variable exists, tracing is off, and the credential has not expired |
 | `host.registration`, exit `10` | Publisher authentication, tenant policy, metadata, quota, or HTTP failure | Use the typed status; compare credential scope with game/environment/region and the provisioned policy, then correlate protected server telemetry by operation and time |
 | `browse.sessions`, exit `10` | Directory request failed | Check HTTP reachability, `/health/ready`, rate limiting, and contract compatibility |
 | `browse.completed` count `0`, or `join.selection` empty, exit `11` | Healthy directory but no compatible visible listing | Match game, environment, region, and exact gameplay protocol; then confirm a host lease is still active |
 | Exact `join.selection` failure, exit `10` | Listing disappeared, is hidden, or scope no longer matches | Browse again; do not retry an old listing ID forever |
 | `join.authorization`, exit `12` | Service rejected the attempt before NAT traversal | Inspect typed category/outcome for policy, capacity, stale host, or active-attempt limits |
 | `join.punch` / `join.traversal`, exit `12` | Mediation or NAT traversal did not establish a peer | Confirm UDP endpoint/reply path, host presence, clocks, firewall/NAT behavior, and topology; use a game-owned fallback if policy supplies one |
 | `join.direct-connect`, exit `12` | Introduction occurred but authenticated direct admission failed | Confirm host is polling the same socket, the one-time ticket is current, and game admission did not reject capacity, identity, or bans |
 | `join.connected` followed by exit `13` | Peer connected but the direct gameplay-like echo did not finish | Inspect the peer lifecycle and caller polling; this is not an HTTP/directory failure |
 Stopping a host without deregistration may leave its listing visible only until
 the bounded lease expires. During that window, a join can produce a typed stale
 host or traversal outcome; it must not be interpreted as a healthy host. Restarting
 the single-active service intentionally loses all ephemeral listings and attempts,
 so hosts re-register and clients browse again.
 If a terminal outcome reports an authoritative dedicated fallback,
 `join.fallback` exposes only availability and endpoint type. TestClient never
 connects to it automatically. The game owns the decision, authentication, and
 connection policy. If no fallback is present, Rendezvous v1 offers no relay.
 ## Production use
 Do not copy a production signing key to a diagnostic host. Supply a short-lived,
 least-scope publisher credential from the deployment secret boundary and set the
 external service, mediator, and matching scope variables described in the
 [secure Linux deployment smoke](../deployment/linux.md#http-and-udp-smoke).
 Run representative external-network tests; loopback success is not NAT coverage.
@@ -0,0 +1,339 @@
 # Incident and change runbooks
 Tracking: #20
 These runbooks supplement the [signal and operator reference](observability-and-operator-runbook.md).
 Every procedure has four explicit gates: detect, contain, recover, and verify.
 Record timestamps, the release digest, bounded aggregates, audit fingerprints,
 and `X-Rendezvous-Correlation-ID` values. Never copy credentials, capabilities,
 connection tickets, signing material, player identity, raw IP addresses,
 endpoints, listing metadata, or full request bodies into an incident record.
 Operator routes must be reachable only from an allowed management source. Use a
 short-lived, least-permission operator credential minted outside Rendezvous.
 Pass it to an approved operator client through protected stdin or a secret agent,
 not a URL, command argument, environment-wide process launcher, shell trace, or
 ticket. All request shapes and responses are defined by the generated
 [OpenAPI v1 document](../api/rendezvous-v1.json).
 Before an incident, keep these protected records available without depending on
 the affected service: current and previous image digests, matching configuration,
 key IDs and lifecycle windows (not raw key values), the game owner/on-call map,
 capacity baselines, collector destinations, and a separately authorized
 break-glass operator key. Test management-source allowlisting and credential
 permissions at least once per release.
 Use the exact versioned action shapes below. Confirmation fields deliberately
 repeat the target so a stale UI selection or copy error fails closed. Responses
 do not echo targets.
 | Operation | JSON body |
 | --- | --- |
 | `POST /v1/operator/listings/revoke` | `{"listingId":"<uuid>","confirmListingId":"<same uuid>"}` |
 | `POST /v1/operator/principals/revoke` | `{"subject":"<exact subject>","confirmSubject":"<same subject>","lifetimeSeconds":60}` |
 | `POST /v1/operator/keys/revoke` | `{"keyId":"<key id>","confirmKeyId":"<same key id>"}` |
 | `POST /v1/operator/drain` | `{"confirmation":"DRAIN"}` |
 ## Abuse or authentication spike
 ### Detect
 - Alert on a baseline-relative increase in `rendezvous.limiter.drops`, HTTP/UDP
  request rate, `rendezvous.operator.authentication` rejected/forbidden results,
  registration requests by authentication status, queue depth, or p95/p99 latency.
 - Check `/health/live`, `/health/ready`, `rendezvous.store.available`, and
  authenticated `GET /v1/operator/status`. Separate public-source rejection,
  publisher credential failure, operator probing, and ordinary capacity growth.
 - Use only bounded operation/result dimensions and correlation IDs. Do not group
  by raw address, token, subject, listing ID, or metadata.
 ### Contain
 - Preserve the dedicated operator partition. Do not raise public limits during
  an active spike. Apply source-preserving edge rate controls only when their
  collateral effect is understood and UDP source address/port remains intact.
 - For one abusive session, call `POST /v1/operator/listings/revoke` with identical
  `listingId` and `confirmListingId`. For a confirmed publisher subject, call
  `POST /v1/operator/principals/revoke` with identical `subject` and
  `confirmSubject` and a 1–600 second lifetime.
 - Revoke a signing key only when compromise evidence implicates that issuer;
  broad key revocation invalidates every credential signed by it. Drain only if
  the process itself must be isolated.
 ### Recover
 - Correct the source integration, edge rule, leaked principal grant, or tenant
  budget under change control. Let a bounded principal revocation expire only
  after the owner confirms remediation; a repeated shorter revocation never
  shortens the original deadline.
 - Restore normal limits gradually. If saturation caused state churn, allow leases
  and attempts to expire naturally rather than deleting arbitrary state.
 ### Verify
 - Require limiter drops, authentication result ratios, queue depth, latency, and
  direct-connect outcomes to return to the same-region baseline for the agreed
  observation window.
 - Confirm readiness stayed healthy or recovered, operator audit contains the
  intended action/result fingerprint, revoked resources cannot create new work,
  and unaffected tenants can still publish, browse, and connect.
 ## Signing key or issuer compromise
 ### Detect
 - Treat secret-manager access alerts, unexpected issuance, credentials outside
  the expected region/kind, a signing-key expiry alarm, or unexplained publisher
  authentication growth as compromise until disproved.
 - Identify the non-secret key ID, allowed credential kinds, game/environment
  binding, `NotBefore`, `SignUntil`, and `VerifyUntil`. Do not retrieve or paste
  raw material merely to compare it.
 ### Contain
 - Stop the affected external issuer and deny further access to its secret.
 - From a separate uncompromised break-glass operator key with `RotateKeys`, call
  `POST /v1/operator/keys/revoke` with identical `keyId` and `confirmKeyId`.
  Runtime revocation is immediate but process-local.
 - Remove or mark the key revoked in authoritative provisioning before any
  restart. Revoke affected principals/listings when narrower evidence supports
  it. Do not drain automatically unless the running instance cannot be trusted.
 ### Recover
 - Generate replacement material in the approved secret boundary, use a new key
  ID, bind it to the exact credential kind and tenant, and deploy configuration
  referencing the secret—never the secret value.
 - Resume issuance with short lifetimes. Reissue only to authenticated workloads.
  When confidentiality is lost, do not use normal overlap to keep compromised
  credentials valid; document the intentional invalidation window.
 - Rotate any release, registry, or operator credential exposed by the same
  incident through its owning system; Rendezvous key revocation cannot revoke
  unrelated systems.
 ### Verify
 - Confirm `GET /v1/operator/status` shows the compromised key revoked and the
  replacement signing, old credentials fail, new exact-scope credentials work,
  and the result survives a controlled restart from updated provisioning.
 - Pass TestClient registration, browse, authenticated mediation, and direct
  traffic with the replacement; monitor authentication and audit results through
  at least the maximum newly issued credential lifetime.
 ## Targeted listing or publisher revocation
 ### Detect
 - Validate the abuse report against game-owned records and bounded Rendezvous
  evidence. Determine whether the target is one listing or an authenticated
  publisher subject. Do not use display name, metadata, or a raw address as
  identity.
 - Confirm current aggregate state through `GET /v1/operator/status` and record
  the correlation IDs that justified action.
 ### Contain
 - Revoke one listing with `POST /v1/operator/listings/revoke`; the exact listing
  UUID must appear in both confirmation fields.
 - Revoke a publisher with `POST /v1/operator/principals/revoke`; the exact subject
  must appear in both confirmation fields and `lifetimeSeconds` must be 1–600.
  This removes that principal's active listings and attempts and blocks new ones
  for the bounded lifetime.
 - Choose the narrowest action. Do not revoke a tenant key for a single listing.
 ### Recover
 - The game owner resolves the ban, account, workload, or configuration issue in
  the authoritative game system. Rendezvous does not own user accounts or bans.
 - After the original revocation deadline, permit a newly authenticated publisher
  to register. There is no un-revoke endpoint and no recovery of removed
  ephemeral listings; the host creates a new listing.
 ### Verify
 - Confirm the old listing is no longer browsable or joinable, the principal
  cannot publish during its lifetime, and the audit action/result is present
  without the raw target.
 - Confirm unrelated publishers in the same tenant and another tenant still pass
  publish/browse/join/direct-traffic checks.
 ## Planned restart or crash recovery
 ### Detect
 - Planned restart begins with a recorded change and a healthy current baseline.
  Crash recovery begins when liveness/process state fails or both TCP 8080 and
  UDP 9050 stop answering. Distinguish dependency/readiness failure from a dead
  process; liveness deliberately remains healthy for some recoverable failures.
 - Record active listing/lease/attempt aggregates. They are informational only:
  v1 has no durable runtime database to restore.
 ### Contain
 - For a planned stop, call `POST /v1/operator/drain` with confirmation exactly
  `DRAIN`. Require readiness `503`, liveness `200`, and removal from new traffic.
  Allow the bounded drain deadline to finish, then send SIGTERM.
 - Never start a second active instance while the old process owns the advertised
  HTTP/UDP endpoints. On crash, fence the old process/host and verify both sockets
  are released before replacement.
 ### Recover
 - Start exactly one instance from the recorded immutable image digest and matching
  reviewed configuration/key references. A restart intentionally loses listings,
  observed endpoints, attempts, replay markers, and runtime-only revocations.
 - Ensure any emergency key revocation is also present in authoritative
  provisioning. Hosts must re-register; clients must browse and start new
  attempts. Do not restore stale ephemeral state from logs or backups.
 ### Verify
 - Require live and ready health, UDP bind, store availability, and one active
  target. Run the full deployment smoke and confirm host re-registration begins.
 - Verify no pre-restart listing or capability is accepted, runtime revocations
  that should persist are configuration-backed, and latency/outcomes stabilize.
 ## Release rollback
 ### Detect
 - Trigger rollback from a predeclared objective: readiness loss, failed deployment
  smoke, contract/package incompatibility, security regression, direct-success
  regression beyond threshold, or sustained resource regression. Record the new
  and previous digests and the evidence; do not move a tag.
 ### Contain
 - Stop promotion and new rollout work. Drain and stop the faulty single active
  instance, then verify both public sockets are released. Revoke affected keys or
  principals only when the defect creates an authorization risk.
 - Preserve logs, artifacts, provenance, signatures, and the faulty release record.
  Never overwrite or delete an immutable package/image to reuse its version.
 ### Recover
 - Deploy the previous known-good image by digest with its compatible configuration
  and key set. Do not run old and new concurrently. If configuration changed,
  apply its reviewed down-migration before starting.
 - Publish a corrected build under a new SemVer after diagnosis; mark faulty release
  notes withdrawn when appropriate.
 ### Verify
 - Check the running image digest, live/ready health, one active target, UDP source
  preservation, and the complete TestClient deployment smoke.
 - Confirm package/server compatibility from `GET /v1/operator/status`, hosts
  re-register, and the rollback objective returns to baseline for the observation
  window.
 ## Capacity saturation
 ### Detect
 - Page when `rendezvous.queue.depth` remains above 90% of the configured attempt
  limit, lease-critical work is shed, `rendezvous.store.available` is zero, or no
  ready instance remains. Warn at 70%, sustained `rendezvous.limiter.drops`, or
  p95 latency above objective.
 - Compare CPU, memory, file descriptors, UDP errors, expiry churn, HTTP operation
  rate, and typed connection outcomes with the measured
  [capacity profile](capacity-and-resilience.md). Distinguish legitimate growth,
  attack traffic, downstream telemetry pressure, and a regression.
 ### Contain
 - Preserve lease-critical and operator reserves. Shed new browse/join work with
  the existing typed `429`/`Retry-After` behavior; do not add an unbounded queue.
 - Apply per-tenant/source controls at the appropriate trusted boundary. If the
  process is unstable, drain new work and recover on one replacement rather than
  adding a second active replica; v1 state is process-local.
 ### Recover
 - Remove the causal load or deploy a tested higher single-instance resource and
  budget profile. Change CPU/memory and server limits together, using the numeric
  gate and accelerated soak before production.
 - Long-term horizontal scaling requires a designed shared directory, replay, and
  attempt authority. A generic load balancer is not that design.
 ### Verify
 - Re-run the capacity/resilience gate at the chosen profile, then require queue,
  limiter drops, expiry churn, latency, store health, and direct-success ratio to
  remain within objectives through the production observation window.
 - Confirm termination still completes within `DrainDeadlineSeconds + 5` and the
  public and operator partitions behave independently.
 ## Privacy or telemetry incident
 ### Detect
 - Trigger on any credential, token, capability, player identity, raw IP/endpoint,
  listing ID, metadata, or caller-reported exact connection duration tied to an
  event or identity found in logs, metrics, traces, crash reports, support systems,
  or analytics. Aggregate HTTP/UDP duration histograms with bounded operation tags
  are expected telemetry. Also trigger when audit data exceeds its approved 30-day
  retention without an incident hold.
 - Identify the producing version, sink, access population, retention/replication
  path, and time window without copying the exposed value into a new system.
 ### Contain
 - Stop or filter the offending export and restrict access to affected sinks.
  Preserve the minimum evidence under the incident process; do not take broad
  diagnostic dumps that amplify exposure.
 - Revoke exposed reusable credentials/keys through their owning boundary. Listing
  IDs and endpoints are not authentication secrets, but remove affected listings
  if continued exposure creates risk. Notify privacy/security owners according to
  applicable policy and law.
 ### Recover
 - Patch the producer to the allowlisted telemetry model, test canary redaction
  across logs/metrics/traces/output, and deploy through the immutable release
  path. Delete or age out affected data from every sink according to approved
  retention and legal-hold direction.
 - Replace exposed credentials and re-register hosts when necessary. Do not claim
  that a service restart deletes copies already exported to collectors.
 ### Verify
 - Search new telemetry using non-secret synthetic canaries and confirm no canary
  or prohibited field crosses the boundary. Verify audit records contain only
  fixed fields and fingerprints and that retention/eviction is operating.
 - Security/privacy owners confirm sink cleanup, access review, notification, and
  monitoring closure before the incident is resolved.
 ## Dependency or base-image upgrade
 ### Detect
 - Open a reviewed change for an advisory, end-of-support date, pinned-digest
  refresh, or planned package update. Record affected package/image, current and
  proposed exact version/digest, advisory severity, exploitability, and required
  deadline. Never float to `latest` as remediation.
 ### Contain
 - For an actively exploited critical issue, restrict exposure or stop the service
  under incident authority while building the fix. Revoking publisher keys does
  not repair a vulnerable runtime. Otherwise keep the known-good release running
  while the candidate is tested.
 ### Recover
 - Update the SDK/base-image digest, lock files, license/advisory evidence, SBOM,
  compatibility matrix, and release notes together. For LiteNetLib or a wire/API
  change, apply the explicit version/migration policy rather than silently
  replacing compatible bytes.
 - Run locked restore, formatting, Debug and Release builds/tests, public contract
  and package gates, real consumer restores, reproducible artifact/image builds,
  vulnerability scan, signatures, topology/deployment smoke, and capacity checks
  proportional to the change. Promote the exact tested digest.
 ### Verify
 - Verify signatures, provenance, checksums, SBOM contents, running digest, and
  absence of the advisory in the shipped artifact—not merely the build host.
 - Require live/ready health, TestClient direct traffic, real consumer compatibility,
  and normal latency/outcomes. Keep the previous digest and compatible config for
  rollback until the observation window closes.
@@ -1,4 +1,4 @@
-# Observability and operator runbook
+# Observability and operator reference
 This runbook defines the production signals and privileged controls for the
 Rendezvous service. The service emits `System.Diagnostics.Metrics` instruments
@@ -6,6 +6,10 @@ from the `FinalFactory.Rendezvous` meter and distributed-tracing activities from
 `FinalFactory.Rendezvous.Server`. Connect those sources to the deployment's
 OpenTelemetry or equivalent collector. Do not add identifiers to metric labels.
 Concrete detect/contain/recover/verify procedures for abuse, key compromise,
 targeted revocation, restart, rollback, saturation, privacy incidents, and
 dependency upgrades are in the [incident and change runbooks](incident-runbooks.md).
 ## Health and readiness
 - `GET /health/live` proves that the HTTP process can answer. It deliberately
@@ -0,0 +1,77 @@
 #!/usr/bin/env bash
 set -euo pipefail
 ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
 LOCAL_KEY="${RENDEZVOUS_SMOKE_LOCAL_KEY:-$ROOT/deploy/compose/secrets/signing-key}"
 if (( $# != 0 )); then
    printf 'This helper accepts no arguments and mints only the fixed local Compose smoke scope.\n' >&2
    exit 2
 fi
 command -v python3 >/dev/null || {
    printf 'Missing required command: python3\n' >&2
    exit 2
 }
 # This is deliberately a local-fixture tool, not a general credential issuer.
 # Python reads the raw key from the protected file; key material never appears in
 # a child process argument, environment value, temporary file, or command output.
 python3 - "$LOCAL_KEY" <<'PY'
 import base64
 import hashlib
 import hmac
 import json
 import os
 import secrets
 import stat
 import sys
 import time
 key_path = sys.argv[1]
 try:
    metadata = os.lstat(key_path)
 except FileNotFoundError:
    raise SystemExit(f"Local Compose smoke key does not exist: {key_path}")
 if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISREG(metadata.st_mode):
    raise SystemExit(f"Local Compose smoke key must be a regular non-symlink file: {key_path}")
 parent_path = os.path.dirname(os.path.abspath(key_path))
 parent = os.lstat(parent_path)
 if stat.S_ISLNK(parent.st_mode) or not stat.S_ISDIR(parent.st_mode):
    raise SystemExit(f"Local Compose secret directory must be a non-symlink directory: {parent_path}")
 if parent.st_uid != os.geteuid() or parent.st_mode & 0o077:
    raise SystemExit(f"Local Compose secret directory must be owned by this user with mode 0700: {parent_path}")
 if metadata.st_uid != os.geteuid() or metadata.st_mode & 0o022 or metadata.st_nlink != 1:
    raise SystemExit(f"Local Compose smoke key must be owned by this user, single-linked, and not group/world writable: {key_path}")
 with open(key_path, "rb") as key_file:
    key = key_file.read(33)
 if len(key) != 32:
    raise SystemExit(f"Local Compose smoke key must be exactly 32 bytes: {key_path}")
 now = int(time.time())
 payload = {
    "version": 1,
    "issuer": "final-factory-rendezvous-smoke",
    "audience": "rendezvous-service",
    "subject": "local-smoke-host",
    "kind": "dedicatedPublisher",
    "gameId": "space-game",
    "environmentId": "smoke",
    "regions": ["local"],
    "permissions": [],
    "issuedAtUnixSeconds": now,
    "notBeforeUnixSeconds": now,
    "expiresAtUnixSeconds": now + 600,
    "nonce": secrets.token_hex(16),
 }
 def base64url(value: bytes) -> str:
    return base64.urlsafe_b64encode(value).rstrip(b"=").decode("ascii")
 encoded = base64url(json.dumps(payload, separators=(",", ":")).encode("utf-8"))
 signed = f"rv1.local-smoke-1.{encoded}"
 signature = base64url(hmac.new(key, signed.encode("ascii"), hashlib.sha256).digest())
 print(f"{signed}.{signature}")
 PY
@@ -13,7 +13,7 @@ ENVIRONMENT_ID="${RENDEZVOUS_SMOKE_ENVIRONMENT_ID:-smoke}"
 REGION="${RENDEZVOUS_SMOKE_REGION:-local}"
 PROTOCOL_VERSION="${RENDEZVOUS_SMOKE_PROTOCOL_VERSION:-1}"
-for command in curl date dotnet jq mktemp od openssl tail tr wc; do
+for command in curl dotnet jq mktemp tail; do
    command -v "$command" >/dev/null || {
        printf 'Missing required command: %s\n' "$command" >&2
        exit 2
@@ -35,39 +35,14 @@ for scoped_value in "$GAME_ID" "$ENVIRONMENT_ID" "$REGION"; do
    fi
 done
 base64url() {
    openssl base64 -A | tr '+/' '-_' | tr -d '='
 }
 local_credential() {
-    if [[ ! -f "$LOCAL_KEY" ]] || [[ "$(wc -c < "$LOCAL_KEY")" -ne 32 ]]; then
+    if [[ "$GAME_ID" != space-game || "$ENVIRONMENT_ID" != smoke \
-        printf 'Local Compose smoke key must be exactly 32 bytes: %s\n' "$LOCAL_KEY" >&2
+        || "$REGION" != local || "$PROTOCOL_VERSION" != 1 ]]; then
        printf 'The local credential helper supports only space-game/smoke/local protocol 1. Supply RENDEZVOUS_PUBLISHER_CREDENTIAL for any other scope.\n' >&2
        exit 2
    fi
-
+    RENDEZVOUS_SMOKE_LOCAL_KEY="$LOCAL_KEY" \
-    local now expires nonce payload encoded signed hex signature
+        "$ROOT/scripts/mint-local-publisher-credential.sh"
    now="$(date +%s)"
    expires="$((now + 600))"
    nonce="$(openssl rand -hex 16)"
    payload="$(jq -cn \
        --arg issuer final-factory-rendezvous-smoke \
        --arg audience rendezvous-service \
        --arg subject local-smoke-host \
        --arg kind dedicatedPublisher \
        --arg gameId "$GAME_ID" \
        --arg environmentId "$ENVIRONMENT_ID" \
        --arg region "$REGION" \
        --arg nonce "$nonce" \
        --argjson now "$now" \
        --argjson expires "$expires" \
        '{version:1,issuer:$issuer,audience:$audience,subject:$subject,kind:$kind,gameId:$gameId,environmentId:$environmentId,regions:[$region],permissions:[],issuedAtUnixSeconds:$now,notBeforeUnixSeconds:$now,expiresAtUnixSeconds:$expires,nonce:$nonce}')"
    encoded="$(printf '%s' "$payload" | base64url)"
    signed="rv1.local-smoke-1.$encoded"
    hex="$(od -An -v -tx1 "$LOCAL_KEY" | tr -d ' \n')"
    signature="$(printf '%s' "$signed" \
        | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$hex" -binary \
        | base64url)"
    printf '%s.%s' "$signed" "$signature"
 }
 credential="${RENDEZVOUS_PUBLISHER_CREDENTIAL:-}"
@@ -13,13 +13,16 @@ authenticated direct peer, and answers a bounded ping/echo/ack/completion exchan
  LiteNetLib socket, proves direct traffic, reports the typed outcome, and exits.
 Run `dotnet run --project src/FinalFactory.Rendezvous.TestClient -- --help` for
-the complete option reference. A typical script-mode invocation is:
+the complete option reference. The repository's
 [start-to-finish guide](../../docs/integration/test-client.md) provides an
 executable local Compose setup, safe failure drill, JSON automation, and a
 phase-by-phase diagnostic table. A typical deployment invocation is:
 ```bash
 export RENDEZVOUS_PUBLISHER_CREDENTIAL='<credential from the deployment boundary>'
 dotnet run --project src/FinalFactory.Rendezvous.TestClient -- \
-  host --service http://127.0.0.1:5000/ --mediator 127.0.0.1:9050 \
+  host --service https://rendezvous.example/ --mediator rendezvous.example:9050 \
-  --game space-game --environment development --region local --protocol 1 \
+  --game space-game --environment production --region eu-central --protocol 1 \
  --script --json --exit-after-echo
 ```
@@ -207,10 +207,16 @@ public sealed class ProductionProcessTests
        string root = RepositoryRoot();
        int httpPort = ReserveTcpPort();
        int udpPort = ReserveUdpPort();
-        string secretPath = Path.Combine(
+        string secretDirectory = Path.Combine(
            Path.GetTempPath(),
            $"rendezvous-smoke-secret-{Guid.NewGuid():N}");
        Directory.CreateDirectory(secretDirectory);
        File.SetUnixFileMode(
            secretDirectory,
            UnixFileMode.UserRead | UnixFileMode.UserWrite | UnixFileMode.UserExecute);
        string secretPath = Path.Combine(secretDirectory, "signing-key");
        await File.WriteAllBytesAsync(secretPath, RandomNumberGenerator.GetBytes(32));
        File.SetUnixFileMode(secretPath, UnixFileMode.UserRead | UnixFileMode.UserWrite);
        Process? server = null;
        Process? smoke = null;
        try
@@ -294,6 +300,7 @@ public sealed class ProductionProcessTests
            }
            File.Delete(secretPath);
            Directory.Delete(secretDirectory);
        }
    }
@@ -0,0 +1,213 @@
 using System.Text.Json;
 using System.Text.RegularExpressions;
 using System.Xml.Linq;
 namespace FinalFactory.Rendezvous.Tests.Documentation;
 public sealed partial class DocumentationContractTests
 {
    private static readonly string[] IncidentScenarios =
    [
        "Abuse or authentication spike",
        "Signing key or issuer compromise",
        "Targeted listing or publisher revocation",
        "Planned restart or crash recovery",
        "Release rollback",
        "Capacity saturation",
        "Privacy or telemetry incident",
        "Dependency or base-image upgrade",
    ];
    [Fact]
    public void TestClientGuideDocumentsTheExecutableSuccessAndFailureContracts()
    {
        string root = FindRepositoryRoot();
        string guide = File.ReadAllText(Path.Combine(root, "docs", "integration", "test-client.md"));
        Assert.Contains("space-game --environment smoke --region local --protocol 1", guide, StringComparison.Ordinal);
        Assert.Contains("mint-local-publisher-credential.sh", guide, StringComparison.Ordinal);
        Assert.Contains("join.connected", guide, StringComparison.Ordinal);
        Assert.Contains("join.direct-traffic", guide, StringComparison.Ordinal);
        Assert.Contains("browse.completed", guide, StringComparison.Ordinal);
        Assert.Contains("--script --json", guide, StringComparison.Ordinal);
        Assert.Contains("--run-seconds 60", guide, StringComparison.Ordinal);
        Assert.Contains("for attempt in {1..45}", guide, StringComparison.Ordinal);
        Assert.Contains("before the terminal-3", guide, StringComparison.Ordinal);
        Assert.Contains("test \"$status\" -eq 11", guide, StringComparison.Ordinal);
        Assert.Contains("no relay", guide, StringComparison.OrdinalIgnoreCase);
        Assert.Contains("cannot guarantee", guide, StringComparison.OrdinalIgnoreCase);
        Assert.DoesNotMatch(ReusableCredential(), guide);
    }
    [Fact]
    public void LocalCredentialHelperIsFixedScopeAndSmokeDelegatesToIt()
    {
        string root = FindRepositoryRoot();
        string helper = File.ReadAllText(Path.Combine(root, "scripts", "mint-local-publisher-credential.sh"));
        string smoke = File.ReadAllText(Path.Combine(root, "scripts", "smoke-deployment.sh"));
        Assert.Contains("if (( $# != 0 ));", helper, StringComparison.Ordinal);
        Assert.Contains("\"gameId\": \"space-game\"", helper, StringComparison.Ordinal);
        Assert.Contains("\"environmentId\": \"smoke\"", helper, StringComparison.Ordinal);
        Assert.Contains("\"regions\": [\"local\"]", helper, StringComparison.Ordinal);
        Assert.Contains("now + 600", helper, StringComparison.Ordinal);
        Assert.Contains("stat.S_ISLNK", helper, StringComparison.Ordinal);
        Assert.Contains("parent.st_mode & 0o077", helper, StringComparison.Ordinal);
        Assert.Contains("metadata.st_mode & 0o022", helper, StringComparison.Ordinal);
        Assert.Contains("metadata.st_nlink != 1", helper, StringComparison.Ordinal);
        Assert.Contains("mint-local-publisher-credential.sh", smoke, StringComparison.Ordinal);
        Assert.DoesNotContain("hexkey:", smoke, StringComparison.Ordinal);
        Assert.DoesNotContain("openssl dgst", smoke, StringComparison.Ordinal);
    }
    [Fact]
    public void EveryIncidentRunbookHasDetectContainRecoverAndVerifyGates()
    {
        string root = FindRepositoryRoot();
        string runbooks = File.ReadAllText(Path.Combine(root, "docs", "operations", "incident-runbooks.md"));
        for (int index = 0; index < IncidentScenarios.Length; index++)
        {
            string heading = $"## {IncidentScenarios[index]}";
            int start = runbooks.IndexOf(heading, StringComparison.Ordinal);
            Assert.True(start >= 0, $"Missing incident runbook heading: {heading}");
            int end = index + 1 < IncidentScenarios.Length
                ? runbooks.IndexOf($"## {IncidentScenarios[index + 1]}", start, StringComparison.Ordinal)
                : runbooks.Length;
            Assert.True(end > start, $"Could not find the end of runbook: {heading}");
            string scenario = runbooks[start..end];
            Assert.Contains("### Detect", scenario, StringComparison.Ordinal);
            Assert.Contains("### Contain", scenario, StringComparison.Ordinal);
            Assert.Contains("### Recover", scenario, StringComparison.Ordinal);
            Assert.Contains("### Verify", scenario, StringComparison.Ordinal);
        }
        Assert.DoesNotMatch(ReusableCredential(), runbooks);
    }
    [Fact]
    public void DocumentedOperatorOperationsAndBodiesMatchTheReleasedOpenApi()
    {
        string root = FindRepositoryRoot();
        string runbooks = File.ReadAllText(Path.Combine(root, "docs", "operations", "incident-runbooks.md"));
        using JsonDocument openApi = JsonDocument.Parse(File.ReadAllText(
            Path.Combine(root, "docs", "api", "rendezvous-v1.json")));
        JsonElement paths = openApi.RootElement.GetProperty("paths");
        (string Method, string Path)[] documentedOperations = OperatorRoute().Matches(runbooks)
            .Cast<Match>()
            .Select(static match => (
                match.Groups["method"].Value.ToLowerInvariant(),
                match.Groups["path"].Value))
            .Distinct()
            .ToArray();
        Assert.NotEmpty(documentedOperations);
        Assert.All(documentedOperations, operation =>
            Assert.True(
                paths.TryGetProperty(operation.Path, out JsonElement path)
                && path.TryGetProperty(operation.Method, out _),
                $"OpenAPI does not contain {operation.Method.ToUpperInvariant()} {operation.Path}."));
        Match[] actions = OperatorAction().Matches(runbooks).Cast<Match>().ToArray();
        Assert.Equal(4, actions.Length);
        foreach (Match action in actions)
        {
            string method = action.Groups["method"].Value.ToLowerInvariant();
            string path = action.Groups["path"].Value;
            using JsonDocument body = JsonDocument.Parse(action.Groups["body"].Value);
            string reference = paths.GetProperty(path)
                .GetProperty(method)
                .GetProperty("requestBody")
                .GetProperty("content")
                .GetProperty("application/json")
                .GetProperty("schema")
                .GetProperty("$ref")
                .GetString()!;
            string schemaName = reference["#/components/schemas/".Length..];
            string[] required = openApi.RootElement.GetProperty("components")
                .GetProperty("schemas")
                .GetProperty(schemaName)
                .GetProperty("required")
                .EnumerateArray()
                .Select(static property => property.GetString()!)
                .Order(StringComparer.Ordinal)
                .ToArray();
            string[] documented = body.RootElement.EnumerateObject()
                .Select(static property => property.Name)
                .Order(StringComparer.Ordinal)
                .ToArray();
            Assert.Equal(required, documented);
        }
    }
    [Fact]
    public void SdkGuideMatchesTheReleasedPackageAndTransportMatrix()
    {
        string root = FindRepositoryRoot();
        string guide = File.ReadAllText(Path.Combine(root, "docs", "integration", "sdk-seams.md"));
        using JsonDocument compatibility = JsonDocument.Parse(File.ReadAllText(
            Path.Combine(root, "docs", "releases", "compatibility.json")));
        JsonElement matrix = compatibility.RootElement;
        JsonElement packages = matrix.GetProperty("packages");
        string clientVersion = packages.GetProperty("FinalFactory.Rendezvous.Client").GetString()!;
        string contractsVersion = packages.GetProperty("FinalFactory.Rendezvous.Contracts").GetString()!;
        string liteNetLibVersion = matrix.GetProperty("transport").GetProperty("version").GetString()!;
        string clientFramework = XDocument.Load(Path.Combine(
                root,
                "src",
                "FinalFactory.Rendezvous.Client",
                "FinalFactory.Rendezvous.Client.csproj"))
            .Descendants("TargetFramework")
            .Single()
            .Value;
        int httpVersion = matrix.GetProperty("contracts").GetProperty("http")[0].GetInt32();
        int udpVersion = matrix.GetProperty("contracts").GetProperty("udp")[0].GetInt32();
        int ticketVersion = matrix.GetProperty("contracts").GetProperty("connectionTicket")[0].GetInt32();
        Assert.Contains(
            $"FinalFactory.Rendezvous.Client\" Version=\"{clientVersion}\"",
            guide,
            StringComparison.Ordinal);
        Assert.Contains(
            $"FinalFactory.Rendezvous.Contracts\" Version=\"{contractsVersion}\"",
            guide,
            StringComparison.Ordinal);
        Assert.Contains($"targets `{clientFramework}`", guide, StringComparison.Ordinal);
        Assert.Contains($"LiteNetLib `{liteNetLibVersion}`", guide, StringComparison.Ordinal);
        Assert.Contains(
            "https://git.finalfactory.de/api/packages/HeiKyu/nuget/index.json",
            guide,
            StringComparison.Ordinal);
        Assert.Equal(httpVersion, udpVersion);
        Assert.Equal(httpVersion, ticketVersion);
        Assert.Contains($"contract version `{httpVersion}`", guide, StringComparison.Ordinal);
    }
    [GeneratedRegex(@"rv1\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+", RegexOptions.CultureInvariant)]
    private static partial Regex ReusableCredential();
    [GeneratedRegex(@"(?<method>GET|POST) `?(?<path>/v1/operator/[a-z/-]+)`?", RegexOptions.CultureInvariant)]
    private static partial Regex OperatorRoute();
    [GeneratedRegex(@"\| `(?<method>POST) (?<path>/v1/operator/[a-z/-]+)` \| `(?<body>\{[^`]+\})` \|", RegexOptions.CultureInvariant)]
    private static partial Regex OperatorAction();
    private static string FindRepositoryRoot()
    {
        DirectoryInfo? current = new(AppContext.BaseDirectory);
        while (current is not null)
        {
            if (File.Exists(Path.Combine(current.FullName, "Rendezvous.slnx")))
            {
                return current.FullName;
            }
            current = current.Parent;
        }
        throw new InvalidOperationException("Could not locate the repository root.");
    }
 }