HeiKyu/Rendezvous
2
0
Fork 0
Code Issues 27 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add integration guides and incident runbooks (#20)
quality-gate / quality (push) Failing after 1m28s
Details
quality-gate / container (push) Has been skipped
Details

Browse Source
This commit is contained in:
KyuubiYoru
2026-07-16 18:25:10 +02:00
parent cc5793f935
commit 7fb85059fb
11 changed files with 1082 additions and 104 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/deployment/linux.md
+3
View File
@@ -177,6 +177,9 @@ dotnet build src/FinalFactory.Rendezvous.TestClient --configuration Release
For the local Compose profile, the script derives a ten-minute diagnostic
publisher credential from the ignored local key without printing either secret.
The fixed-scope helper used by the smoke can also support the manual
[TestClient local flow](../integration/test-client.md); it is deliberately not a
production issuer.
For production, do not copy the signing key to the smoke host. Instead inject a
short-lived, region-scoped credential through
`RENDEZVOUS_PUBLISHER_CREDENTIAL`, and set the external endpoints:
docs/integration/sdk-seams.md
+227
View File
@@ -0,0 +1,227 @@
# Game integration seams
Tracking: #20
Use the [TestClient start-to-finish guide](test-client.md) before integrating a
game. It proves the service and network path without engine or game code. This
page documents only the seams that the diagnostic cannot choose for a game:
package/version policy, ownership of the gameplay socket, host admission,
metadata, credential custody, and deployment compatibility.
## Packages and compatibility
Consume `FinalFactory.Rendezvous.Client` and
`FinalFactory.Rendezvous.Contracts` from the approved Gitea NuGet source and pin
both to the same exact released version. Do not use a floating version range.
The current release matrix is machine-readable in
[`compatibility.json`](../releases/compatibility.json); the same window is
available from authenticated `GET /v1/operator/status`.
The authoritative package feed is
`https://git.finalfactory.de/api/packages/HeiKyu/nuget/index.json`. Add it to the
consumer's `NuGet.config` and map only Rendezvous packages to it; retain the
consumer's existing NuGet.org mapping for other dependencies:
```xml
<packageSources>
<add key="FinalFactory" value="https://git.finalfactory.de/api/packages/HeiKyu/nuget/index.json" />
</packageSources>
<packageSourceMapping>
<packageSource key="FinalFactory">
<package pattern="FinalFactory.Rendezvous.*" />
</packageSource>
<packageSource key="nuget.org">
<package pattern="*" />
</packageSource>
</packageSourceMapping>
```
When the feed is anonymously readable, no reader credential is needed. If
registry policy requires authentication, use the platform's NuGet credential
provider or a protected per-user/CI NuGet configuration populated by the secret
manager. Never put a registry token in the project file, repository,
package-source URL, or `dotnet` command argument.
```xml
<ItemGroup>
<PackageReference Include="FinalFactory.Rendezvous.Client" Version="1.0.0" />
<PackageReference Include="FinalFactory.Rendezvous.Contracts" Version="1.0.0" />
</ItemGroup>
```
Run `dotnet restore`, then `dotnet list package --include-transitive` and verify
that Client and Contracts resolve to the same exact version and LiteNetLib to the
release matrix version before compiling the game.
Release 1.0.0 targets `netstandard2.1`, requires LiteNetLib `2.1.4`, speaks HTTP,
UDP, and connection-ticket contract version `1`, and requires an exact
tenant-configured gameplay protocol match. A package patch does not silently
change a wire version. Follow the [release and migration policy](../releases/README.md)
when changing any dimension, and validate the generated
[OpenAPI v1 document](../api/rendezvous-v1.json) rather than hand-building HTTP.
## One caller-owned gameplay socket
Create the game's LiteNetLib manager through `RendezvousNetListener`; do not open
a separate NAT socket. The game owns start, stop, and disposal. A coordinator
owns polling while it is active, so call its `Poll()` once from the game/network
thread and do not also call `NetManager.PollEvents()` during that period.
```csharp
RendezvousNetListener networkEvents = new();
NetManager gameplayNetwork = networkEvents.CreateManager();
if (!gameplayNetwork.Start(gameplayPort))
{
throw new InvalidOperationException("Gameplay UDP socket could not start.");
}
using RendezvousHostCoordinator host = new(
gameplayNetwork,
networkEvents,
mediatorEndPoint,
publishedSession,
joinClient);
host.Poll(); // call each game frame while this coordinator owns polling
```
Register normal game callbacks on `networkEvents.GameplayEvents`. Rendezvous
reserves only its authenticated direct requests and forwards other callbacks.
The same socket sends host presence, punches through the mediator, establishes
the peer, and then carries gameplay. A NAT introduction is not success; accept a
peer only after the coordinator reports the typed `Connected` outcome.
`Poll()` does not fetch new invitations. Schedule
`RefreshJoinAttemptsAsync` repeatedly for the entire hosting lifetime using a
bounded caller-owned timer (the diagnostic uses 250 ms), never allow two refreshes
to overlap, and inspect each typed result. The refresh performs HTTP work and
queues a snapshot; it does not call the LiteNetLib manager. Continue calling
`Poll()` on the manager's owning thread so the queued snapshot, presence traffic,
and callbacks are processed. Run the lease maintainer concurrently and cancel
both loops before disposing the coordinator.
For example, start one sequential refresh loop when hosting begins and await it
during shutdown:
```csharp
static async Task RefreshInvitationsAsync(
RendezvousHostCoordinator host,
CancellationToken cancellationToken)
{
using PeriodicTimer timer = new(TimeSpan.FromMilliseconds(250));
do
{
RendezvousClientResult<int> result =
await host.RefreshJoinAttemptsAsync(cancellationToken);
if (!result.IsSuccess)
{
ObserveBoundedHostRefreshFailure(result.Error);
}
}
while (await timer.WaitForNextTickAsync(cancellationToken));
}
```
On the joining side, create an attempt through `RendezvousJoinClient`, then give
the issued attempt to `RendezvousClientCoordinator` using the same manager and
listener. Cancellation, outcome reporting, bounded deadlines, fallback, and
lease-maintainer examples are in the packaged
[`FinalFactory.Rendezvous.Client` README](../../src/FinalFactory.Rendezvous.Client/README.md).
## Host admission remains game-owned
The coordinator privately validates and consumes the signed one-time connection
ticket before accepting the LiteNetLib transport request. Do not create a second
`ConnectionTicketValidator` beside it: the coordinator deliberately does not
expose the expected or presented ticket. A connected transport proves only that
Rendezvous authorized one attempt; it does not prove player identity,
entitlement, capacity, ban status, or gameplay compatibility.
Treat `AttemptCompleted` with a successful outcome and non-null `Peer` as the
start of game-owned admission. Keep that peer outside authoritative gameplay
until the game's normal authentication and admission exchange succeeds; disconnect
it on rejection or timeout:
```csharp
host.AttemptCompleted += (_, completed) =>
{
if (!completed.Outcome.IsSuccess || completed.Peer is null)
{
return;
}
BeginBoundedGameAuthentication(
completed.Peer,
onAccepted: AdmitToAuthoritativeGameplay,
onRejected: peer => peer.Disconnect());
};
```
Revoke an attempt when the game cancels it. Never log a ticket or capability. A
successful Rendezvous check must not bypass the game's authentication or
authoritative server rules. `ConnectionTicketValidator` is a lower-level
primitive for a custom transport integration that owns the complete request
acceptance path; it is not an extra gate for `RendezvousHostCoordinator`.
## Provision each game and environment
Provision game/environment scope before issuing credentials. The policy fixes
enabled regions, exact gameplay protocols, visibility and publisher trust modes,
metadata schema and byte budgets, quotas, and whether a dedicated fallback may
be published. Unknown or disabled scope fails closed. Follow
[game provisioning and signing-key lifecycle](../security/provisioning.md) for
the complete schema, principal kinds, secret providers, overlap, and revocation.
Dedicated publisher credentials belong only on trusted hosting infrastructure.
Never ship one in a player build, repository, image layer, appsettings file, URL,
argument, log, crash report, or analytics event. Issue a short-lived credential
scoped to one game/environment and its allowed regions from the trusted
deployment boundary. Player-host grants are issued to an authenticated player
session at runtime and are never embedded in the build. Player-host grants,
dedicated publishers, anonymous unlisted hosts, and operators are separate
principal kinds; do not interchange them.
Rotate signing keys with an overlap:
1. install a new authorized key inside its `NotBefore`/`SignUntil` window;
2. begin issuing with it while the old key remains verify-only;
3. wait at least the maximum credential lifetime plus allowed clock skew;
4. retire the old verifier after `VerifyUntil` and preserve custody records.
A suspected compromise is not routine rotation: stop issuance, revoke the exact
key through the protected operator route, remove or replace it in provisioning,
invalidate affected credentials, and follow the
[key-compromise runbook](../operations/incident-runbooks.md#signing-key-or-issuer-compromise).
## Metadata is public and policy-owned
Treat listing metadata as untrusted public input. Define a small allowlist in
each provisioned game's `MetadataValueMaxBytes`, set `RequiredMetadataKeys`, and
keep `MetadataMaxKeys` and `MetadataMaxBytes` to the smallest useful values. Values
must be display data only—for example a bounded map or ruleset identifier. Never
publish player identity, free-form chat, secrets, access tokens, internal
addresses, world state, or data needed for authoritative gameplay.
The platform contract caps metadata at 32 keys, 256 UTF-8 bytes per value, and
4096 encoded bytes total; tenant policy can and should be smaller. Build version
and display name are separately bounded public fields. Games must escape metadata
for their UI and must not infer trust from a listing being present.
## Local, staging, and production path
Use the checked-in Compose profile only for the local TestClient guide. For a
real environment:
1. provision the game/environment policy and externally held signing keys;
2. deploy one active service behind the source-preserving HTTPS/UDP topology in
[secure single-active Linux deployment](../deployment/linux.md);
3. install matching exact package versions in the game and set its service and
mediator endpoints through environment-specific configuration;
4. pass the TestClient health/publish/browse/punch/direct-traffic smoke using a
short-lived diagnostic credential;
5. run the topology harness and representative consumer-network trials; and
6. monitor typed outcomes and bounded metrics before broad rollout.
Rendezvous v1 has no relay, account system, matchmaking engine, server-browser
UI, gameplay authority, or durable session database. A game owns player-facing
recovery and an explicit fallback. Do not describe direct traversal as guaranteed.
docs/integration/test-client.md
+193 -68
View File
@@ -1,97 +1,222 @@
# Diagnostic TestClient integration guide
# Start-to-finish TestClient guide
Tracking: #25
Tracking: #20, #25
`FinalFactory.Rendezvous.TestClient` is the smallest supported public-SDK consumer.
It exists for integration development, CI smoke checks, deployment verification,
and operator diagnosis. It is intentionally not a production game client, game
server, matchmaking UI, or relay.
`FinalFactory.Rendezvous.TestClient` is the supported executable proof that a
consumer can publish, browse, authorize, punch, connect, exchange direct traffic,
and diagnose a failure using only the public Client and Contracts packages. It is
intentionally thin: a polished server browser and player-facing connection UI
belong in each game repository.
The automated scenario matrix, privileged Linux namespace run, and topology
limitations are documented in the [deterministic topology harness](topology-harness.md).
> **Traversal boundary:** Rendezvous v1 is not a relay and cannot guarantee a
> connection through symmetric NAT, carrier-grade NAT, restrictive firewalls,
> VPNs, or platform policy. It provides no accounts, social system, skill-based
> matchmaking, gameplay server, gameplay authority, or gameplay transport.
## Prerequisites
The automated scenario matrix, privileged Linux namespace run, and simulation
limits are in the [deterministic topology harness](topology-harness.md). The
[SDK seam guide](sdk-seams.md) covers the few integration details that this
executable cannot show.
Start a configured Rendezvous service and note both its HTTP base URL and UDP
mediator endpoint. The host needs a tenant-scoped publisher credential from the
deployment secret boundary. Put it in an environment variable and pass only that
variable's name when the default is unsuitable:
## First local connection from a clean checkout
Prerequisites are the pinned .NET SDK, Docker with Compose, OpenSSL, Python 3,
`curl`, and `jq`. Run these commands from the repository root. The generated key
and credential are disposable local fixtures, not production provisioning.
```bash
export RENDEZVOUS_PUBLISHER_CREDENTIAL='<deployment-supplied value>'
install -d -m 0700 deploy/compose/secrets
umask 077
openssl rand -out deploy/compose/secrets/signing-key 32
export RENDEZVOUS_UID="$(id -u)"
export RENDEZVOUS_GID="$(id -g)"
test "$RENDEZVOUS_UID" -ne 0
docker compose -f deploy/compose/compose.yaml up --build --detach
ready=false
for attempt in {1..45}; do
if curl --fail --silent http://127.0.0.1:8080/health/ready >/dev/null; then
ready=true
break
fi
sleep 1
done
test "$ready" = true
curl --fail http://127.0.0.1:8080/health/live
curl --fail http://127.0.0.1:8080/health/ready
dotnet build src/FinalFactory.Rendezvous.TestClient --configuration Release
```
Never put the credential in a command argument, URL, checked-in configuration,
shell trace, or captured test fixture. The development server's signing material
is process-ephemeral; credentials from a prior development process are invalid.
## Manual three-terminal flow
Start the host:
Only the host terminal needs a publisher credential. Disable shell tracing before
capturing it; the helper prints the credential on stdout so command substitution
can place it directly in the environment without writing it to disk.
```bash
dotnet run --project src/FinalFactory.Rendezvous.TestClient -- \
host --service http://127.0.0.1:5000/ --mediator 127.0.0.1:9050 \
--game space-game --environment development --region local --protocol 1
set +x
export RENDEZVOUS_PUBLISHER_CREDENTIAL="$(./scripts/mint-local-publisher-credential.sh)"
```
Browse from another terminal:
The helper accepts no arguments, reads the ignored `0600` local Compose key, and
mints only `space-game` / `smoke` / `local` / protocol `1` for ten minutes. It is
not a reusable issuer or an example for production. Never put the result in a
command argument, URL, shell history, log, screenshot, support ticket, captured
fixture, or source file.
In terminal 1, publish a host. It stays alive for at most 60 seconds and exits
after a joining peer completes the authenticated echo exchange:
```bash
dotnet run --project src/FinalFactory.Rendezvous.TestClient -- \
browse --service http://127.0.0.1:5000/ \
--game space-game --environment development --region local --protocol 1
dotnet run --project src/FinalFactory.Rendezvous.TestClient \
--configuration Release --no-build -- \
host --service http://127.0.0.1:8080/ --mediator 127.0.0.1:9050 \
--game space-game --environment smoke --region local --protocol 1 \
--display-name "Local diagnostic" --timeout-seconds 60 --run-seconds 60 \
--exit-after-echo
```
Join from a third terminal. Omit `--listing` for an interactive choice:
Copy the public listing ID printed by the host, or discover it from terminal 2:
```bash
dotnet run --project src/FinalFactory.Rendezvous.TestClient -- \
join --service http://127.0.0.1:5000/ --mediator 127.0.0.1:9050 \
--game space-game --environment development --region local --protocol 1 \
--listing 00000000-0000-0000-0000-000000000000
dotnet run --project src/FinalFactory.Rendezvous.TestClient \
--configuration Release --no-build -- \
browse --service http://127.0.0.1:8080/ \
--game space-game --environment smoke --region local --protocol 1
```
Replace the sample UUID with the public listing ID printed by host or browse.
Host and join each create one caller-owned LiteNetLib manager. That same socket
sends presence/punch traffic, establishes the authenticated direct connection,
and carries the ping/echo/ack/completion payload. The final completion confirms
that the host received the reliable acknowledgement; none of this traffic passes through the HTTP
service or UDP mediator.
In terminal 3, either omit `--listing` and select interactively, or provide the
copied ID for deterministic selection:
## CI and deployment smoke flow
```bash
dotnet run --project src/FinalFactory.Rendezvous.TestClient \
--configuration Release --no-build -- \
join --service http://127.0.0.1:8080/ --mediator 127.0.0.1:9050 \
--game space-game --environment smoke --region local --protocol 1 \
--listing REPLACE_WITH_LISTING_UUID --timeout-seconds 30
```
Use `--script --json`, set `--listing` when deterministic selection matters, and
check the documented process exit code. `--timeout-seconds` bounds each startup,
traversal, or direct-traffic stage; a script host also uses it as its total runtime
unless `--run-seconds` is explicit. A host can add `--exit-after-echo` so it
terminates after the joining peer acknowledges direct traffic and receives the
host's completion confirmation. Every wait is
bounded by coordinator state and `--timeout-seconds`; no orchestration should use
an unbounded sleep.
Success means the joiner prints `join.connected` and verified direct traffic,
and the host prints verified direct traffic before deregistering. The host and
joiner each create one caller-owned LiteNetLib manager. The same UDP socket sends
presence and punch traffic, accepts the authenticated peer, and carries the
ping/echo/ack/completion payload; direct traffic does not pass through the HTTP
service or mediator.
The normal test suite contains a real process gate that starts the built Server,
host TestClient, and join TestClient, waits for readiness and versioned events,
and verifies direct traffic, cleanup, JSON shape, and secret canaries. Process
trees are force-terminated in the test cleanup path if normal shutdown fails.
Clean up secrets and the disposable service when finished:
Useful success events are:
```bash
unset RENDEZVOUS_PUBLISHER_CREDENTIAL
docker compose -f deploy/compose/compose.yaml down
rm deploy/compose/secrets/signing-key
```
- `host.registered`, `host.ready`, `host.direct-traffic`, and `host.deregistered`;
- `browse.completed` and `browse.session`; and
- `join.connected`, `join.direct-traffic`, and `join.outcome-report`.
## Script and JSON automation
Failure events preserve stable typed phases and outcomes. When a terminal outcome
contains a configured dedicated endpoint, `join.fallback` reports `available`
with endpoint type `dedicated`; no raw address is printed and no fallback is
started implicitly.
`--script` forbids prompts and selects the first compatible listing unless
`--listing UUID` fixes the choice. `--json` emits one JSON object per line with
`version: 1`. New optional properties may be added, but event names and exit
codes are stable automation contracts. Informational events use stdout and
failures use stderr.
## What the proof does and does not establish
The deployment smoke performs the full health, publish, join, mediation, direct
traffic, outcome-report, and cleanup flow using bounded waits:
The deterministic loopback test proves the complete service/host/client protocol,
ticket admission, and peer-to-peer payload path. Loopback is not evidence that all
consumer routers, carrier-grade NATs, symmetric NATs, firewalls, VPNs, IPv6 paths,
or platform policies permit hole punching. Same-LAN, separated observed endpoints,
network namespaces/containers, mediator restart, and adverse topology coverage
belong to the topology harness tracked by #14. Production rollout still requires
tests from representative networks and a game-owned fallback policy.
```bash
dotnet build src/FinalFactory.Rendezvous.TestClient --configuration Release
./scripts/smoke-deployment.sh
```
For custom automation, capture JSON and preserve the process status separately:
```bash
set +e
dotnet run --project src/FinalFactory.Rendezvous.TestClient \
--configuration Release --no-build -- \
browse --service http://127.0.0.1:8080/ \
--game space-game --environment smoke --region local --protocol 1 \
--script --json >browse.jsonl
status=$?
set -e
jq -e 'select(.version == 1 and .event == "browse.completed")' browse.jsonl
test "$status" -eq 0
```
Never use an unbounded sleep to orchestrate processes. Wait for versioned events
such as `host.ready` and apply a deadline. Useful success events are
`host.registered`, `host.ready`, `host.direct-traffic`, `host.deregistered`,
`browse.completed`, `browse.session`, `join.connected`, `join.direct-traffic`,
and `join.outcome-report`.
| Exit | Meaning |
| ---: | --- |
| `0` | Requested diagnostic flow completed successfully |
| `2` | Invalid command or options |
| `3` | Missing or invalid local configuration |
| `10` | HTTP, registration, browser, lease, or socket failure |
| `11` | No compatible session was available or selected |
| `12` | Authorization or traversal reached a typed terminal failure |
| `13` | Direct connection succeeded but the direct traffic proof failed |
| `130` | Caller cancellation or Ctrl+C |
## Observe a safe failure
Run this after the protocol-1 browse in terminal 2 and before the terminal-3
join (or restart terminal 1 first). The preceding browse proves that one
protocol-1 host is present. Now browse for deliberately incompatible protocol
`999`. The command emits a successful directory response with
`browse.completed`, `count: 0`, then exits `11` to distinguish compatibility
from a service outage:
```bash
set +e
dotnet run --project src/FinalFactory.Rendezvous.TestClient \
--configuration Release --no-build -- \
browse --service http://127.0.0.1:8080/ \
--game space-game --environment smoke --region local --protocol 999 \
--script --json >incompatible.jsonl
status=$?
set -e
jq -e 'select(.event == "browse.completed" and .phase == "directory" and .count == 0)' \
incompatible.jsonl
test "$status" -eq 11
```
This is a diagnostic failure drill, not a bypass: unknown tenant scope and
protocols still fail closed, and the local helper cannot mint a credential for
them.
## Diagnose by phase, not by guesswork
Start with the exit code, then the last versioned event and its `phase`, `status`,
and typed `outcome`. Endpoint categories may be
reported as `loopback`, `private`, or `public`; raw endpoints, credentials,
capabilities, metadata, and player identities are never emitted.
| Symptom or last event | Distinction | Check next |
| --- | --- | --- |
| `host.configuration`, exit `3` | Local credential variable is missing or malformed before any request | Confirm the named environment variable exists, tracing is off, and the credential has not expired |
| `host.registration`, exit `10` | Publisher authentication, tenant policy, metadata, quota, or HTTP failure | Use the typed status; compare credential scope with game/environment/region and the provisioned policy, then correlate protected server telemetry by operation and time |
| `browse.sessions`, exit `10` | Directory request failed | Check HTTP reachability, `/health/ready`, rate limiting, and contract compatibility |
| `browse.completed` count `0`, or `join.selection` empty, exit `11` | Healthy directory but no compatible visible listing | Match game, environment, region, and exact gameplay protocol; then confirm a host lease is still active |
| Exact `join.selection` failure, exit `10` | Listing disappeared, is hidden, or scope no longer matches | Browse again; do not retry an old listing ID forever |
| `join.authorization`, exit `12` | Service rejected the attempt before NAT traversal | Inspect typed category/outcome for policy, capacity, stale host, or active-attempt limits |
| `join.punch` / `join.traversal`, exit `12` | Mediation or NAT traversal did not establish a peer | Confirm UDP endpoint/reply path, host presence, clocks, firewall/NAT behavior, and topology; use a game-owned fallback if policy supplies one |
| `join.direct-connect`, exit `12` | Introduction occurred but authenticated direct admission failed | Confirm host is polling the same socket, the one-time ticket is current, and game admission did not reject capacity, identity, or bans |
| `join.connected` followed by exit `13` | Peer connected but the direct gameplay-like echo did not finish | Inspect the peer lifecycle and caller polling; this is not an HTTP/directory failure |
Stopping a host without deregistration may leave its listing visible only until
the bounded lease expires. During that window, a join can produce a typed stale
host or traversal outcome; it must not be interpreted as a healthy host. Restarting
the single-active service intentionally loses all ephemeral listings and attempts,
so hosts re-register and clients browse again.
If a terminal outcome reports an authoritative dedicated fallback,
`join.fallback` exposes only availability and endpoint type. TestClient never
connects to it automatically. The game owns the decision, authentication, and
connection policy. If no fallback is present, Rendezvous v1 offers no relay.
## Production use
Do not copy a production signing key to a diagnostic host. Supply a short-lived,
least-scope publisher credential from the deployment secret boundary and set the
external service, mediator, and matching scope variables described in the
[secure Linux deployment smoke](../deployment/linux.md#http-and-udp-smoke).
Run representative external-network tests; loopback success is not NAT coverage.
docs/operations/incident-runbooks.md
+339
View File
@@ -0,0 +1,339 @@
# Incident and change runbooks
Tracking: #20
These runbooks supplement the [signal and operator reference](observability-and-operator-runbook.md).
Every procedure has four explicit gates: detect, contain, recover, and verify.
Record timestamps, the release digest, bounded aggregates, audit fingerprints,
and `X-Rendezvous-Correlation-ID` values. Never copy credentials, capabilities,
connection tickets, signing material, player identity, raw IP addresses,
endpoints, listing metadata, or full request bodies into an incident record.
Operator routes must be reachable only from an allowed management source. Use a
short-lived, least-permission operator credential minted outside Rendezvous.
Pass it to an approved operator client through protected stdin or a secret agent,
not a URL, command argument, environment-wide process launcher, shell trace, or
ticket. All request shapes and responses are defined by the generated
[OpenAPI v1 document](../api/rendezvous-v1.json).
Before an incident, keep these protected records available without depending on
the affected service: current and previous image digests, matching configuration,
key IDs and lifecycle windows (not raw key values), the game owner/on-call map,
capacity baselines, collector destinations, and a separately authorized
break-glass operator key. Test management-source allowlisting and credential
permissions at least once per release.
Use the exact versioned action shapes below. Confirmation fields deliberately
repeat the target so a stale UI selection or copy error fails closed. Responses
do not echo targets.
| Operation | JSON body |
| --- | --- |
| `POST /v1/operator/listings/revoke` | `{"listingId":"<uuid>","confirmListingId":"<same uuid>"}` |
| `POST /v1/operator/principals/revoke` | `{"subject":"<exact subject>","confirmSubject":"<same subject>","lifetimeSeconds":60}` |
| `POST /v1/operator/keys/revoke` | `{"keyId":"<key id>","confirmKeyId":"<same key id>"}` |
| `POST /v1/operator/drain` | `{"confirmation":"DRAIN"}` |
## Abuse or authentication spike
### Detect
- Alert on a baseline-relative increase in `rendezvous.limiter.drops`, HTTP/UDP
request rate, `rendezvous.operator.authentication` rejected/forbidden results,
registration requests by authentication status, queue depth, or p95/p99 latency.
- Check `/health/live`, `/health/ready`, `rendezvous.store.available`, and
authenticated `GET /v1/operator/status`. Separate public-source rejection,
publisher credential failure, operator probing, and ordinary capacity growth.
- Use only bounded operation/result dimensions and correlation IDs. Do not group
by raw address, token, subject, listing ID, or metadata.
### Contain
- Preserve the dedicated operator partition. Do not raise public limits during
an active spike. Apply source-preserving edge rate controls only when their
collateral effect is understood and UDP source address/port remains intact.
- For one abusive session, call `POST /v1/operator/listings/revoke` with identical
`listingId` and `confirmListingId`. For a confirmed publisher subject, call
`POST /v1/operator/principals/revoke` with identical `subject` and
`confirmSubject` and a 1–600 second lifetime.
- Revoke a signing key only when compromise evidence implicates that issuer;
broad key revocation invalidates every credential signed by it. Drain only if
the process itself must be isolated.
### Recover
- Correct the source integration, edge rule, leaked principal grant, or tenant
budget under change control. Let a bounded principal revocation expire only
after the owner confirms remediation; a repeated shorter revocation never
shortens the original deadline.
- Restore normal limits gradually. If saturation caused state churn, allow leases
and attempts to expire naturally rather than deleting arbitrary state.
### Verify
- Require limiter drops, authentication result ratios, queue depth, latency, and
direct-connect outcomes to return to the same-region baseline for the agreed
observation window.
- Confirm readiness stayed healthy or recovered, operator audit contains the
intended action/result fingerprint, revoked resources cannot create new work,
and unaffected tenants can still publish, browse, and connect.
## Signing key or issuer compromise
### Detect
- Treat secret-manager access alerts, unexpected issuance, credentials outside
the expected region/kind, a signing-key expiry alarm, or unexplained publisher
authentication growth as compromise until disproved.
- Identify the non-secret key ID, allowed credential kinds, game/environment
binding, `NotBefore`, `SignUntil`, and `VerifyUntil`. Do not retrieve or paste
raw material merely to compare it.
### Contain
- Stop the affected external issuer and deny further access to its secret.
- From a separate uncompromised break-glass operator key with `RotateKeys`, call
`POST /v1/operator/keys/revoke` with identical `keyId` and `confirmKeyId`.
Runtime revocation is immediate but process-local.
- Remove or mark the key revoked in authoritative provisioning before any
restart. Revoke affected principals/listings when narrower evidence supports
it. Do not drain automatically unless the running instance cannot be trusted.
### Recover
- Generate replacement material in the approved secret boundary, use a new key
ID, bind it to the exact credential kind and tenant, and deploy configuration
referencing the secret—never the secret value.
- Resume issuance with short lifetimes. Reissue only to authenticated workloads.
When confidentiality is lost, do not use normal overlap to keep compromised
credentials valid; document the intentional invalidation window.
- Rotate any release, registry, or operator credential exposed by the same
incident through its owning system; Rendezvous key revocation cannot revoke
unrelated systems.
### Verify
- Confirm `GET /v1/operator/status` shows the compromised key revoked and the
replacement signing, old credentials fail, new exact-scope credentials work,
and the result survives a controlled restart from updated provisioning.
- Pass TestClient registration, browse, authenticated mediation, and direct
traffic with the replacement; monitor authentication and audit results through
at least the maximum newly issued credential lifetime.
## Targeted listing or publisher revocation
### Detect
- Validate the abuse report against game-owned records and bounded Rendezvous
evidence. Determine whether the target is one listing or an authenticated
publisher subject. Do not use display name, metadata, or a raw address as
identity.
- Confirm current aggregate state through `GET /v1/operator/status` and record
the correlation IDs that justified action.
### Contain
- Revoke one listing with `POST /v1/operator/listings/revoke`; the exact listing
UUID must appear in both confirmation fields.
- Revoke a publisher with `POST /v1/operator/principals/revoke`; the exact subject
must appear in both confirmation fields and `lifetimeSeconds` must be 1–600.
This removes that principal's active listings and attempts and blocks new ones
for the bounded lifetime.
- Choose the narrowest action. Do not revoke a tenant key for a single listing.
### Recover
- The game owner resolves the ban, account, workload, or configuration issue in
the authoritative game system. Rendezvous does not own user accounts or bans.
- After the original revocation deadline, permit a newly authenticated publisher
to register. There is no un-revoke endpoint and no recovery of removed
ephemeral listings; the host creates a new listing.
### Verify
- Confirm the old listing is no longer browsable or joinable, the principal
cannot publish during its lifetime, and the audit action/result is present
without the raw target.
- Confirm unrelated publishers in the same tenant and another tenant still pass
publish/browse/join/direct-traffic checks.
## Planned restart or crash recovery
### Detect
- Planned restart begins with a recorded change and a healthy current baseline.
Crash recovery begins when liveness/process state fails or both TCP 8080 and
UDP 9050 stop answering. Distinguish dependency/readiness failure from a dead
process; liveness deliberately remains healthy for some recoverable failures.
- Record active listing/lease/attempt aggregates. They are informational only:
v1 has no durable runtime database to restore.
### Contain
- For a planned stop, call `POST /v1/operator/drain` with confirmation exactly
`DRAIN`. Require readiness `503`, liveness `200`, and removal from new traffic.
Allow the bounded drain deadline to finish, then send SIGTERM.
- Never start a second active instance while the old process owns the advertised
HTTP/UDP endpoints. On crash, fence the old process/host and verify both sockets
are released before replacement.
### Recover
- Start exactly one instance from the recorded immutable image digest and matching
reviewed configuration/key references. A restart intentionally loses listings,
observed endpoints, attempts, replay markers, and runtime-only revocations.
- Ensure any emergency key revocation is also present in authoritative
provisioning. Hosts must re-register; clients must browse and start new
attempts. Do not restore stale ephemeral state from logs or backups.
### Verify
- Require live and ready health, UDP bind, store availability, and one active
target. Run the full deployment smoke and confirm host re-registration begins.
- Verify no pre-restart listing or capability is accepted, runtime revocations
that should persist are configuration-backed, and latency/outcomes stabilize.
## Release rollback
### Detect
- Trigger rollback from a predeclared objective: readiness loss, failed deployment
smoke, contract/package incompatibility, security regression, direct-success
regression beyond threshold, or sustained resource regression. Record the new
and previous digests and the evidence; do not move a tag.
### Contain
- Stop promotion and new rollout work. Drain and stop the faulty single active
instance, then verify both public sockets are released. Revoke affected keys or
principals only when the defect creates an authorization risk.
- Preserve logs, artifacts, provenance, signatures, and the faulty release record.
Never overwrite or delete an immutable package/image to reuse its version.
### Recover
- Deploy the previous known-good image by digest with its compatible configuration
and key set. Do not run old and new concurrently. If configuration changed,
apply its reviewed down-migration before starting.
- Publish a corrected build under a new SemVer after diagnosis; mark faulty release
notes withdrawn when appropriate.
### Verify
- Check the running image digest, live/ready health, one active target, UDP source
preservation, and the complete TestClient deployment smoke.
- Confirm package/server compatibility from `GET /v1/operator/status`, hosts
re-register, and the rollback objective returns to baseline for the observation
window.
## Capacity saturation
### Detect
- Page when `rendezvous.queue.depth` remains above 90% of the configured attempt
limit, lease-critical work is shed, `rendezvous.store.available` is zero, or no
ready instance remains. Warn at 70%, sustained `rendezvous.limiter.drops`, or
p95 latency above objective.
- Compare CPU, memory, file descriptors, UDP errors, expiry churn, HTTP operation
rate, and typed connection outcomes with the measured
[capacity profile](capacity-and-resilience.md). Distinguish legitimate growth,
attack traffic, downstream telemetry pressure, and a regression.
### Contain
- Preserve lease-critical and operator reserves. Shed new browse/join work with
the existing typed `429`/`Retry-After` behavior; do not add an unbounded queue.
- Apply per-tenant/source controls at the appropriate trusted boundary. If the
process is unstable, drain new work and recover on one replacement rather than
adding a second active replica; v1 state is process-local.
### Recover
- Remove the causal load or deploy a tested higher single-instance resource and
budget profile. Change CPU/memory and server limits together, using the numeric
gate and accelerated soak before production.
- Long-term horizontal scaling requires a designed shared directory, replay, and
attempt authority. A generic load balancer is not that design.
### Verify
- Re-run the capacity/resilience gate at the chosen profile, then require queue,
limiter drops, expiry churn, latency, store health, and direct-success ratio to
remain within objectives through the production observation window.
- Confirm termination still completes within `DrainDeadlineSeconds + 5` and the
public and operator partitions behave independently.
## Privacy or telemetry incident
### Detect
- Trigger on any credential, token, capability, player identity, raw IP/endpoint,
listing ID, metadata, or caller-reported exact connection duration tied to an
event or identity found in logs, metrics, traces, crash reports, support systems,
or analytics. Aggregate HTTP/UDP duration histograms with bounded operation tags
are expected telemetry. Also trigger when audit data exceeds its approved 30-day
retention without an incident hold.
- Identify the producing version, sink, access population, retention/replication
path, and time window without copying the exposed value into a new system.
### Contain
- Stop or filter the offending export and restrict access to affected sinks.
Preserve the minimum evidence under the incident process; do not take broad
diagnostic dumps that amplify exposure.
- Revoke exposed reusable credentials/keys through their owning boundary. Listing
IDs and endpoints are not authentication secrets, but remove affected listings
if continued exposure creates risk. Notify privacy/security owners according to
applicable policy and law.
### Recover
- Patch the producer to the allowlisted telemetry model, test canary redaction
across logs/metrics/traces/output, and deploy through the immutable release
path. Delete or age out affected data from every sink according to approved
retention and legal-hold direction.
- Replace exposed credentials and re-register hosts when necessary. Do not claim
that a service restart deletes copies already exported to collectors.
### Verify
- Search new telemetry using non-secret synthetic canaries and confirm no canary
or prohibited field crosses the boundary. Verify audit records contain only
fixed fields and fingerprints and that retention/eviction is operating.
- Security/privacy owners confirm sink cleanup, access review, notification, and
monitoring closure before the incident is resolved.
## Dependency or base-image upgrade
### Detect
- Open a reviewed change for an advisory, end-of-support date, pinned-digest
refresh, or planned package update. Record affected package/image, current and
proposed exact version/digest, advisory severity, exploitability, and required
deadline. Never float to `latest` as remediation.
### Contain
- For an actively exploited critical issue, restrict exposure or stop the service
under incident authority while building the fix. Revoking publisher keys does
not repair a vulnerable runtime. Otherwise keep the known-good release running
while the candidate is tested.
### Recover
- Update the SDK/base-image digest, lock files, license/advisory evidence, SBOM,
compatibility matrix, and release notes together. For LiteNetLib or a wire/API
change, apply the explicit version/migration policy rather than silently
replacing compatible bytes.
- Run locked restore, formatting, Debug and Release builds/tests, public contract
and package gates, real consumer restores, reproducible artifact/image builds,
vulnerability scan, signatures, topology/deployment smoke, and capacity checks
proportional to the change. Promote the exact tested digest.
### Verify
- Verify signatures, provenance, checksums, SBOM contents, running digest, and
absence of the advisory in the shipped artifact—not merely the build host.
- Require live/ready health, TestClient direct traffic, real consumer compatibility,
and normal latency/outcomes. Keep the previous digest and compatible config for
rollback until the observation window closes.
docs/operations/observability-and-operator-runbook.md
+5 -1
View File
@@ -1,4 +1,4 @@
# Observability and operator runbook
# Observability and operator reference
This runbook defines the production signals and privileged controls for the
Rendezvous service. The service emits `System.Diagnostics.Metrics` instruments
@@ -6,6 +6,10 @@ from the `FinalFactory.Rendezvous` meter and distributed-tracing activities from
`FinalFactory.Rendezvous.Server`. Connect those sources to the deployment's
OpenTelemetry or equivalent collector. Do not add identifiers to metric labels.
Concrete detect/contain/recover/verify procedures for abuse, key compromise,
targeted revocation, restart, rollback, saturation, privacy incidents, and
dependency upgrades are in the [incident and change runbooks](incident-runbooks.md).
## Health and readiness
- `GET /health/live` proves that the HTTP process can answer. It deliberately