Rendezvous/docs/operations/production-readiness.md

Production-readiness decision and real-network canary

Tracking: #23

Rendezvous v1 is not production-ready until every required gate in production-readiness-v1.json is recorded as pass. The machine-checkable decision is intentionally fail-closed:

./scripts/check-production-readiness.sh

Exit 0 means every required gate is present and passing, exit 3 means the record is valid but at least one gate is pending or failed, and exit 2 means the record itself is malformed or contains identifier-, endpoint-, account-, or credential-shaped data. Editing only the top-level decision cannot make the check pass.

The checked-in record is an index, not a log archive. It contains one repository-relative evidence reference and a short categorical note per gate. Raw packet captures, client event streams, publisher credentials, public or private network endpoints, listing IDs, and player/account identifiers must not be committed.

Required decision matrix

The local matrix covers immutable artifacts, Debug and Release verification, both real game consumers, the candidate capacity/resilience profile, production-process recovery, and the combined security/privacy/observability gate. These may be reproduced by the project team on a clean candidate commit.

The external matrix remains distinct because a local namespace, loopback, container bridge, or second process on one machine cannot prove it:

Gate	Required evidence
Public package empty-cache restore	A clean machine restores the exact Client and Contracts version using only the documented public sources.
Signed publication	The immutable tag publishes packages, image digest, SBOMs, provenance, checksums, and verifiable signatures through the protected release workflow.
Source-preserving UDP ingress	Packet capture on the service host proves the mediator observes each peer's real source tuple and replies from the advertised public tuple; no UDP proxy rewrites either direction.
Same-LAN direct canary	Two independently operated game clients establish authenticated direct LiteNetLib traffic.
Home-NAT direct canary	Host and joiner on distinct residential networks establish authenticated direct LiteNetLib traffic.
Restrictive/CGNAT and blocked-UDP canaries	Each bounded join exits 12, records a typed terminal category, and exposes the game-owned fallback policy without hanging or claiming success.
IPv6 direct canary	Two external IPv6 clients record authenticated direct traffic and an observed ipv6 peer address family.
Public rate-shaped capacity	The documented HTTP/UDP workload mix meets its objectives through TLS, Kestrel, JSON, LiteNetLib, kernel sockets, and public ingress.
One-hour endurance	The immutable production-shaped candidate completes the one-hour profile without a state, handle, memory, readiness, or latency failure.
Alert delivery	A real alert sink receives both trigger and recovery notifications for the rehearsed outage.
Cold-standby rollback	Drain, stop, socket release, replacement start, host re-registration, and rollback meet the process and host-visible recovery objectives.
Documentation-only exercise	An operator who did not author the runbooks completes key rotation/revocation, outage, restart, re-registration, and rollback using only the checked-in documentation.

Failure or missing evidence is blocking. It is never converted into an accepted risk by changing the wording of the readiness note.

When an external gate passes, add a redacted repository JSON attestation and point that gate's evidenceRef to it. The checker requires this exact shape and binds the gate to the evaluated candidate commit. artifactDigest is the SHA-256 of the protected evidence bundle or public release record, not a peer endpoint, listing identifier, account identifier, or credential:

{
  "schemaVersion": 1,
  "kind": "rendezvous-external-gate-attestation",
  "gateId": "replace-with-the-exact-gate-id",
  "candidateCommit": "replace-with-the-40-character-candidate-commit",
  "result": "pass",
  "performedAtUtc": "2026-01-01T00:00:00Z",
  "artifactDigest": "replace-with-the-64-character-sha256",
  "evidenceLocation": "protected-operations-record",
  "reviewerRole": "independent-operator"
}

Allowed evidence locations are protected-operations-record and public-release-record. Allowed reviewer roles are release-operator, network-operator, security-operator, and independent-operator. The checker rejects a missing file, wrong gate, wrong candidate, malformed digest, naive timestamp, extra fields, or sensitive-data-shaped contents.

Prepare one immutable canary build

Use the exact release candidate on every canary machine. Verify a clean checkout, restore in locked mode, and build the TestClient before changing networks:

test -z "$(git status --porcelain)"
dotnet restore Rendezvous.slnx --locked-mode
dotnet build Rendezvous.slnx --configuration Release --no-restore

Keep shell tracing disabled. The host receives a short-lived, least-scope publisher credential through RENDEZVOUS_PUBLISHER_CREDENTIAL; it must never be put in an argument, coordination file, evidence file, command transcript, or support message. Set the public HTTPS service URL and advertised UDP mediator tuple separately. TestClient rejects credentials embedded in the service URL.

Run a success canary across two machines

On the host machine, choose same-lan, home-nat, or ipv6-direct. The coordination file is mode 0600 and contains only the temporary listing UUID. It is not evidence; transfer it through an approved private channel, then delete both copies.

set +x
export RENDEZVOUS_PUBLISHER_CREDENTIAL='supplied-by-the-approved-secret-boundary'
export RENDEZVOUS_CANARY_ROLE=host
export RENDEZVOUS_CANARY_TOPOLOGY=home-nat
export RENDEZVOUS_CANARY_ADDRESS_FAMILY=ipv4
export RENDEZVOUS_CANARY_HTTP_URL='https://service.example.invalid/'
export RENDEZVOUS_CANARY_UDP_ENDPOINT='203.0.113.10:9050'
export RENDEZVOUS_CANARY_COORDINATION_FILE="$HOME/.local/state/rendezvous-canary-listing"
export RENDEZVOUS_CANARY_OUTPUT="$PWD/artifacts/canary/home-nat-host.json"
./scripts/run-real-network-canary.sh

The host prints only that it is ready and waits for the authenticated exchange. On the joiner, read the securely transferred UUID without placing it in shell history and run the matching topology:

set +x
read -r RENDEZVOUS_CANARY_LISTING_ID < "$HOME/.local/state/rendezvous-canary-listing"
export RENDEZVOUS_CANARY_LISTING_ID
export RENDEZVOUS_CANARY_ROLE=client-success
export RENDEZVOUS_CANARY_TOPOLOGY=home-nat
export RENDEZVOUS_CANARY_ADDRESS_FAMILY=ipv4
export RENDEZVOUS_CANARY_HTTP_URL='https://service.example.invalid/'
export RENDEZVOUS_CANARY_UDP_ENDPOINT='203.0.113.10:9050'
export RENDEZVOUS_CANARY_OUTPUT="$PWD/artifacts/canary/home-nat-client.json"
./scripts/run-real-network-canary.sh
unset RENDEZVOUS_CANARY_LISTING_ID

The host summary requires authenticated direct traffic and deregistration. The client summary requires connection, authenticated direct traffic, accepted outcome reporting, and the declared address family observed on the actual peer. The summaries deliberately contain no network tuple or listing identifier.

For IPv6, set the topology to ipv6-direct, the family to ipv6, and use the deployment's bracketed IPv6 mediator form. Record unsupported operating systems, console platforms, VPNs, and address families as untested; an IPv4 pass is not evidence for IPv6 or a platform network policy.

Run a bounded failure canary

Start the host from an independently reachable network as above. On the joiner, apply the reviewed firewall rule that blocks the relevant UDP path, or use the known restrictive carrier network, then set client-expected-failure and the matching topology:

export RENDEZVOUS_CANARY_ROLE=client-expected-failure
export RENDEZVOUS_CANARY_TOPOLOGY=firewall-blocked-udp
export RENDEZVOUS_CANARY_ADDRESS_FAMILY=ipv4
export RENDEZVOUS_CANARY_OUTPUT="$PWD/artifacts/canary/firewall-blocked-client.json"
./scripts/run-real-network-canary.sh

This role passes only when TestClient exits exactly 12, emits a non-empty typed authorization/traversal outcome, and emits the authoritative fallback category. A timeout without the typed terminal outcome, exit 0, direct-traffic success, or an unbounded process is a failed canary. Restore the firewall after the drill and verify normal traffic again.

Private diagnostics and retention

The harness creates raw JSON events under a randomly named 0700-equivalent temporary directory with a process umask of 077. Successful raw events are deleted automatically. On failure they remain in that private directory so the operator can triage locally; do not attach them to an issue before removing listing IDs and reviewing every field. Set RENDEZVOUS_CANARY_KEEP_RAW=true only for an approved short-lived diagnostic capture, then delete it manually.

The sanitized summary contains the commit, clean/dirty tree state, UTC time, role, declared topology, observed address-family gate, aggregate booleans, and the retention policy. Formal evidence requires the default clean-tree check.

Public ingress proof

Success through a public hostname is insufficient proof that UDP source/reply addressing is preserved. During a canary, an authorized operator must capture only packet headers at the service host and verify:

1. each authenticated contribution reaches the mediator with the external peer source tuple visible to the server;
2. introductions are sent from the same advertised public mediator tuple;
3. no load balancer, user-space proxy, service mesh, or destination NAT changes the source or reply tuple expected by LiteNetLib; and
4. malformed or unauthenticated traffic receives no amplified response.

Store the approval, capture time window, candidate digest, topology category, and pass/fail result. Do not retain packet payloads or peer tuples in the repository. A failed tuple check blocks release even if one canary happened to connect.

Rehearsal and triage

Run the security, capacity, observability, deployment, rollback, privacy, and incident procedures against the same immutable candidate. The independent operator records which runbook revision they followed, start/end time, observed alerts, recovery time, unexpected decisions, and pass/fail result. Update the documentation and repeat any failed or ambiguous step.

Before changing the readiness record, reconcile every open roadmap issue as one of: blocking with an owner and evidence needed, accepted-v1 with a bounded documented limitation, or post-v1 with a filed issue. HA, active-active or multi-region routing, relays, platform authentication, and scale above the single-active v1 envelope are not silently accepted; each needs a traceable post-v1 issue. The current follow-ups are relay decision #24, HA/multi-region shared state and routing #28, scale beyond the measured envelope #29, and platform authentication adapters #30. Run the checker after every evidence update. Only its READY result may support a production-ready claim.