KyuubiYoru
218 lines
11 KiB
Markdown
218 lines
11 KiB
Markdown
# Production-readiness decision and real-network canary
|
|
|
|
Tracking: #23
|
|
|
|
Rendezvous v1 is **not production-ready** until every required gate in
|
|
[`production-readiness-v1.json`](../evidence/production-readiness-v1.json) is
|
|
recorded as `pass`. The machine-checkable decision is intentionally fail-closed:
|
|
|
|
```bash
|
|
./scripts/check-production-readiness.sh
|
|
```
|
|
|
|
Exit `0` means every required gate is present and passing, exit `3` means the
|
|
record is valid but at least one gate is pending or failed, and exit `2` means
|
|
the record itself is malformed or contains identifier-, endpoint-, account-, or
|
|
credential-shaped data. Editing only the top-level decision cannot make the
|
|
check pass.
|
|
|
|
The checked-in record is an index, not a log archive. It contains one
|
|
repository-relative evidence reference and a short categorical note per gate.
|
|
Raw packet captures, client event streams, publisher credentials, public or
|
|
private network endpoints, listing IDs, and player/account identifiers must not
|
|
be committed.
|
|
|
|
## Required decision matrix
|
|
|
|
The local matrix covers immutable artifacts, Debug and Release verification,
|
|
both real game consumers, the candidate capacity/resilience profile,
|
|
production-process recovery, and the combined security/privacy/observability
|
|
gate. These may be reproduced by the project team on a clean candidate commit.
|
|
|
|
The external matrix remains distinct because a local namespace, loopback,
|
|
container bridge, or second process on one machine cannot prove it:
|
|
|
|
| Gate | Required evidence |
|
|
| --- | --- |
|
|
| Public package empty-cache restore | A clean machine restores the exact Client and Contracts version using only the documented public sources. |
|
|
| Signed publication | The immutable tag publishes packages, image digest, SBOMs, provenance, checksums, and verifiable signatures through the protected release workflow. |
|
|
| Source-preserving UDP ingress | Packet capture on the service host proves the mediator observes each peer's real source tuple and replies from the advertised public tuple; no UDP proxy rewrites either direction. |
|
|
| Same-LAN direct canary | Two independently operated game clients establish authenticated direct LiteNetLib traffic. |
|
|
| Home-NAT direct canary | Host and joiner on distinct residential networks establish authenticated direct LiteNetLib traffic. |
|
|
| Restrictive/CGNAT and blocked-UDP canaries | Each bounded join exits `12`, records a typed terminal category, and exposes the game-owned fallback policy without hanging or claiming success. |
|
|
| IPv6 direct canary | Two external IPv6 clients record authenticated direct traffic and an observed `ipv6` peer address family. |
|
|
| Public rate-shaped capacity | The documented HTTP/UDP workload mix meets its objectives through TLS, Kestrel, JSON, LiteNetLib, kernel sockets, and public ingress. |
|
|
| One-hour endurance | The immutable production-shaped candidate completes the one-hour profile without a state, handle, memory, readiness, or latency failure. |
|
|
| Alert delivery | A real alert sink receives both trigger and recovery notifications for the rehearsed outage. |
|
|
| Cold-standby rollback | Drain, stop, socket release, replacement start, host re-registration, and rollback meet the process and host-visible recovery objectives. |
|
|
| Documentation-only exercise | An operator who did not author the runbooks completes key rotation/revocation, outage, restart, re-registration, and rollback using only the checked-in documentation. |
|
|
|
|
Failure or missing evidence is blocking. It is never converted into an accepted
|
|
risk by changing the wording of the readiness note.
|
|
|
|
When an external gate passes, add a redacted repository JSON attestation and
|
|
point that gate's `evidenceRef` to it. The checker requires this exact shape and
|
|
binds the gate to the evaluated candidate commit. `artifactDigest` is the SHA-256
|
|
of the protected evidence bundle or public release record, not a peer endpoint,
|
|
listing identifier, account identifier, or credential:
|
|
|
|
```json
|
|
{
|
|
"schemaVersion": 1,
|
|
"kind": "rendezvous-external-gate-attestation",
|
|
"gateId": "replace-with-the-exact-gate-id",
|
|
"candidateCommit": "replace-with-the-40-character-candidate-commit",
|
|
"result": "pass",
|
|
"performedAtUtc": "2026-01-01T00:00:00Z",
|
|
"artifactDigest": "replace-with-the-64-character-sha256",
|
|
"evidenceLocation": "protected-operations-record",
|
|
"reviewerRole": "independent-operator"
|
|
}
|
|
```
|
|
|
|
Allowed evidence locations are `protected-operations-record` and
|
|
`public-release-record`. Allowed reviewer roles are `release-operator`,
|
|
`network-operator`, `security-operator`, and `independent-operator`. The checker
|
|
rejects a missing file, wrong gate, wrong candidate, malformed digest, naive
|
|
timestamp, extra fields, or sensitive-data-shaped contents.
|
|
|
|
## Prepare one immutable canary build
|
|
|
|
Use the exact release candidate on every canary machine. Verify a clean checkout,
|
|
restore in locked mode, and build the TestClient before changing networks:
|
|
|
|
```bash
|
|
test -z "$(git status --porcelain)"
|
|
dotnet restore Rendezvous.slnx --locked-mode
|
|
dotnet build Rendezvous.slnx --configuration Release --no-restore
|
|
```
|
|
|
|
Keep shell tracing disabled. The host receives a short-lived, least-scope
|
|
publisher credential through `RENDEZVOUS_PUBLISHER_CREDENTIAL`; it must never be
|
|
put in an argument, coordination file, evidence file, command transcript, or
|
|
support message. Set the public HTTPS service URL and advertised UDP mediator
|
|
tuple separately. TestClient rejects credentials embedded in the service URL.
|
|
|
|
## Run a success canary across two machines
|
|
|
|
On the host machine, choose `same-lan`, `home-nat`, or `ipv6-direct`. The
|
|
coordination file is mode `0600` and contains only the temporary listing UUID.
|
|
It is not evidence; transfer it through an approved private channel, then delete
|
|
both copies.
|
|
|
|
```bash
|
|
set +x
|
|
export RENDEZVOUS_PUBLISHER_CREDENTIAL='supplied-by-the-approved-secret-boundary'
|
|
export RENDEZVOUS_CANARY_ROLE=host
|
|
export RENDEZVOUS_CANARY_TOPOLOGY=home-nat
|
|
export RENDEZVOUS_CANARY_ADDRESS_FAMILY=ipv4
|
|
export RENDEZVOUS_CANARY_HTTP_URL='https://service.example.invalid/'
|
|
export RENDEZVOUS_CANARY_UDP_ENDPOINT='203.0.113.10:9050'
|
|
export RENDEZVOUS_CANARY_COORDINATION_FILE="$HOME/.local/state/rendezvous-canary-listing"
|
|
export RENDEZVOUS_CANARY_OUTPUT="$PWD/artifacts/canary/home-nat-host.json"
|
|
./scripts/run-real-network-canary.sh
|
|
```
|
|
|
|
The host prints only that it is ready and waits for the authenticated exchange.
|
|
On the joiner, read the securely transferred UUID without placing it in shell
|
|
history and run the matching topology:
|
|
|
|
```bash
|
|
set +x
|
|
read -r RENDEZVOUS_CANARY_LISTING_ID < "$HOME/.local/state/rendezvous-canary-listing"
|
|
export RENDEZVOUS_CANARY_LISTING_ID
|
|
export RENDEZVOUS_CANARY_ROLE=client-success
|
|
export RENDEZVOUS_CANARY_TOPOLOGY=home-nat
|
|
export RENDEZVOUS_CANARY_ADDRESS_FAMILY=ipv4
|
|
export RENDEZVOUS_CANARY_HTTP_URL='https://service.example.invalid/'
|
|
export RENDEZVOUS_CANARY_UDP_ENDPOINT='203.0.113.10:9050'
|
|
export RENDEZVOUS_CANARY_OUTPUT="$PWD/artifacts/canary/home-nat-client.json"
|
|
./scripts/run-real-network-canary.sh
|
|
unset RENDEZVOUS_CANARY_LISTING_ID
|
|
```
|
|
|
|
The host summary requires authenticated direct traffic and deregistration. The
|
|
client summary requires connection, authenticated direct traffic, accepted
|
|
outcome reporting, and the declared address family observed on the actual peer.
|
|
The summaries deliberately contain no network tuple or listing identifier.
|
|
|
|
For IPv6, set the topology to `ipv6-direct`, the family to `ipv6`, and use the
|
|
deployment's bracketed IPv6 mediator form. Record unsupported operating systems,
|
|
console platforms, VPNs, and address families as untested; an IPv4 pass is not
|
|
evidence for IPv6 or a platform network policy.
|
|
|
|
## Run a bounded failure canary
|
|
|
|
Start the host from an independently reachable network as above. On the joiner,
|
|
apply the reviewed firewall rule that blocks the relevant UDP path, or use the
|
|
known restrictive carrier network, then set `client-expected-failure` and the
|
|
matching topology:
|
|
|
|
```bash
|
|
export RENDEZVOUS_CANARY_ROLE=client-expected-failure
|
|
export RENDEZVOUS_CANARY_TOPOLOGY=firewall-blocked-udp
|
|
export RENDEZVOUS_CANARY_ADDRESS_FAMILY=ipv4
|
|
export RENDEZVOUS_CANARY_OUTPUT="$PWD/artifacts/canary/firewall-blocked-client.json"
|
|
./scripts/run-real-network-canary.sh
|
|
```
|
|
|
|
This role passes only when TestClient exits exactly `12`, emits a non-empty typed
|
|
authorization/traversal outcome, and emits the authoritative fallback category.
|
|
A timeout without the typed terminal outcome, exit `0`, direct-traffic success,
|
|
or an unbounded process is a failed canary. Restore the firewall after the drill
|
|
and verify normal traffic again.
|
|
|
|
## Private diagnostics and retention
|
|
|
|
The harness creates raw JSON events under a randomly named `0700`-equivalent
|
|
temporary directory with a process `umask` of `077`. Successful raw events are
|
|
deleted automatically. On failure they remain in that private directory so the
|
|
operator can triage locally; do not attach them to an issue before removing
|
|
listing IDs and reviewing every field. Set `RENDEZVOUS_CANARY_KEEP_RAW=true`
|
|
only for an approved short-lived diagnostic capture, then delete it manually.
|
|
|
|
The sanitized summary contains the commit, clean/dirty tree state, UTC time,
|
|
role, declared topology, observed address-family gate, aggregate booleans, and
|
|
the retention policy. Formal evidence requires the default clean-tree check.
|
|
|
|
## Public ingress proof
|
|
|
|
Success through a public hostname is insufficient proof that UDP source/reply
|
|
addressing is preserved. During a canary, an authorized operator must capture
|
|
only packet headers at the service host and verify:
|
|
|
|
1. each authenticated contribution reaches the mediator with the external peer
|
|
source tuple visible to the server;
|
|
2. introductions are sent from the same advertised public mediator tuple;
|
|
3. no load balancer, user-space proxy, service mesh, or destination NAT changes
|
|
the source or reply tuple expected by LiteNetLib; and
|
|
4. malformed or unauthenticated traffic receives no amplified response.
|
|
|
|
Store the approval, capture time window, candidate digest, topology category,
|
|
and pass/fail result. Do not retain packet payloads or peer tuples in the
|
|
repository. A failed tuple check blocks release even if one canary happened to
|
|
connect.
|
|
|
|
## Rehearsal and triage
|
|
|
|
Run the security, capacity, observability, deployment, rollback, privacy, and
|
|
incident procedures against the same immutable candidate. The independent
|
|
operator records which runbook revision they followed, start/end time, observed
|
|
alerts, recovery time, unexpected decisions, and pass/fail result. Update the
|
|
documentation and repeat any failed or ambiguous step.
|
|
|
|
Before changing the readiness record, reconcile every open roadmap issue as one
|
|
of: `blocking` with an owner and evidence needed, `accepted-v1` with a bounded
|
|
documented limitation, or `post-v1` with a filed issue. HA, active-active or
|
|
multi-region routing, relays, platform authentication, and scale above the
|
|
single-active v1 envelope are not silently accepted; each needs a traceable
|
|
post-v1 issue. The current follow-ups are relay decision [#24], HA/multi-region
|
|
shared state and routing [#28], scale beyond the measured envelope [#29], and
|
|
platform authentication adapters [#30]. Run the checker after every evidence
|
|
update. Only its `READY` result may support a production-ready claim.
|
|
|
|
[#24]: https://git.finalfactory.de/HeiKyu/Rendezvous/issues/24
|
|
[#28]: https://git.finalfactory.de/HeiKyu/Rendezvous/issues/28
|
|
[#29]: https://git.finalfactory.de/HeiKyu/Rendezvous/issues/29
|
|
[#30]: https://git.finalfactory.de/HeiKyu/Rendezvous/issues/30
|