HeiKyu/Rendezvous
2
0
Fork 0
Code Issues 27 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(operations): add production readiness gate (#23)

Browse Source
This commit is contained in:
KyuubiYoru
2026-07-16 22:19:51 +02:00
parent 6bad659c12
commit 00d5ff7764
13 changed files with 782 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/evidence/production-readiness-v1.json
+124
View File
@@ -0,0 +1,124 @@
{
"schemaVersion": 1,
"kind": "rendezvous-production-readiness",
"evaluatedCommit": "6bad659c123ad45ad0d9d07f93217c2e8c42d459",
"decision": "not-ready",
"localGates": [
{
"id": "immutable-release-artifacts",
"status": "pending",
"evidenceRef": "docs/operations/production-readiness.md",
"note": "Rebuild after the readiness tooling checkpoint."
},
{
"id": "debug-and-release-verification",
"status": "pending",
"evidenceRef": "docs/operations/production-readiness.md",
"note": "Re-run after the readiness tooling checkpoint."
},
{
"id": "real-consumer-pilots",
"status": "pending",
"evidenceRef": "docs/integration/spacegame-pilot.md",
"note": "Pin and re-run both real consumer revisions."
},
{
"id": "candidate-capacity-resilience",
"status": "pending",
"evidenceRef": "docs/evidence/capacity/v2/candidate-2cpu.json",
"note": "Re-run the five-minute candidate on the tooling checkpoint."
},
{
"id": "production-process-recovery",
"status": "pending",
"evidenceRef": "docs/operations/capacity-and-resilience.md",
"note": "Re-run process restart, drain, and rollback gates."
},
{
"id": "security-privacy-observability",
"status": "pending",
"evidenceRef": "docs/operations/production-readiness.md",
"note": "Re-run the combined release verification matrix."
}
],
"externalGates": [
{
"id": "public-package-empty-cache-restore",
"status": "pending",
"evidenceRef": "docs/operations/production-readiness.md",
"note": "The public registry does not currently resolve version 1.0.0."
},
{
"id": "signed-publication",
"status": "pending",
"evidenceRef": "docs/releases/README.md",
"note": "Protected release credentials and immutable tag publication are required."
},
{
"id": "source-preserving-udp-ingress",
"status": "pending",
"evidenceRef": "docs/operations/production-readiness.md",
"note": "The public ingress path needs packet-level source and reply validation."
},
{
"id": "same-lan-direct-canary",
"status": "pending",
"evidenceRef": "docs/operations/production-readiness.md",
"note": "Requires two independently operated game clients."
},
{
"id": "home-nat-direct-canary",
"status": "pending",
"evidenceRef": "docs/operations/production-readiness.md",
"note": "Requires distinct residential networks."
},
{
"id": "restrictive-cgnat-typed-failure",
"status": "pending",
"evidenceRef": "docs/operations/production-readiness.md",
"note": "Requires a known restrictive carrier topology."
},
{
"id": "firewall-blocked-udp-typed-failure",
"status": "pending",
"evidenceRef": "docs/operations/production-readiness.md",
"note": "Requires an independently controlled firewall rule."
},
{
"id": "ipv6-direct-canary",
"status": "pending",
"evidenceRef": "docs/operations/production-readiness.md",
"note": "Requires two IPv6-capable external clients and public ingress."
},
{
"id": "public-rate-shaped-capacity",
"status": "pending",
"evidenceRef": "docs/operations/capacity-and-resilience.md",
"note": "The full public HTTP and UDP traffic mix has not been measured."
},
{
"id": "one-hour-candidate-endurance",
"status": "pending",
"evidenceRef": "docs/operations/capacity-and-resilience.md",
"note": "A production-shaped one-hour candidate run is required."
},
{
"id": "alert-delivery",
"status": "pending",
"evidenceRef": "docs/operations/incident-runbooks.md",
"note": "A real alert sink must observe trigger and recovery notifications."
},
{
"id": "cold-standby-rollback-drill",
"status": "pending",
"evidenceRef": "docs/operations/capacity-and-resilience.md",
"note": "The deployment must demonstrate the host-visible recovery objective."
},
{
"id": "documentation-only-runbook-exercise",
"status": "pending",
"evidenceRef": "docs/operations/incident-runbooks.md",
"note": "An independent operator must execute the runbooks using only the docs."
}
]
}
docs/integration/test-client.md
+6
View File
@@ -116,6 +116,9 @@ rm deploy/compose/secrets/signing-key
codes are stable automation contracts. Informational events use stdout and
failures use stderr.
Successful direct-connection and direct-traffic events include the coarse
`addressFamily` value `ipv4` or `ipv6`. They never include the peer address.
The deployment smoke performs the full health, publish, join, mediation, direct
traffic, outcome-report, and cleanup flow using bounded waits:
@@ -220,3 +223,6 @@ least-scope publisher credential from the deployment secret boundary and set the
external service, mediator, and matching scope variables described in the
[secure Linux deployment smoke](../deployment/linux.md#http-and-udp-smoke).
Run representative external-network tests; loopback success is not NAT coverage.
Use the redacting, bounded
[real-network canary procedure](../operations/production-readiness.md) for formal
production evidence rather than committing raw TestClient JSON.
docs/operations/production-readiness.md
+191
View File
@@ -0,0 +1,191 @@
# Production-readiness decision and real-network canary
Tracking: #23
Rendezvous v1 is **not production-ready** until every required gate in
[`production-readiness-v1.json`](../evidence/production-readiness-v1.json) is
recorded as `pass`. The machine-checkable decision is intentionally fail-closed:
```bash
./scripts/check-production-readiness.sh
```
Exit `0` means every required gate is present and passing, exit `3` means the
record is valid but at least one gate is pending or failed, and exit `2` means
the record itself is malformed or contains identifier-, endpoint-, account-, or
credential-shaped data. Editing only the top-level decision cannot make the
check pass.
The checked-in record is an index, not a log archive. It contains one
repository-relative evidence reference and a short categorical note per gate.
Raw packet captures, client event streams, publisher credentials, public or
private network endpoints, listing IDs, and player/account identifiers must not
be committed.
## Required decision matrix
The local matrix covers immutable artifacts, Debug and Release verification,
both real game consumers, the candidate capacity/resilience profile,
production-process recovery, and the combined security/privacy/observability
gate. These may be reproduced by the project team on a clean candidate commit.
The external matrix remains distinct because a local namespace, loopback,
container bridge, or second process on one machine cannot prove it:
| Gate | Required evidence |
| --- | --- |
| Public package empty-cache restore | A clean machine restores the exact Client and Contracts version using only the documented public sources. |
| Signed publication | The immutable tag publishes packages, image digest, SBOMs, provenance, checksums, and verifiable signatures through the protected release workflow. |
| Source-preserving UDP ingress | Packet capture on the service host proves the mediator observes each peer's real source tuple and replies from the advertised public tuple; no UDP proxy rewrites either direction. |
| Same-LAN direct canary | Two independently operated game clients establish authenticated direct LiteNetLib traffic. |
| Home-NAT direct canary | Host and joiner on distinct residential networks establish authenticated direct LiteNetLib traffic. |
| Restrictive/CGNAT and blocked-UDP canaries | Each bounded join exits `12`, records a typed terminal category, and exposes the game-owned fallback policy without hanging or claiming success. |
| IPv6 direct canary | Two external IPv6 clients record authenticated direct traffic and an observed `ipv6` peer address family. |
| Public rate-shaped capacity | The documented HTTP/UDP workload mix meets its objectives through TLS, Kestrel, JSON, LiteNetLib, kernel sockets, and public ingress. |
| One-hour endurance | The immutable production-shaped candidate completes the one-hour profile without a state, handle, memory, readiness, or latency failure. |
| Alert delivery | A real alert sink receives both trigger and recovery notifications for the rehearsed outage. |
| Cold-standby rollback | Drain, stop, socket release, replacement start, host re-registration, and rollback meet the process and host-visible recovery objectives. |
| Documentation-only exercise | An operator who did not author the runbooks completes key rotation/revocation, outage, restart, re-registration, and rollback using only the checked-in documentation. |
Failure or missing evidence is blocking. It is never converted into an accepted
risk by changing the wording of the readiness note.
## Prepare one immutable canary build
Use the exact release candidate on every canary machine. Verify a clean checkout,
restore in locked mode, and build the TestClient before changing networks:
```bash
test -z "$(git status --porcelain)"
dotnet restore Rendezvous.slnx --locked-mode
dotnet build Rendezvous.slnx --configuration Release --no-restore
```
Keep shell tracing disabled. The host receives a short-lived, least-scope
publisher credential through `RENDEZVOUS_PUBLISHER_CREDENTIAL`; it must never be
put in an argument, coordination file, evidence file, command transcript, or
support message. Set the public HTTPS service URL and advertised UDP mediator
tuple separately. TestClient rejects credentials embedded in the service URL.
## Run a success canary across two machines
On the host machine, choose `same-lan`, `home-nat`, or `ipv6-direct`. The
coordination file is mode `0600` and contains only the temporary listing UUID.
It is not evidence; transfer it through an approved private channel, then delete
both copies.
```bash
set +x
export RENDEZVOUS_PUBLISHER_CREDENTIAL='supplied-by-the-approved-secret-boundary'
export RENDEZVOUS_CANARY_ROLE=host
export RENDEZVOUS_CANARY_TOPOLOGY=home-nat
export RENDEZVOUS_CANARY_ADDRESS_FAMILY=ipv4
export RENDEZVOUS_CANARY_HTTP_URL='https://service.example.invalid/'
export RENDEZVOUS_CANARY_UDP_ENDPOINT='203.0.113.10:9050'
export RENDEZVOUS_CANARY_COORDINATION_FILE="$HOME/.local/state/rendezvous-canary-listing"
export RENDEZVOUS_CANARY_OUTPUT="$PWD/artifacts/canary/home-nat-host.json"
./scripts/run-real-network-canary.sh
```
The host prints only that it is ready and waits for the authenticated exchange.
On the joiner, read the securely transferred UUID without placing it in shell
history and run the matching topology:
```bash
set +x
read -r RENDEZVOUS_CANARY_LISTING_ID < "$HOME/.local/state/rendezvous-canary-listing"
export RENDEZVOUS_CANARY_LISTING_ID
export RENDEZVOUS_CANARY_ROLE=client-success
export RENDEZVOUS_CANARY_TOPOLOGY=home-nat
export RENDEZVOUS_CANARY_ADDRESS_FAMILY=ipv4
export RENDEZVOUS_CANARY_HTTP_URL='https://service.example.invalid/'
export RENDEZVOUS_CANARY_UDP_ENDPOINT='203.0.113.10:9050'
export RENDEZVOUS_CANARY_OUTPUT="$PWD/artifacts/canary/home-nat-client.json"
./scripts/run-real-network-canary.sh
unset RENDEZVOUS_CANARY_LISTING_ID
```
The host summary requires authenticated direct traffic and deregistration. The
client summary requires connection, authenticated direct traffic, accepted
outcome reporting, and the declared address family observed on the actual peer.
The summaries deliberately contain no network tuple or listing identifier.
For IPv6, set the topology to `ipv6-direct`, the family to `ipv6`, and use the
deployment's bracketed IPv6 mediator form. Record unsupported operating systems,
console platforms, VPNs, and address families as untested; an IPv4 pass is not
evidence for IPv6 or a platform network policy.
## Run a bounded failure canary
Start the host from an independently reachable network as above. On the joiner,
apply the reviewed firewall rule that blocks the relevant UDP path, or use the
known restrictive carrier network, then set `client-expected-failure` and the
matching topology:
```bash
export RENDEZVOUS_CANARY_ROLE=client-expected-failure
export RENDEZVOUS_CANARY_TOPOLOGY=firewall-blocked-udp
export RENDEZVOUS_CANARY_ADDRESS_FAMILY=ipv4
export RENDEZVOUS_CANARY_OUTPUT="$PWD/artifacts/canary/firewall-blocked-client.json"
./scripts/run-real-network-canary.sh
```
This role passes only when TestClient exits exactly `12`, emits a non-empty typed
authorization/traversal outcome, and emits the authoritative fallback category.
A timeout without the typed terminal outcome, exit `0`, direct-traffic success,
or an unbounded process is a failed canary. Restore the firewall after the drill
and verify normal traffic again.
## Private diagnostics and retention
The harness creates raw JSON events under a randomly named `0700`-equivalent
temporary directory with a process `umask` of `077`. Successful raw events are
deleted automatically. On failure they remain in that private directory so the
operator can triage locally; do not attach them to an issue before removing
listing IDs and reviewing every field. Set `RENDEZVOUS_CANARY_KEEP_RAW=true`
only for an approved short-lived diagnostic capture, then delete it manually.
The sanitized summary contains the commit, clean/dirty tree state, UTC time,
role, declared topology, observed address-family gate, aggregate booleans, and
the retention policy. Formal evidence requires the default clean-tree check.
## Public ingress proof
Success through a public hostname is insufficient proof that UDP source/reply
addressing is preserved. During a canary, an authorized operator must capture
only packet headers at the service host and verify:
1. each authenticated contribution reaches the mediator with the external peer
source tuple visible to the server;
2. introductions are sent from the same advertised public mediator tuple;
3. no load balancer, user-space proxy, service mesh, or destination NAT changes
the source or reply tuple expected by LiteNetLib; and
4. malformed or unauthenticated traffic receives no amplified response.
Store the approval, capture time window, candidate digest, topology category,
and pass/fail result. Do not retain packet payloads or peer tuples in the
repository. A failed tuple check blocks release even if one canary happened to
connect.
## Rehearsal and triage
Run the security, capacity, observability, deployment, rollback, privacy, and
incident procedures against the same immutable candidate. The independent
operator records which runbook revision they followed, start/end time, observed
alerts, recovery time, unexpected decisions, and pass/fail result. Update the
documentation and repeat any failed or ambiguous step.
Before changing the readiness record, reconcile every open roadmap issue as one
of: `blocking` with an owner and evidence needed, `accepted-v1` with a bounded
documented limitation, or `post-v1` with a filed issue. HA, active-active or
multi-region routing, relays, platform authentication, and scale above the
single-active v1 envelope are not silently accepted; each needs a traceable
post-v1 issue. The current follow-ups are relay decision [#24], HA/multi-region
shared state and routing [#28], scale beyond the measured envelope [#29], and
platform authentication adapters [#30]. Run the checker after every evidence
update. Only its `READY` result may support a production-ready claim.
[#24]: https://git.finalfactory.de/HeiKyu/Rendezvous/issues/24
[#28]: https://git.finalfactory.de/HeiKyu/Rendezvous/issues/28
[#29]: https://git.finalfactory.de/HeiKyu/Rendezvous/issues/29
[#30]: https://git.finalfactory.de/HeiKyu/Rendezvous/issues/30