feat(operations): add capacity and resilience gates (#18)

docs/operations/capacity-and-resilience.md

@@ -0,0 +1,189 @@
+# Capacity, resilience, and availability gate
+
+Tracking: #18
+
+This gate turns the v1 budgets in ADR 0003 into a repeatable release decision.
+It does not turn Rendezvous into a horizontally scalable service: v1 remains one
+active process with bounded in-memory state. A second process may be a cold
+standby, but it must not accept traffic until the first process has stopped and
+released the public HTTP and UDP endpoints.
+
+## Launch envelope and approved core-state profile
+
+The approved core-state profile is one Linux process limited to 2 vCPU and
+2 GiB RAM. Public HTTP/UDP numbers are launch objectives that require the #23
+real-network canary before they become a supported service claim:
+
+| Dimension | Value | Evidence status |
+| --- | --- | --- |
+| Visible listings | 25,000 | Enforced and measured here |
+| Active join attempts | 10,000 | Enforced and measured here |
+| Core control path | 200 operations/second; p95 at most 200 ms | Measured here |
+| Core mediation path | 2,000 pairings/second; p95 at most 100 ms | Measured here |
+| Sustained HTTP demand | 200 requests/second | #23 launch objective; not yet a supported claim |
+| Sustained UDP demand | 2,000 datagrams/second | #23 launch objective; not yet a supported claim |
+| Public HTTP/UDP latency | p95 at most 200 ms / 100 ms | #23 launch objective; not yet a supported claim |
+| Capacity-phase average CPU / peak working memory | below 70% / below 1.5 GiB | Measured for the core candidate |
+| Valid in-profile monthly availability | 99.5%, excluding announced maintenance | Operational objective |
+| Process-ready RTO / host-visible recovery | 15 seconds / 90 seconds | 15 seconds automated; 90-second deployment drill required |
+
+The proposed public-network mix is 20% registration/update, 30% lease-critical
+renew/delete, 30% browse, and 20% join authorization for HTTP. The UDP mix is
+60% authenticated host-presence refresh, 30% attempt contributions, and 10%
+invalid or duplicate traffic that must be dropped early. A deployment may use a
+lower per-game profile, but must not claim a higher one without new versioned
+evidence.
+
+The capacity harness fills the complete state ceilings, then measures
+registration plus presence, renewal, a 100-item compatible browse, join
+issuance, simultaneous two-peer pairing, principal revocation, and telemetry.
+It applies 200/100 ms guardrails and minimum 200 control / 2,000 mediation
+operations per second to the core hot path. Those measurements deliberately
+exclude Kestrel, LiteNetLib, TLS, JSON, socket scheduling, and the documented
+mixed traffic shape. The #23 real-network canary must exercise those layers,
+rate-shape the mix, record errors and shedding, and meet the public objectives
+before launch; a core result is not a public-network latency or throughput claim.
+
+## Reproduce the evidence
+
+Every push runs the quick profile and the selected fault matrix:
+
+```bash
+./scripts/run-capacity-gate.sh
+```
+
+Run the production candidate on an otherwise idle Linux host and restrict the
+runtime to two logical CPUs. The default candidate includes a five-minute,
+high-intensity expiry soak; use 3,600 seconds for a release-candidate endurance
+run:
+
+```bash
+export RENDEZVOUS_CAPACITY_PROFILE=candidate
+export RENDEZVOUS_CAPACITY_CPUSET=0,1
+export RENDEZVOUS_CAPACITY_OUTPUT="$PWD/artifacts/capacity/candidate.json"
+./scripts/run-capacity-gate.sh
+
+# Release-candidate endurance override:
+dotnet run --project tests/FinalFactory.Rendezvous.Capacity \
+  --configuration Release --no-build -- \
+  --profile candidate --soak-seconds 3600 \
+  --output artifacts/capacity/candidate-endurance.json
+```
+
+The machine must have at least 2 GiB available to the process. For formal
+deployment evidence, run inside the same cgroup/container shape as production.
+The v2 JSON embeds the commit and tree state, command, image context, CPU model,
+kernel, affinity, cgroup quota/limit, collector mode, and workload seed. Supply
+`RENDEZVOUS_EVIDENCE_IMAGE_DIGEST` when running a release image. Do not compare
+results collected under a debugger,
+concurrent build, thermal throttling, or oversubscribed CI host.
+
+The checked-in baseline is
+[`candidate-2cpu.json`](../evidence/capacity/v2/candidate-2cpu.json). It was
+produced on .NET 10.0.9/Linux x64 with CPU affinity restricted to two logical
+CPUs. It filled 25,000 listings and 10,000 attempts, peaked at about 162 MiB,
+and cleared all active/retained state. The five-minute baseline supersedes any
+earlier local probe when its timestamp and target duration differ.
+
+## Soak and bounded-state interpretation
+
+Each soak cycle creates a listing, repeatedly renews its lease and refreshes
+presence, creates a join attempt, replay marker, and retained outcome, checks
+that scheduled expiry entries remain proportional to live keys, then advances
+the injected monotonic clock beyond all
+deadlines, and verifies that listings, attempts, replay, idempotency, and outcome
+state return to zero. The candidate also measures managed-memory and process
+handle deltas after full collection. Failure is any retained state, more than
+64 MiB retained managed memory, more than eight retained handles, a working set
+above 1.5 GiB, an untyped capacity result, or failure to admit work after expiry.
+
+This accelerated soak intentionally executes far more state lifecycle/cleanup
+events than wall-clock traffic would permit. It catches stale deadline-queue
+entries, cache growth, replay/idempotency retention, and cleanup cost. Because
+it does not open Kestrel/LiteNetLib connections, its process-handle delta is only
+a harness guard and is not evidence of transport stability by itself. The
+selected production-process gate adds a ten-second real HTTP/UDP transport soak,
+samples child-process handles and RSS, asserts bounded growth, then verifies a
+clean SIGTERM and socket release. #23 must extend that into the full rate-shaped
+multi-client canary while sampling queues, managed memory, and state
+cardinalities. A one-hour core override remains required before tagging a
+production release.
+
+## Fault and recovery matrix
+
+`run-capacity-gate.sh` runs these deterministic production paths before the
+numeric profile:
+
+| Fault | Required result |
+| --- | --- |
+| HTTP/UDP overload and tracker exhaustion | Typed HTTP `429`/`CapacityExceeded`, silent UDP drop, bounded tracker keys, recovery after the window |
+| Optional traffic saturation | Lease-critical renew/update/delete capacity remains available |
+| Store/dependency unavailable | Readiness fails; new authorization returns typed `ServiceUnavailable`; liveness remains independent |
+| Graceful drain/SIGTERM | New work returns `Draining`; existing pairing may finish; process exits 0 and releases TCP/UDP before the deadline |
+| Hard restart | In-flight state is lost; SDK reports typed `ServiceUnavailable`; a host re-registers, rebinds presence, and becomes the only browser-visible replacement |
+| UDP listener bind/restart | Readiness stays false without the required listener; rebinding the advertised port restores native LiteNetLib pairing |
+| Wall-clock jump/skew | Monotonic lease/attempt authority is neither shortened nor extended; credential skew remains capped at 30 seconds |
+| Signing-secret rotation | New key signs, overlap verifies, retired/revoked key rejects, missing material fails startup |
+| Principal revocation | Listing, presence, attempts, and outcome paths are removed atomically within the latency budget |
+
+No external database exists in v1, so “dependency/store failure” means the
+process-local atomic store is marked unavailable or a required listener/key is
+unready. The service fails closed rather than pretending a degraded writable
+mode exists.
+
+## Bandwidth and amplification
+
+- Accepted application datagrams are at most 1,200 bytes.
+- Malformed, oversized, unauthenticated, stale, replayed, wrong-role, and
+  rate-limited traffic receives zero response bytes.
+- A completing authenticated contribution produces at most one introduction to
+  each observed peer, and the combined response is at most 2.0 times that
+  contribution's bytes.
+- The frozen-envelope and native LiteNetLib socket tests measure this on the real
+  UDP listener; the hostile corpus and allocation gate exercise 10,000+ inputs
+  without input-sized logs, tasks, or queues.
+
+Bandwidth planning must therefore reserve ingress for the configured 2,000
+datagrams/second plus edge overhead and egress for a worst-case verified 2.0
+amplification. Actual successful pairs normally use two contributions and two
+introductions; normal gameplay leaves Rendezvous entirely.
+
+## Availability decision
+
+Single-active remains the v1 topology. The measured core profile proves bounded
+state and substantial core-path headroom, while public launch capacity remains
+conditional on #23. The service has a bounded stop-before-start restart path.
+Its failure domain is deliberately
+one process/node/public UDP endpoint: node, kernel, host network, DNS/TLS edge,
+secret configuration, or operator error can remove all readiness until the cold
+replacement owns the same source-preserving endpoint.
+
+The 99.5% objective permits about 216 minutes of unannounced downtime in a
+30-day month. Operations must target process readiness within 15 seconds and
+host-visible re-registration within 90 seconds, page when no ready instance
+exists, and include detection plus recovery in the monthly budget. The current
+in-process test validates typed downtime, same-port HTTP restart, fresh
+registration, presence rebinding, and browser visibility in under five seconds;
+the production-process test separately validates graceful termination, TCP/UDP
+release, replacement startup on the same endpoints, UDP readiness, and the
+15-second process-ready RTO. Cold-standby activation policy and the 90-second
+operator-to-host recovery objective still require a deployment drill before
+release. Rollout and rollback use the
+deployment runbook's drain, stop, socket-release, start, smoke sequence; never
+overlap old and new active processes.
+
+Bring shared TTL/CAS state and deterministic mediator routing forward before
+enabling two active instances if any of these occurs:
+
+- one node cannot sustain 150% of the measured 30-day peak while meeting SLOs;
+- CPU stays above 70%, memory above 75%, attempt depth above 70%, or limiter
+  drops/latency remain elevated after abusive traffic is excluded;
+- the availability target rises above 99.5% or planned maintenance must preserve
+  listings; or
+- one region requires multiple simultaneously active mediator endpoints.
+
+Rendezvous makes no multi-instance claim today, so a two-node atomic-pairing
+test is intentionally not applicable. It becomes a hard release gate with the
+shared-state/routing implementation; until then `SingleActiveInstance=false`
+fails production startup. Multi-region and relay remain separate evidence-driven
+decisions.

docs/operations/observability-and-operator-runbook.md

@@ -115,7 +115,9 @@ permission denial is audited with actor and target fingerprints.
 Key revocation is process-local in the current single-instance store. Apply the
 same revocation to every instance, then replace configuration before restarting;
 a restart reconstructs the configured key ring. Principal revocation is bounded
-to ten minutes and removes that principal's active listings and attempts. Use
+to ten minutes and removes that principal's active listings and attempts. A
+repeat action may extend an active revocation but never shortens it; wait for its
+original deadline rather than treating a shorter repeat as an un-revoke. Use
 listing revocation for one targeted session and drain before planned shutdown.
 
 ## Audit retention and incident handling