docs: add integration guides and incident runbooks (#20)

docs/operations/incident-runbooks.md

@@ -0,0 +1,339 @@
+# Incident and change runbooks
+
+Tracking: #20
+
+These runbooks supplement the [signal and operator reference](observability-and-operator-runbook.md).
+Every procedure has four explicit gates: detect, contain, recover, and verify.
+Record timestamps, the release digest, bounded aggregates, audit fingerprints,
+and `X-Rendezvous-Correlation-ID` values. Never copy credentials, capabilities,
+connection tickets, signing material, player identity, raw IP addresses,
+endpoints, listing metadata, or full request bodies into an incident record.
+
+Operator routes must be reachable only from an allowed management source. Use a
+short-lived, least-permission operator credential minted outside Rendezvous.
+Pass it to an approved operator client through protected stdin or a secret agent,
+not a URL, command argument, environment-wide process launcher, shell trace, or
+ticket. All request shapes and responses are defined by the generated
+[OpenAPI v1 document](../api/rendezvous-v1.json).
+
+Before an incident, keep these protected records available without depending on
+the affected service: current and previous image digests, matching configuration,
+key IDs and lifecycle windows (not raw key values), the game owner/on-call map,
+capacity baselines, collector destinations, and a separately authorized
+break-glass operator key. Test management-source allowlisting and credential
+permissions at least once per release.
+
+Use the exact versioned action shapes below. Confirmation fields deliberately
+repeat the target so a stale UI selection or copy error fails closed. Responses
+do not echo targets.
+
+| Operation | JSON body |
+| --- | --- |
+| `POST /v1/operator/listings/revoke` | `{"listingId":"<uuid>","confirmListingId":"<same uuid>"}` |
+| `POST /v1/operator/principals/revoke` | `{"subject":"<exact subject>","confirmSubject":"<same subject>","lifetimeSeconds":60}` |
+| `POST /v1/operator/keys/revoke` | `{"keyId":"<key id>","confirmKeyId":"<same key id>"}` |
+| `POST /v1/operator/drain` | `{"confirmation":"DRAIN"}` |
+
+## Abuse or authentication spike
+
+### Detect
+
+- Alert on a baseline-relative increase in `rendezvous.limiter.drops`, HTTP/UDP
+  request rate, `rendezvous.operator.authentication` rejected/forbidden results,
+  registration requests by authentication status, queue depth, or p95/p99 latency.
+- Check `/health/live`, `/health/ready`, `rendezvous.store.available`, and
+  authenticated `GET /v1/operator/status`. Separate public-source rejection,
+  publisher credential failure, operator probing, and ordinary capacity growth.
+- Use only bounded operation/result dimensions and correlation IDs. Do not group
+  by raw address, token, subject, listing ID, or metadata.
+
+### Contain
+
+- Preserve the dedicated operator partition. Do not raise public limits during
+  an active spike. Apply source-preserving edge rate controls only when their
+  collateral effect is understood and UDP source address/port remains intact.
+- For one abusive session, call `POST /v1/operator/listings/revoke` with identical
+  `listingId` and `confirmListingId`. For a confirmed publisher subject, call
+  `POST /v1/operator/principals/revoke` with identical `subject` and
+  `confirmSubject` and a 1–600 second lifetime.
+- Revoke a signing key only when compromise evidence implicates that issuer;
+  broad key revocation invalidates every credential signed by it. Drain only if
+  the process itself must be isolated.
+
+### Recover
+
+- Correct the source integration, edge rule, leaked principal grant, or tenant
+  budget under change control. Let a bounded principal revocation expire only
+  after the owner confirms remediation; a repeated shorter revocation never
+  shortens the original deadline.
+- Restore normal limits gradually. If saturation caused state churn, allow leases
+  and attempts to expire naturally rather than deleting arbitrary state.
+
+### Verify
+
+- Require limiter drops, authentication result ratios, queue depth, latency, and
+  direct-connect outcomes to return to the same-region baseline for the agreed
+  observation window.
+- Confirm readiness stayed healthy or recovered, operator audit contains the
+  intended action/result fingerprint, revoked resources cannot create new work,
+  and unaffected tenants can still publish, browse, and connect.
+
+## Signing key or issuer compromise
+
+### Detect
+
+- Treat secret-manager access alerts, unexpected issuance, credentials outside
+  the expected region/kind, a signing-key expiry alarm, or unexplained publisher
+  authentication growth as compromise until disproved.
+- Identify the non-secret key ID, allowed credential kinds, game/environment
+  binding, `NotBefore`, `SignUntil`, and `VerifyUntil`. Do not retrieve or paste
+  raw material merely to compare it.
+
+### Contain
+
+- Stop the affected external issuer and deny further access to its secret.
+- From a separate uncompromised break-glass operator key with `RotateKeys`, call
+  `POST /v1/operator/keys/revoke` with identical `keyId` and `confirmKeyId`.
+  Runtime revocation is immediate but process-local.
+- Remove or mark the key revoked in authoritative provisioning before any
+  restart. Revoke affected principals/listings when narrower evidence supports
+  it. Do not drain automatically unless the running instance cannot be trusted.
+
+### Recover
+
+- Generate replacement material in the approved secret boundary, use a new key
+  ID, bind it to the exact credential kind and tenant, and deploy configuration
+  referencing the secret—never the secret value.
+- Resume issuance with short lifetimes. Reissue only to authenticated workloads.
+  When confidentiality is lost, do not use normal overlap to keep compromised
+  credentials valid; document the intentional invalidation window.
+- Rotate any release, registry, or operator credential exposed by the same
+  incident through its owning system; Rendezvous key revocation cannot revoke
+  unrelated systems.
+
+### Verify
+
+- Confirm `GET /v1/operator/status` shows the compromised key revoked and the
+  replacement signing, old credentials fail, new exact-scope credentials work,
+  and the result survives a controlled restart from updated provisioning.
+- Pass TestClient registration, browse, authenticated mediation, and direct
+  traffic with the replacement; monitor authentication and audit results through
+  at least the maximum newly issued credential lifetime.
+
+## Targeted listing or publisher revocation
+
+### Detect
+
+- Validate the abuse report against game-owned records and bounded Rendezvous
+  evidence. Determine whether the target is one listing or an authenticated
+  publisher subject. Do not use display name, metadata, or a raw address as
+  identity.
+- Confirm current aggregate state through `GET /v1/operator/status` and record
+  the correlation IDs that justified action.
+
+### Contain
+
+- Revoke one listing with `POST /v1/operator/listings/revoke`; the exact listing
+  UUID must appear in both confirmation fields.
+- Revoke a publisher with `POST /v1/operator/principals/revoke`; the exact subject
+  must appear in both confirmation fields and `lifetimeSeconds` must be 1–600.
+  This removes that principal's active listings and attempts and blocks new ones
+  for the bounded lifetime.
+- Choose the narrowest action. Do not revoke a tenant key for a single listing.
+
+### Recover
+
+- The game owner resolves the ban, account, workload, or configuration issue in
+  the authoritative game system. Rendezvous does not own user accounts or bans.
+- After the original revocation deadline, permit a newly authenticated publisher
+  to register. There is no un-revoke endpoint and no recovery of removed
+  ephemeral listings; the host creates a new listing.
+
+### Verify
+
+- Confirm the old listing is no longer browsable or joinable, the principal
+  cannot publish during its lifetime, and the audit action/result is present
+  without the raw target.
+- Confirm unrelated publishers in the same tenant and another tenant still pass
+  publish/browse/join/direct-traffic checks.
+
+## Planned restart or crash recovery
+
+### Detect
+
+- Planned restart begins with a recorded change and a healthy current baseline.
+  Crash recovery begins when liveness/process state fails or both TCP 8080 and
+  UDP 9050 stop answering. Distinguish dependency/readiness failure from a dead
+  process; liveness deliberately remains healthy for some recoverable failures.
+- Record active listing/lease/attempt aggregates. They are informational only:
+  v1 has no durable runtime database to restore.
+
+### Contain
+
+- For a planned stop, call `POST /v1/operator/drain` with confirmation exactly
+  `DRAIN`. Require readiness `503`, liveness `200`, and removal from new traffic.
+  Allow the bounded drain deadline to finish, then send SIGTERM.
+- Never start a second active instance while the old process owns the advertised
+  HTTP/UDP endpoints. On crash, fence the old process/host and verify both sockets
+  are released before replacement.
+
+### Recover
+
+- Start exactly one instance from the recorded immutable image digest and matching
+  reviewed configuration/key references. A restart intentionally loses listings,
+  observed endpoints, attempts, replay markers, and runtime-only revocations.
+- Ensure any emergency key revocation is also present in authoritative
+  provisioning. Hosts must re-register; clients must browse and start new
+  attempts. Do not restore stale ephemeral state from logs or backups.
+
+### Verify
+
+- Require live and ready health, UDP bind, store availability, and one active
+  target. Run the full deployment smoke and confirm host re-registration begins.
+- Verify no pre-restart listing or capability is accepted, runtime revocations
+  that should persist are configuration-backed, and latency/outcomes stabilize.
+
+## Release rollback
+
+### Detect
+
+- Trigger rollback from a predeclared objective: readiness loss, failed deployment
+  smoke, contract/package incompatibility, security regression, direct-success
+  regression beyond threshold, or sustained resource regression. Record the new
+  and previous digests and the evidence; do not move a tag.
+
+### Contain
+
+- Stop promotion and new rollout work. Drain and stop the faulty single active
+  instance, then verify both public sockets are released. Revoke affected keys or
+  principals only when the defect creates an authorization risk.
+- Preserve logs, artifacts, provenance, signatures, and the faulty release record.
+  Never overwrite or delete an immutable package/image to reuse its version.
+
+### Recover
+
+- Deploy the previous known-good image by digest with its compatible configuration
+  and key set. Do not run old and new concurrently. If configuration changed,
+  apply its reviewed down-migration before starting.
+- Publish a corrected build under a new SemVer after diagnosis; mark faulty release
+  notes withdrawn when appropriate.
+
+### Verify
+
+- Check the running image digest, live/ready health, one active target, UDP source
+  preservation, and the complete TestClient deployment smoke.
+- Confirm package/server compatibility from `GET /v1/operator/status`, hosts
+  re-register, and the rollback objective returns to baseline for the observation
+  window.
+
+## Capacity saturation
+
+### Detect
+
+- Page when `rendezvous.queue.depth` remains above 90% of the configured attempt
+  limit, lease-critical work is shed, `rendezvous.store.available` is zero, or no
+  ready instance remains. Warn at 70%, sustained `rendezvous.limiter.drops`, or
+  p95 latency above objective.
+- Compare CPU, memory, file descriptors, UDP errors, expiry churn, HTTP operation
+  rate, and typed connection outcomes with the measured
+  [capacity profile](capacity-and-resilience.md). Distinguish legitimate growth,
+  attack traffic, downstream telemetry pressure, and a regression.
+
+### Contain
+
+- Preserve lease-critical and operator reserves. Shed new browse/join work with
+  the existing typed `429`/`Retry-After` behavior; do not add an unbounded queue.
+- Apply per-tenant/source controls at the appropriate trusted boundary. If the
+  process is unstable, drain new work and recover on one replacement rather than
+  adding a second active replica; v1 state is process-local.
+
+### Recover
+
+- Remove the causal load or deploy a tested higher single-instance resource and
+  budget profile. Change CPU/memory and server limits together, using the numeric
+  gate and accelerated soak before production.
+- Long-term horizontal scaling requires a designed shared directory, replay, and
+  attempt authority. A generic load balancer is not that design.
+
+### Verify
+
+- Re-run the capacity/resilience gate at the chosen profile, then require queue,
+  limiter drops, expiry churn, latency, store health, and direct-success ratio to
+  remain within objectives through the production observation window.
+- Confirm termination still completes within `DrainDeadlineSeconds + 5` and the
+  public and operator partitions behave independently.
+
+## Privacy or telemetry incident
+
+### Detect
+
+- Trigger on any credential, token, capability, player identity, raw IP/endpoint,
+  listing ID, metadata, or caller-reported exact connection duration tied to an
+  event or identity found in logs, metrics, traces, crash reports, support systems,
+  or analytics. Aggregate HTTP/UDP duration histograms with bounded operation tags
+  are expected telemetry. Also trigger when audit data exceeds its approved 30-day
+  retention without an incident hold.
+- Identify the producing version, sink, access population, retention/replication
+  path, and time window without copying the exposed value into a new system.
+
+### Contain
+
+- Stop or filter the offending export and restrict access to affected sinks.
+  Preserve the minimum evidence under the incident process; do not take broad
+  diagnostic dumps that amplify exposure.
+- Revoke exposed reusable credentials/keys through their owning boundary. Listing
+  IDs and endpoints are not authentication secrets, but remove affected listings
+  if continued exposure creates risk. Notify privacy/security owners according to
+  applicable policy and law.
+
+### Recover
+
+- Patch the producer to the allowlisted telemetry model, test canary redaction
+  across logs/metrics/traces/output, and deploy through the immutable release
+  path. Delete or age out affected data from every sink according to approved
+  retention and legal-hold direction.
+- Replace exposed credentials and re-register hosts when necessary. Do not claim
+  that a service restart deletes copies already exported to collectors.
+
+### Verify
+
+- Search new telemetry using non-secret synthetic canaries and confirm no canary
+  or prohibited field crosses the boundary. Verify audit records contain only
+  fixed fields and fingerprints and that retention/eviction is operating.
+- Security/privacy owners confirm sink cleanup, access review, notification, and
+  monitoring closure before the incident is resolved.
+
+## Dependency or base-image upgrade
+
+### Detect
+
+- Open a reviewed change for an advisory, end-of-support date, pinned-digest
+  refresh, or planned package update. Record affected package/image, current and
+  proposed exact version/digest, advisory severity, exploitability, and required
+  deadline. Never float to `latest` as remediation.
+
+### Contain
+
+- For an actively exploited critical issue, restrict exposure or stop the service
+  under incident authority while building the fix. Revoking publisher keys does
+  not repair a vulnerable runtime. Otherwise keep the known-good release running
+  while the candidate is tested.
+
+### Recover
+
+- Update the SDK/base-image digest, lock files, license/advisory evidence, SBOM,
+  compatibility matrix, and release notes together. For LiteNetLib or a wire/API
+  change, apply the explicit version/migration policy rather than silently
+  replacing compatible bytes.
+- Run locked restore, formatting, Debug and Release builds/tests, public contract
+  and package gates, real consumer restores, reproducible artifact/image builds,
+  vulnerability scan, signatures, topology/deployment smoke, and capacity checks
+  proportional to the change. Promote the exact tested digest.
+
+### Verify
+
+- Verify signatures, provenance, checksums, SBOM contents, running digest, and
+  absence of the advisory in the shipped artifact—not merely the build host.
+- Require live/ready health, TestClient direct traffic, real consumer compatibility,
+  and normal latency/outcomes. Keep the previous digest and compatible config for
+  rollback until the observation window closes.