Rendezvous/docs/operations/incident-runbooks.md

# Incident and change runbooks

Tracking: #20

These runbooks supplement the [signal and operator reference](observability-and-operator-runbook.md).
Every procedure has four explicit gates: detect, contain, recover, and verify.
Record timestamps, the release digest, bounded aggregates, audit fingerprints,
and `X-Rendezvous-Correlation-ID` values. Never copy credentials, capabilities,
connection tickets, signing material, player identity, raw IP addresses,
endpoints, listing metadata, or full request bodies into an incident record.

Operator routes must be reachable only from an allowed management source. Use a
short-lived, least-permission operator credential minted outside Rendezvous.
Pass it to an approved operator client through protected stdin or a secret agent,
not a URL, command argument, environment-wide process launcher, shell trace, or
ticket. All request shapes and responses are defined by the generated
[OpenAPI v1 document](../api/rendezvous-v1.json).

Before an incident, keep these protected records available without depending on
the affected service: current and previous image digests, matching configuration,
key IDs and lifecycle windows (not raw key values), the game owner/on-call map,
capacity baselines, collector destinations, and a separately authorized
break-glass operator key. Test management-source allowlisting and credential
permissions at least once per release.

Use the exact versioned action shapes below. Confirmation fields deliberately
repeat the target so a stale UI selection or copy error fails closed. Responses
do not echo targets.

| Operation | JSON body |
| --- | --- |
| `POST /v1/operator/listings/revoke` | `{"listingId":"<uuid>","confirmListingId":"<same uuid>"}` |
| `POST /v1/operator/principals/revoke` | `{"subject":"<exact subject>","confirmSubject":"<same subject>","lifetimeSeconds":60}` |
| `POST /v1/operator/keys/revoke` | `{"keyId":"<key id>","confirmKeyId":"<same key id>"}` |
| `POST /v1/operator/drain` | `{"confirmation":"DRAIN"}` |

## Abuse or authentication spike

### Detect

- Alert on a baseline-relative increase in `rendezvous.limiter.drops`, HTTP/UDP
  request rate, `rendezvous.operator.authentication` rejected/forbidden results,
  registration requests by authentication status, queue depth, or p95/p99 latency.
- Check `/health/live`, `/health/ready`, `rendezvous.store.available`, and
  authenticated `GET /v1/operator/status`. Separate public-source rejection,
  publisher credential failure, operator probing, and ordinary capacity growth.
- Use only bounded operation/result dimensions and correlation IDs. Do not group
  by raw address, token, subject, listing ID, or metadata.

### Contain

- Preserve the dedicated operator partition. Do not raise public limits during
  an active spike. Apply source-preserving edge rate controls only when their
  collateral effect is understood and UDP source address/port remains intact.
- For one abusive session, call `POST /v1/operator/listings/revoke` with identical
  `listingId` and `confirmListingId`. For a confirmed publisher subject, call
  `POST /v1/operator/principals/revoke` with identical `subject` and
  `confirmSubject` and a 1–600 second lifetime.
- Revoke a signing key only when compromise evidence implicates that issuer;
  broad key revocation invalidates every credential signed by it. Drain only if
  the process itself must be isolated.

### Recover

- Correct the source integration, edge rule, leaked principal grant, or tenant
  budget under change control. Let a bounded principal revocation expire only
  after the owner confirms remediation; a repeated shorter revocation never
  shortens the original deadline.
- Restore normal limits gradually. If saturation caused state churn, allow leases
  and attempts to expire naturally rather than deleting arbitrary state.

### Verify

- Require limiter drops, authentication result ratios, queue depth, latency, and
  direct-connect outcomes to return to the same-region baseline for the agreed
  observation window.
- Confirm readiness stayed healthy or recovered, operator audit contains the
  intended action/result fingerprint, revoked resources cannot create new work,
  and unaffected tenants can still publish, browse, and connect.

## Signing key or issuer compromise

### Detect

- Treat secret-manager access alerts, unexpected issuance, credentials outside
  the expected region/kind, a signing-key expiry alarm, or unexplained publisher
  authentication growth as compromise until disproved.
- Identify the non-secret key ID, allowed credential kinds, game/environment
  binding, `NotBefore`, `SignUntil`, and `VerifyUntil`. Do not retrieve or paste
  raw material merely to compare it.

### Contain

- Stop the affected external issuer and deny further access to its secret.
- From a separate uncompromised break-glass operator key with `RotateKeys`, call
  `POST /v1/operator/keys/revoke` with identical `keyId` and `confirmKeyId`.
  Runtime revocation is immediate but process-local.
- Remove or mark the key revoked in authoritative provisioning before any
  restart. Revoke affected principals/listings when narrower evidence supports
  it. Do not drain automatically unless the running instance cannot be trusted.

### Recover

- Generate replacement material in the approved secret boundary, use a new key
  ID, bind it to the exact credential kind and tenant, and deploy configuration
  referencing the secret—never the secret value.
- Resume issuance with short lifetimes. Reissue only to authenticated workloads.
  When confidentiality is lost, do not use normal overlap to keep compromised
  credentials valid; document the intentional invalidation window.
- Rotate any release, registry, or operator credential exposed by the same
  incident through its owning system; Rendezvous key revocation cannot revoke
  unrelated systems.

### Verify

- Confirm `GET /v1/operator/status` shows the compromised key revoked and the
  replacement signing, old credentials fail, new exact-scope credentials work,
  and the result survives a controlled restart from updated provisioning.
- Pass TestClient registration, browse, authenticated mediation, and direct
  traffic with the replacement; monitor authentication and audit results through
  at least the maximum newly issued credential lifetime.

## Targeted listing or publisher revocation

### Detect

- Validate the abuse report against game-owned records and bounded Rendezvous
  evidence. Determine whether the target is one listing or an authenticated
  publisher subject. Do not use display name, metadata, or a raw address as
  identity.
- Confirm current aggregate state through `GET /v1/operator/status` and record
  the correlation IDs that justified action.

### Contain

- Revoke one listing with `POST /v1/operator/listings/revoke`; the exact listing
  UUID must appear in both confirmation fields.
- Revoke a publisher with `POST /v1/operator/principals/revoke`; the exact subject
  must appear in both confirmation fields and `lifetimeSeconds` must be 1–600.
  This removes that principal's active listings and attempts and blocks new ones
  for the bounded lifetime.
- Choose the narrowest action. Do not revoke a tenant key for a single listing.

### Recover

- The game owner resolves the ban, account, workload, or configuration issue in
  the authoritative game system. Rendezvous does not own user accounts or bans.
- After the original revocation deadline, permit a newly authenticated publisher
  to register. There is no un-revoke endpoint and no recovery of removed
  ephemeral listings; the host creates a new listing.

### Verify

- Confirm the old listing is no longer browsable or joinable, the principal
  cannot publish during its lifetime, and the audit action/result is present
  without the raw target.
- Confirm unrelated publishers in the same tenant and another tenant still pass
  publish/browse/join/direct-traffic checks.

## Planned restart or crash recovery

### Detect

- Planned restart begins with a recorded change and a healthy current baseline.
  Crash recovery begins when liveness/process state fails or both TCP 8080 and
  UDP 9050 stop answering. Distinguish dependency/readiness failure from a dead
  process; liveness deliberately remains healthy for some recoverable failures.
- Record active listing/lease/attempt aggregates. They are informational only:
  v1 has no durable runtime database to restore.

### Contain

- For a planned stop, call `POST /v1/operator/drain` with confirmation exactly
  `DRAIN`. Require readiness `503`, liveness `200`, and removal from new traffic.
  Allow the bounded drain deadline to finish, then send SIGTERM.
- Never start a second active instance while the old process owns the advertised
  HTTP/UDP endpoints. On crash, fence the old process/host and verify both sockets
  are released before replacement.

### Recover

- Start exactly one instance from the recorded immutable image digest and matching
  reviewed configuration/key references. A restart intentionally loses listings,
  observed endpoints, attempts, replay markers, and runtime-only revocations.
- Ensure any emergency key revocation is also present in authoritative
  provisioning. Hosts must re-register; clients must browse and start new
  attempts. Do not restore stale ephemeral state from logs or backups.

### Verify

- Require live and ready health, UDP bind, store availability, and one active
  target. Run the full deployment smoke and confirm host re-registration begins.
- Verify no pre-restart listing or capability is accepted, runtime revocations
  that should persist are configuration-backed, and latency/outcomes stabilize.

## Release rollback

### Detect

- Trigger rollback from a predeclared objective: readiness loss, failed deployment
  smoke, contract/package incompatibility, security regression, direct-success
  regression beyond threshold, or sustained resource regression. Record the new
  and previous digests and the evidence; do not move a tag.

### Contain

- Stop promotion and new rollout work. Drain and stop the faulty single active
  instance, then verify both public sockets are released. Revoke affected keys or
  principals only when the defect creates an authorization risk.
- Preserve logs, artifacts, provenance, signatures, and the faulty release record.
  Never overwrite or delete an immutable package/image to reuse its version.

### Recover

- Deploy the previous known-good image by digest with its compatible configuration
  and key set. Do not run old and new concurrently. If configuration changed,
  apply its reviewed down-migration before starting.
- Publish a corrected build under a new SemVer after diagnosis; mark faulty release
  notes withdrawn when appropriate.

### Verify

- Check the running image digest, live/ready health, one active target, UDP source
  preservation, and the complete TestClient deployment smoke.
- Confirm package/server compatibility from `GET /v1/operator/status`, hosts
  re-register, and the rollback objective returns to baseline for the observation
  window.

## Capacity saturation

### Detect

- Page when `rendezvous.queue.depth` remains above 90% of the configured attempt
  limit, lease-critical work is shed, `rendezvous.store.available` is zero, or no
  ready instance remains. Warn at 70%, sustained `rendezvous.limiter.drops`, or
  p95 latency above objective.
- Compare CPU, memory, file descriptors, UDP errors, expiry churn, HTTP operation
  rate, and typed connection outcomes with the measured
  [capacity profile](capacity-and-resilience.md). Distinguish legitimate growth,
  attack traffic, downstream telemetry pressure, and a regression.

### Contain

- Preserve lease-critical and operator reserves. Shed new browse/join work with
  the existing typed `429`/`Retry-After` behavior; do not add an unbounded queue.
- Apply per-tenant/source controls at the appropriate trusted boundary. If the
  process is unstable, drain new work and recover on one replacement rather than
  adding a second active replica; v1 state is process-local.

### Recover

- Remove the causal load or deploy a tested higher single-instance resource and
  budget profile. Change CPU/memory and server limits together, using the numeric
  gate and accelerated soak before production.
- Long-term horizontal scaling requires a designed shared directory, replay, and
  attempt authority. A generic load balancer is not that design.

### Verify

- Re-run the capacity/resilience gate at the chosen profile, then require queue,
  limiter drops, expiry churn, latency, store health, and direct-success ratio to
  remain within objectives through the production observation window.
- Confirm termination still completes within `DrainDeadlineSeconds + 5` and the
  public and operator partitions behave independently.

## Privacy or telemetry incident

### Detect

- Trigger on any credential, token, capability, player identity, raw IP/endpoint,
  listing ID, metadata, or caller-reported exact connection duration tied to an
  event or identity found in logs, metrics, traces, crash reports, support systems,
  or analytics. Aggregate HTTP/UDP duration histograms with bounded operation tags
  are expected telemetry. Also trigger when audit data exceeds its approved 30-day
  retention without an incident hold.
- Identify the producing version, sink, access population, retention/replication
  path, and time window without copying the exposed value into a new system.

### Contain

- Stop or filter the offending export and restrict access to affected sinks.
  Preserve the minimum evidence under the incident process; do not take broad
  diagnostic dumps that amplify exposure.
- Revoke exposed reusable credentials/keys through their owning boundary. Listing
  IDs and endpoints are not authentication secrets, but remove affected listings
  if continued exposure creates risk. Notify privacy/security owners according to
  applicable policy and law.

### Recover

- Patch the producer to the allowlisted telemetry model, test canary redaction
  across logs/metrics/traces/output, and deploy through the immutable release
  path. Delete or age out affected data from every sink according to approved
  retention and legal-hold direction.
- Replace exposed credentials and re-register hosts when necessary. Do not claim
  that a service restart deletes copies already exported to collectors.

### Verify

- Search new telemetry using non-secret synthetic canaries and confirm no canary
  or prohibited field crosses the boundary. Verify audit records contain only
  fixed fields and fingerprints and that retention/eviction is operating.
- Security/privacy owners confirm sink cleanup, access review, notification, and
  monitoring closure before the incident is resolved.

## Dependency or base-image upgrade

### Detect

- Open a reviewed change for an advisory, end-of-support date, pinned-digest
  refresh, or planned package update. Record affected package/image, current and
  proposed exact version/digest, advisory severity, exploitability, and required
  deadline. Never float to `latest` as remediation.

### Contain

- For an actively exploited critical issue, restrict exposure or stop the service
  under incident authority while building the fix. Revoking publisher keys does
  not repair a vulnerable runtime. Otherwise keep the known-good release running
  while the candidate is tested.

### Recover

- Update the SDK/base-image digest, lock files, license/advisory evidence, SBOM,
  compatibility matrix, and release notes together. For LiteNetLib or a wire/API
  change, apply the explicit version/migration policy rather than silently
  replacing compatible bytes.
- Run locked restore, formatting, Debug and Release builds/tests, public contract
  and package gates, real consumer restores, reproducible artifact/image builds,
  vulnerability scan, signatures, topology/deployment smoke, and capacity checks
  proportional to the change. Promote the exact tested digest.

### Verify

- Verify signatures, provenance, checksums, SBOM contents, running digest, and
  absence of the advisory in the shipped artifact—not merely the build host.
- Require live/ready health, TestClient direct traffic, real consumer compatibility,
  and normal latency/outcomes. Keep the previous digest and compatible config for
  rollback until the observation window closes.