Rendezvous/docs/operations/observability-and-operator-runbook.md

# Observability and operator runbook

This runbook defines the production signals and privileged controls for the
Rendezvous service. The service emits `System.Diagnostics.Metrics` instruments
from the `FinalFactory.Rendezvous` meter and distributed-tracing activities from
`FinalFactory.Rendezvous.Server`. Connect those sources to the deployment's
OpenTelemetry or equivalent collector. Do not add identifiers to metric labels.

## Health and readiness

- `GET /health/live` proves that the HTTP process can answer. It deliberately
  remains independent of provisioning, the state store, drain state, and optional
  listeners so an orchestrator does not restart a recoverable dependency failure.
- `GET /health/ready` returns success only after the HTTP path is answering, the
  required IPv4 UDP socket is bound, any configured IPv6 UDP socket is bound,
  provisioning loaded successfully, the store is available, and drain has not
  started. A failed check returns `503` and removes the instance from new work.
- A graceful drain immediately makes readiness fail while liveness remains healthy.
  Existing work may complete until the bounded store drain deadline.

## Metrics and traces

| Instrument | Purpose | Bounded dimensions |
| --- | --- | --- |
| `rendezvous.http.requests` / `rendezvous.http.duration` | HTTP volume and latency | operation, status code |
| `rendezvous.udp.results` / `rendezvous.udp.duration` | UDP mediation volume and processing latency | frozen/litenet operation, result |
| `rendezvous.limiter.drops` | Requests shed by admission controls | transport, fixed partition class |
| `rendezvous.operator.authentication` | Accepted, forbidden, and rejected operator authentication | result |
| `rendezvous.audit.events` | Privileged action outcomes | fixed action, result |
| `rendezvous.connection.outcomes` | Client-reported direct-connect outcomes | normalized outcome, elapsed bucket |
| `rendezvous.pairing.latency` | Time from attempt creation to successful peer introduction | none |
| `rendezvous.queue.depth` | Active join-attempt queue depth | none |
| `rendezvous.store.active_listings` / `active_leases` / `active_attempts` / `replay_markers` | Current ephemeral load | none |
| `rendezvous.store.expiry_churn` | Cumulative natural expiry activity | none |
| `rendezvous.store.available` | Store health (`1` available, `0` unavailable) | none |

HTTP responses include `X-Rendezvous-Correlation-ID`. It is a generated trace ID
or random value, never a caller-supplied session or player identifier. UDP and
HTTP activities contain operation-level data only. Logs and traces must not add
tokens, capabilities, session/listing IDs, player subjects, metadata, raw IP
addresses, or endpoint values.

Recommended dashboard panels are request rate and p50/p95/p99 latency by fixed
operation, UDP result ratio, direct connection success ratio, pairing latency,
active listings/attempts, expiry churn, limiter drops, store availability,
operator authentication results, audit action results, and signing-key windows.

## Alerts

Tune thresholds from the normal production baseline, then keep these conditions
as distinct actionable alerts:

- **Signing key expiry:** page when any required signing key has less than seven
  days before `signUntil`; escalate at 24 hours. Confirm a replacement is signing
  and the previous key remains verify-only for the maximum credential lifetime.
- **Authentication spike:** warn when rejected or forbidden operator authentication
  exceeds five attempts in five minutes. Treat unexpected publisher-authentication
  growth as a possible credential or integration incident.
- **Direct success regression:** warn when the connected outcome ratio falls more
  than 20% below its seven-day same-region baseline for 15 minutes, with a minimum
  sample floor. Break down only by bounded outcome and time bucket.
- **Saturation:** warn when queue depth remains above 70% of the configured attempt
  limit, limiter drops are sustained, or p95 latency exceeds the service objective;
  page at 90% or when lease-critical traffic is shed.
- **Store degradation:** page immediately when `rendezvous.store.available` is zero
  or readiness fails for the store. Rising expiry churn without corresponding new
  work is a warning for stalled clients or clock/configuration mistakes.
- **Listener/config readiness:** page when no ready instances remain. Investigate
  UDP bind failures, a configured-but-unbound IPv6 listener, provisioning errors,
  and unintended drain state separately.

## Operator authentication and controls

Operator credentials use a signing key configured with `CredentialKinds:
["Operator"]`. Operator keys cannot be scoped to a game/environment or used for
publisher credentials. Mint short-lived operator credentials through the trusted
provisioning process, outside the public Rendezvous HTTP service, and grant only
the required permission. Never place credentials in command history, URLs, logs,
or support tickets.

The application also enforces a default-deny source boundary. Configure at most
32 exact operator source IPs in
`Rendezvous:AbuseProtection:OperatorAllowedAddresses`; an empty list disables all
operator HTTP access. Development permits loopback only. Production must place
`/v1/operator/*` behind a private management listener or reverse-proxy ACL, list
only the resulting trusted management source addresses, and block that path on
the public edge. If forwarded headers are enabled, keep the existing exact-proxy,
single-hop trust policy and allowlist the post-forwarding operator source. Verify
from both an allowed management host and a denied public host before deployment.
Denied sources are charged to the bounded general HTTP partition before credential
or request-body processing, then receive `404`; sustained denied traffic receives
the same typed `429` overload response as other public traffic.

Operator traffic has a dedicated, bounded rate/concurrency partition and critical
tracker-key reserve. Public browse/join saturation therefore cannot consume the
operator control budget, while compromised management sources remain rate-limited.

The OpenAPI document defines the separate `OperatorBearer` scheme. All endpoints
are under `/v1/operator`:

| Endpoint | Permission | Confirmation |
| --- | --- | --- |
| `GET /status` | `ReadPolicy` | none; returns aggregates, tenant status, safe key status, and audit counts |
| `POST /listings/revoke` | `RevokePublisher` | repeat the exact listing ID in `confirmListingId` |
| `POST /principals/revoke` | `RevokePublisher` | repeat the exact subject and choose a 1-600 second revocation lifetime |
| `POST /keys/revoke` | `RotateKeys` | repeat the exact key ID; runtime revocation is immediate |
| `POST /drain` | `ManagePolicy` | send the exact value `DRAIN` |

Publisher credentials are rejected on this surface even if their subject resembles
an operator. Destructive responses do not echo identifiers. The status response
does not expose player identities, raw endpoints, session metadata, capabilities,
or tokens. Every authenticated operator action, rejected confirmation, and
permission denial is audited with actor and target fingerprints.

Key revocation is process-local in the current single-instance store. Apply the
same revocation to every instance, then replace configuration before restarting;
a restart reconstructs the configured key ring. Principal revocation is bounded
to ten minutes and removes that principal's active listings and attempts. Use
listing revocation for one targeted session and drain before planned shutdown.

## Audit retention and incident handling

The in-process audit trail defaults to 10,000 entries and 30 days. It evicts the
oldest record at capacity and purges expired records on the next write. Configure
`Rendezvous:Audit:MaxEntries` and `RetentionDays` within their validated bounds.
Export the structured `AuditTrail` log events through the deployment's protected
logging pipeline when durable retention is required; the in-memory trail is not a
durable compliance archive. Those events include only timestamps, fixed action
fields, correlation IDs, and actor/target fingerprints.

Audit records retain timestamp, fixed action/result, target kind, correlation ID,
and 96-bit SHA-256 fingerprints of actor and target. Routine logs contain only the
fixed action/result/target kind and correlation ID. Restrict audit access to the
operator role, retain aggregates only as long as operationally necessary, and
delete raw exported audit data according to the 30-day policy unless an incident
hold is approved.

During an incident: confirm readiness and store health; capture aggregate graphs
and correlation IDs; revoke the narrowest listing, principal, or key; drain only
when isolation is required; record the action in the incident timeline; and verify
that direct success, limiter drops, and authentication rates return to baseline.
Do not copy player data, endpoints, or credentials into the incident record.