feat(server): add observability and operator controls (#16)

docs/operations/observability-and-operator-runbook.md

@@ -0,0 +1,142 @@
+# Observability and operator runbook
+
+This runbook defines the production signals and privileged controls for the
+Rendezvous service. The service emits `System.Diagnostics.Metrics` instruments
+from the `FinalFactory.Rendezvous` meter and distributed-tracing activities from
+`FinalFactory.Rendezvous.Server`. Connect those sources to the deployment's
+OpenTelemetry or equivalent collector. Do not add identifiers to metric labels.
+
+## Health and readiness
+
+- `GET /health/live` proves that the HTTP process can answer. It deliberately
+  remains independent of provisioning, the state store, drain state, and optional
+  listeners so an orchestrator does not restart a recoverable dependency failure.
+- `GET /health/ready` returns success only after the HTTP path is answering, the
+  required IPv4 UDP socket is bound, any configured IPv6 UDP socket is bound,
+  provisioning loaded successfully, the store is available, and drain has not
+  started. A failed check returns `503` and removes the instance from new work.
+- A graceful drain immediately makes readiness fail while liveness remains healthy.
+  Existing work may complete until the bounded store drain deadline.
+
+## Metrics and traces
+
+| Instrument | Purpose | Bounded dimensions |
+| --- | --- | --- |
+| `rendezvous.http.requests` / `rendezvous.http.duration` | HTTP volume and latency | operation, status code |
+| `rendezvous.udp.results` / `rendezvous.udp.duration` | UDP mediation volume and processing latency | frozen/litenet operation, result |
+| `rendezvous.limiter.drops` | Requests shed by admission controls | transport, fixed partition class |
+| `rendezvous.operator.authentication` | Accepted, forbidden, and rejected operator authentication | result |
+| `rendezvous.audit.events` | Privileged action outcomes | fixed action, result |
+| `rendezvous.connection.outcomes` | Client-reported direct-connect outcomes | normalized outcome, elapsed bucket |
+| `rendezvous.pairing.latency` | Time from attempt creation to successful peer introduction | none |
+| `rendezvous.queue.depth` | Active join-attempt queue depth | none |
+| `rendezvous.store.active_listings` / `active_leases` / `active_attempts` / `replay_markers` | Current ephemeral load | none |
+| `rendezvous.store.expiry_churn` | Cumulative natural expiry activity | none |
+| `rendezvous.store.available` | Store health (`1` available, `0` unavailable) | none |
+
+HTTP responses include `X-Rendezvous-Correlation-ID`. It is a generated trace ID
+or random value, never a caller-supplied session or player identifier. UDP and
+HTTP activities contain operation-level data only. Logs and traces must not add
+tokens, capabilities, session/listing IDs, player subjects, metadata, raw IP
+addresses, or endpoint values.
+
+Recommended dashboard panels are request rate and p50/p95/p99 latency by fixed
+operation, UDP result ratio, direct connection success ratio, pairing latency,
+active listings/attempts, expiry churn, limiter drops, store availability,
+operator authentication results, audit action results, and signing-key windows.
+
+## Alerts
+
+Tune thresholds from the normal production baseline, then keep these conditions
+as distinct actionable alerts:
+
+- **Signing key expiry:** page when any required signing key has less than seven
+  days before `signUntil`; escalate at 24 hours. Confirm a replacement is signing
+  and the previous key remains verify-only for the maximum credential lifetime.
+- **Authentication spike:** warn when rejected or forbidden operator authentication
+  exceeds five attempts in five minutes. Treat unexpected publisher-authentication
+  growth as a possible credential or integration incident.
+- **Direct success regression:** warn when the connected outcome ratio falls more
+  than 20% below its seven-day same-region baseline for 15 minutes, with a minimum
+  sample floor. Break down only by bounded outcome and time bucket.
+- **Saturation:** warn when queue depth remains above 70% of the configured attempt
+  limit, limiter drops are sustained, or p95 latency exceeds the service objective;
+  page at 90% or when lease-critical traffic is shed.
+- **Store degradation:** page immediately when `rendezvous.store.available` is zero
+  or readiness fails for the store. Rising expiry churn without corresponding new
+  work is a warning for stalled clients or clock/configuration mistakes.
+- **Listener/config readiness:** page when no ready instances remain. Investigate
+  UDP bind failures, a configured-but-unbound IPv6 listener, provisioning errors,
+  and unintended drain state separately.
+
+## Operator authentication and controls
+
+Operator credentials use a signing key configured with `CredentialKinds:
+["Operator"]`. Operator keys cannot be scoped to a game/environment or used for
+publisher credentials. Mint short-lived operator credentials through the trusted
+provisioning process, outside the public Rendezvous HTTP service, and grant only
+the required permission. Never place credentials in command history, URLs, logs,
+or support tickets.
+
+The application also enforces a default-deny source boundary. Configure at most
+32 exact operator source IPs in
+`Rendezvous:AbuseProtection:OperatorAllowedAddresses`; an empty list disables all
+operator HTTP access. Development permits loopback only. Production must place
+`/v1/operator/*` behind a private management listener or reverse-proxy ACL, list
+only the resulting trusted management source addresses, and block that path on
+the public edge. If forwarded headers are enabled, keep the existing exact-proxy,
+single-hop trust policy and allowlist the post-forwarding operator source. Verify
+from both an allowed management host and a denied public host before deployment.
+Denied sources are charged to the bounded general HTTP partition before credential
+or request-body processing, then receive `404`; sustained denied traffic receives
+the same typed `429` overload response as other public traffic.
+
+Operator traffic has a dedicated, bounded rate/concurrency partition and critical
+tracker-key reserve. Public browse/join saturation therefore cannot consume the
+operator control budget, while compromised management sources remain rate-limited.
+
+The OpenAPI document defines the separate `OperatorBearer` scheme. All endpoints
+are under `/v1/operator`:
+
+| Endpoint | Permission | Confirmation |
+| --- | --- | --- |
+| `GET /status` | `ReadPolicy` | none; returns aggregates, tenant status, safe key status, and audit counts |
+| `POST /listings/revoke` | `RevokePublisher` | repeat the exact listing ID in `confirmListingId` |
+| `POST /principals/revoke` | `RevokePublisher` | repeat the exact subject and choose a 1-600 second revocation lifetime |
+| `POST /keys/revoke` | `RotateKeys` | repeat the exact key ID; runtime revocation is immediate |
+| `POST /drain` | `ManagePolicy` | send the exact value `DRAIN` |
+
+Publisher credentials are rejected on this surface even if their subject resembles
+an operator. Destructive responses do not echo identifiers. The status response
+does not expose player identities, raw endpoints, session metadata, capabilities,
+or tokens. Every authenticated operator action, rejected confirmation, and
+permission denial is audited with actor and target fingerprints.
+
+Key revocation is process-local in the current single-instance store. Apply the
+same revocation to every instance, then replace configuration before restarting;
+a restart reconstructs the configured key ring. Principal revocation is bounded
+to ten minutes and removes that principal's active listings and attempts. Use
+listing revocation for one targeted session and drain before planned shutdown.
+
+## Audit retention and incident handling
+
+The in-process audit trail defaults to 10,000 entries and 30 days. It evicts the
+oldest record at capacity and purges expired records on the next write. Configure
+`Rendezvous:Audit:MaxEntries` and `RetentionDays` within their validated bounds.
+Export the structured `AuditTrail` log events through the deployment's protected
+logging pipeline when durable retention is required; the in-memory trail is not a
+durable compliance archive. Those events include only timestamps, fixed action
+fields, correlation IDs, and actor/target fingerprints.
+
+Audit records retain timestamp, fixed action/result, target kind, correlation ID,
+and 96-bit SHA-256 fingerprints of actor and target. Routine logs contain only the
+fixed action/result/target kind and correlation ID. Restrict audit access to the
+operator role, retain aggregates only as long as operationally necessary, and
+delete raw exported audit data according to the 30-day policy unless an incident
+hold is approved.
+
+During an incident: confirm readiness and store health; capture aggregate graphs
+and correlation IDs; revoke the narrowest listing, principal, or key; drain only
+when isolation is required; record the action in the incident timeline; and verify
+that direct success, limiter drops, and authentication rates return to baseline.
+Do not copy player data, endpoints, or credentials into the incident record.

docs/security/abuse-protection.md

@@ -20,6 +20,10 @@ traffic is also isolated by its authenticated scope.
    Health probes use their own source-prefix budget so public API overload cannot
    make a healthy instance fail its orchestrator probes, while health traffic is
    still bounded.
+   Operator endpoints likewise use a separate bounded rate/concurrency partition
+   backed by the critical tracker reserve. They first require an exact source IP
+   from the default-deny `OperatorAllowedAddresses` policy, so public traffic
+   cannot spend the incident-response budget.
 3. Once an endpoint has safely derived identities, it also acquires applicable
    tenant, principal or capability, and listing/attempt budgets. Secret
    capabilities are represented only by bounded SHA-256 fingerprints.
@@ -41,7 +45,8 @@ load shedding.
 
 `Rendezvous:AbuseProtection:MaxTrackedKeys` is a hard combined ceiling for rate
 and active-concurrency keys. General HTTP and UDP traffic cannot consume the
-configured `CriticalTrackedKeyReserve`; lease operations and health probes may
+configured `CriticalTrackedKeyReserve`; lease operations, health probes, and
+allowlisted operator controls may
 use that reserve but never exceed the hard ceiling. A request that would exceed
 its applicable ceiling fails closed without adding state. Fixed-window rate keys
 are cleared at the next window boundary; concurrency keys are removed as their