feat(observability): add diagnostic dashboards (#27)

2026-07-17 00:22:46 +02:00
parent 06c4ecf8f3
commit 99885f8c8c
30 changed files with 3173 additions and 18 deletions
@@ -0,0 +1,123 @@
+# Diagnostic dashboards
+
+Tracking: #27
+
+Rendezvous provides two deliberately separate, optional views. The public
+session diagnostic helps a player or integration operator understand safe
+session-list state using only the public browse contract. The private Grafana
+dashboard exposes aggregate operational health through authenticated metrics.
+Neither view grants operator privileges or exposes player identity, endpoints,
+credentials, capabilities, or raw session metadata beyond the explicitly
+allowlisted public browse fields.
+
+## Public read-only session diagnostic
+
+The static diagnostic is disabled by default. Enable it only for approved
+game/environment scopes and keep the allowlist narrow:
+
+```json
+"Diagnostics": {
+  "Enabled": true,
+  "PollIntervalSeconds": 10,
+  "MaximumRenderedSessions": 100,
+  "Scopes": [
+    {
+      "GameId": "space-game",
+      "EnvironmentId": "smoke",
+      "ProtocolVersions": [1, 2],
+      "Regions": ["local"]
+    }
+  ]
+}
+```
+
+Open `/diagnostics` on the same origin as Rendezvous. The page cannot choose a
+different backend, request private visibility, join a session, or call the
+operator surface. It renders a bounded snapshot, then applies ordered SSE
+updates. A replay reset, corrupt cursor, incomplete snapshot, transport failure,
+or deliberate reconnect returns to a fresh authoritative snapshot and bounded
+polling. Apply filter changes explicitly; **Reset filters** restores the
+configured defaults, while **Reconnect now** tests recovery without changing
+the selection.
+
+The page uses semantic HTML, labelled controls, visible keyboard focus, status
+text in addition to color, a reduced-motion mode, and a 320-pixel reflow. It
+creates untrusted content with `textContent` only. The endpoint sets a restrictive
+same-origin content-security policy, denies framing, disables MIME sniffing and
+browser capabilities, and marks every asset/config response `no-store`.
+
+This is a diagnostics convenience, not a game browser, management console, or
+availability monitor. Disable it independently by setting `Enabled` to `false`;
+all diagnostic paths then return `404` without affecting game traffic, metrics,
+or health endpoints.
+
+## Private Prometheus and Grafana view
+
+The observability overlay pins Prometheus 3.13.1 and Grafana 13.1.0 by immutable
+multi-platform image digest. Prometheus is not published to the host. Grafana is
+bound to host loopback, disables anonymous access and sign-up, and reads its
+administrator password from a file. The service and Prometheus share only the
+metrics bearer-token file. All three secrets remain ignored by Git.
+
+Create independent random secrets, then start the base service and overlay:
+
+```bash
+install -d -m 0700 deploy/compose/secrets deploy/observability/secrets
+umask 077
+openssl rand -out deploy/compose/secrets/signing-key 32
+openssl rand -hex 32 >deploy/observability/secrets/rendezvous-metrics-token
+openssl rand -base64 36 >deploy/observability/secrets/grafana-admin-password
+export RENDEZVOUS_UID="$(id -u)"
+export RENDEZVOUS_GID="$(id -g)"
+test "$RENDEZVOUS_UID" -ne 0
+docker compose \
+  -f deploy/compose/compose.yaml \
+  -f deploy/observability/compose.yaml \
+  up --build --detach
+```
+
+Visit `http://127.0.0.1:3000`, sign in as `rendezvous-admin`, and open the
+**Rendezvous operational overview** folder/dashboard. The provisioned panels
+cover scrape/store/drain health, listing and join capacity, HTTP volume/errors
+and p95, browse/SSE load, lease and join operations, UDP results/latency/bytes,
+admission drops, connection outcomes, pairing latency, presence/expiry state,
+signing windows, security/audit results, and process/GC/descriptor pressure.
+Capacity gauges use the approved single-process envelope of 25,000 listings and
+10,000 active attempts, with 70% warning and 90% critical thresholds. The UDP
+response series is a conservative admitted maximum, not observed egress.
+
+Validate merged configuration and checked-in dashboard structure before every
+rollout:
+
+```bash
+RENDEZVOUS_UID="$(id -u)" RENDEZVOUS_GID="$(id -g)" \
+  docker compose \
+    -f deploy/compose/compose.yaml \
+    -f deploy/observability/compose.yaml \
+    config --quiet
+./scripts/test-observability-assets.sh
+```
+
+For a real deployment, keep Grafana on a private authenticated management
+network instead of host loopback, replace the local admin login with the
+organization's supported identity boundary, enforce TLS at the edge, and set
+retention to the approved operational period. Do not make Prometheus public.
+Provisioning is read-only so local UI edits cannot silently drift from source.
+
+## Verify, rotate, and disable
+
+After startup, verify the dashboard shows `UP`, store `AVAILABLE`, a nonzero
+signing window, and changing request/UDP panels during a smoke run. Confirm an
+unauthenticated `/metrics` request returns `404`, the bearer-authenticated
+collector target is healthy, Prometheus is not bound on a host port, Grafana is
+not anonymously accessible, and dashboard query labels contain no identifiers.
+
+Rotate metrics access by writing a new 32-128 character token to the secret file
+with private permissions and restarting Rendezvous and Prometheus together.
+Rotate the Grafana administrator password through the same protected secret
+workflow. Delete both secret files after a disposable local run.
+
+To disable aggregate observability independently, stop/remove the overlay and
+set `Rendezvous:Metrics:Enabled` to `false`; `/metrics` returns `404` and the
+core service continues. To disable only the public session diagnostic, leave the
+overlay running and set `Rendezvous:Diagnostics:Enabled` to `false`.