Rendezvous/docs/operations/diagnostic-dashboards.md

Diagnostic dashboards

Tracking: #27

Rendezvous provides two deliberately separate, optional views. The public session diagnostic helps a player or integration operator understand safe session-list state using only the public browse contract. The private Grafana dashboard exposes aggregate operational health through authenticated metrics. Neither view grants operator privileges or exposes player identity, endpoints, credentials, capabilities, or raw session metadata beyond the explicitly allowlisted public browse fields.

Public read-only session diagnostic

The static diagnostic is disabled by default. Enable it only for approved game/environment scopes and keep the allowlist narrow:

"Diagnostics": {
  "Enabled": true,
  "PollIntervalSeconds": 10,
  "MaximumRenderedSessions": 100,
  "Scopes": [
    {
      "GameId": "space-game",
      "EnvironmentId": "smoke",
      "ProtocolVersions": [1, 2],
      "Regions": ["local"]
    }
  ]
}

Open /diagnostics on the same origin as Rendezvous. The page cannot choose a different backend, request private visibility, join a session, or call the operator surface. It renders a bounded snapshot, then applies ordered SSE updates. A replay reset, corrupt cursor, incomplete snapshot, transport failure, or deliberate reconnect returns to a fresh authoritative snapshot and bounded polling. Apply filter changes explicitly; Reset filters restores the configured defaults, while Reconnect now tests recovery without changing the selection.

The page uses semantic HTML, labelled controls, visible keyboard focus, status text in addition to color, a reduced-motion mode, and a 320-pixel reflow. It creates untrusted content with textContent only. The endpoint sets a restrictive same-origin content-security policy, denies framing, disables MIME sniffing and browser capabilities, and marks every asset/config response no-store.

This is a diagnostics convenience, not a game browser, management console, or availability monitor. Disable it independently by setting Enabled to false; all diagnostic paths then return 404 without affecting game traffic, metrics, or health endpoints.

Private Prometheus and Grafana view

The observability overlay pins Prometheus 3.13.1 and Grafana 13.1.0 by immutable multi-platform image digest. Prometheus is not published to the host. Grafana is bound to host loopback, disables anonymous access and sign-up, and reads its administrator password from a file. The service and Prometheus share only the metrics bearer-token file. All three secrets remain ignored by Git.

Create independent random secrets, then start the base service and overlay:

install -d -m 0700 deploy/compose/secrets deploy/observability/secrets
umask 077
openssl rand -out deploy/compose/secrets/signing-key 32
openssl rand -hex 32 >deploy/observability/secrets/rendezvous-metrics-token
openssl rand -base64 36 >deploy/observability/secrets/grafana-admin-password
export RENDEZVOUS_UID="$(id -u)"
export RENDEZVOUS_GID="$(id -g)"
test "$RENDEZVOUS_UID" -ne 0
docker compose \
  -f deploy/compose/compose.yaml \
  -f deploy/observability/compose.yaml \
  up --build --detach

Visit http://127.0.0.1:3000, sign in as rendezvous-admin, and open the Rendezvous operational overview folder/dashboard. The provisioned panels cover scrape/store/drain health, listing and join capacity, HTTP volume/errors and p95, browse/SSE load, lease and join operations, UDP results/latency/bytes, admission drops, connection outcomes, pairing latency, presence/expiry state, signing windows, security/audit results, and process/GC/descriptor pressure. Capacity gauges use the approved single-process envelope of 25,000 listings and 10,000 active attempts, with 70% warning and 90% critical thresholds. The UDP response series is a conservative admitted maximum, not observed egress.

Validate merged configuration and checked-in dashboard structure before every rollout:

RENDEZVOUS_UID="$(id -u)" RENDEZVOUS_GID="$(id -g)" \
  docker compose \
    -f deploy/compose/compose.yaml \
    -f deploy/observability/compose.yaml \
    config --quiet
./scripts/test-observability-assets.sh

For a real deployment, keep Grafana on a private authenticated management network instead of host loopback, replace the local admin login with the organization's supported identity boundary, enforce TLS at the edge, and set retention to the approved operational period. Do not make Prometheus public. Provisioning is read-only so local UI edits cannot silently drift from source.

Verify, rotate, and disable

After startup, verify the dashboard shows UP, store AVAILABLE, a nonzero signing window, and changing request/UDP panels during a smoke run. Confirm an unauthenticated /metrics request returns 404, the bearer-authenticated collector target is healthy, Prometheus is not bound on a host port, Grafana is not anonymously accessible, and dashboard query labels contain no identifiers.

Rotate metrics access by writing a new 32-128 character token to the secret file with private permissions and restarting Rendezvous and Prometheus together. Rotate the Grafana administrator password through the same protected secret workflow. Delete both secret files after a disposable local run.

To disable aggregate observability independently, stop/remove the overlay and set Rendezvous:Metrics:Enabled to false; /metrics returns 404 and the core service continues. To disable only the public session diagnostic, leave the overlay running and set Rendezvous:Diagnostics:Enabled to false.