feat(observability): add diagnostic dashboards (#27)
This commit is contained in:
KyuubiYoru
@@ -0,0 +1,123 @@
|
||||
# Diagnostic dashboards
|
||||
|
||||
Tracking: #27
|
||||
|
||||
Rendezvous provides two deliberately separate, optional views. The public
|
||||
session diagnostic helps a player or integration operator understand safe
|
||||
session-list state using only the public browse contract. The private Grafana
|
||||
dashboard exposes aggregate operational health through authenticated metrics.
|
||||
Neither view grants operator privileges or exposes player identity, endpoints,
|
||||
credentials, capabilities, or raw session metadata beyond the explicitly
|
||||
allowlisted public browse fields.
|
||||
|
||||
## Public read-only session diagnostic
|
||||
|
||||
The static diagnostic is disabled by default. Enable it only for approved
|
||||
game/environment scopes and keep the allowlist narrow:
|
||||
|
||||
```json
|
||||
"Diagnostics": {
|
||||
"Enabled": true,
|
||||
"PollIntervalSeconds": 10,
|
||||
"MaximumRenderedSessions": 100,
|
||||
"Scopes": [
|
||||
{
|
||||
"GameId": "space-game",
|
||||
"EnvironmentId": "smoke",
|
||||
"ProtocolVersions": [1, 2],
|
||||
"Regions": ["local"]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Open `/diagnostics` on the same origin as Rendezvous. The page cannot choose a
|
||||
different backend, request private visibility, join a session, or call the
|
||||
operator surface. It renders a bounded snapshot, then applies ordered SSE
|
||||
updates. A replay reset, corrupt cursor, incomplete snapshot, transport failure,
|
||||
or deliberate reconnect returns to a fresh authoritative snapshot and bounded
|
||||
polling. Apply filter changes explicitly; **Reset filters** restores the
|
||||
configured defaults, while **Reconnect now** tests recovery without changing
|
||||
the selection.
|
||||
|
||||
The page uses semantic HTML, labelled controls, visible keyboard focus, status
|
||||
text in addition to color, a reduced-motion mode, and a 320-pixel reflow. It
|
||||
creates untrusted content with `textContent` only. The endpoint sets a restrictive
|
||||
same-origin content-security policy, denies framing, disables MIME sniffing and
|
||||
browser capabilities, and marks every asset/config response `no-store`.
|
||||
|
||||
This is a diagnostics convenience, not a game browser, management console, or
|
||||
availability monitor. Disable it independently by setting `Enabled` to `false`;
|
||||
all diagnostic paths then return `404` without affecting game traffic, metrics,
|
||||
or health endpoints.
|
||||
|
||||
## Private Prometheus and Grafana view
|
||||
|
||||
The observability overlay pins Prometheus 3.13.1 and Grafana 13.1.0 by immutable
|
||||
multi-platform image digest. Prometheus is not published to the host. Grafana is
|
||||
bound to host loopback, disables anonymous access and sign-up, and reads its
|
||||
administrator password from a file. The service and Prometheus share only the
|
||||
metrics bearer-token file. All three secrets remain ignored by Git.
|
||||
|
||||
Create independent random secrets, then start the base service and overlay:
|
||||
|
||||
```bash
|
||||
install -d -m 0700 deploy/compose/secrets deploy/observability/secrets
|
||||
umask 077
|
||||
openssl rand -out deploy/compose/secrets/signing-key 32
|
||||
openssl rand -hex 32 >deploy/observability/secrets/rendezvous-metrics-token
|
||||
openssl rand -base64 36 >deploy/observability/secrets/grafana-admin-password
|
||||
export RENDEZVOUS_UID="$(id -u)"
|
||||
export RENDEZVOUS_GID="$(id -g)"
|
||||
test "$RENDEZVOUS_UID" -ne 0
|
||||
docker compose \
|
||||
-f deploy/compose/compose.yaml \
|
||||
-f deploy/observability/compose.yaml \
|
||||
up --build --detach
|
||||
```
|
||||
|
||||
Visit `http://127.0.0.1:3000`, sign in as `rendezvous-admin`, and open the
|
||||
**Rendezvous operational overview** folder/dashboard. The provisioned panels
|
||||
cover scrape/store/drain health, listing and join capacity, HTTP volume/errors
|
||||
and p95, browse/SSE load, lease and join operations, UDP results/latency/bytes,
|
||||
admission drops, connection outcomes, pairing latency, presence/expiry state,
|
||||
signing windows, security/audit results, and process/GC/descriptor pressure.
|
||||
Capacity gauges use the approved single-process envelope of 25,000 listings and
|
||||
10,000 active attempts, with 70% warning and 90% critical thresholds. The UDP
|
||||
response series is a conservative admitted maximum, not observed egress.
|
||||
|
||||
Validate merged configuration and checked-in dashboard structure before every
|
||||
rollout:
|
||||
|
||||
```bash
|
||||
RENDEZVOUS_UID="$(id -u)" RENDEZVOUS_GID="$(id -g)" \
|
||||
docker compose \
|
||||
-f deploy/compose/compose.yaml \
|
||||
-f deploy/observability/compose.yaml \
|
||||
config --quiet
|
||||
./scripts/test-observability-assets.sh
|
||||
```
|
||||
|
||||
For a real deployment, keep Grafana on a private authenticated management
|
||||
network instead of host loopback, replace the local admin login with the
|
||||
organization's supported identity boundary, enforce TLS at the edge, and set
|
||||
retention to the approved operational period. Do not make Prometheus public.
|
||||
Provisioning is read-only so local UI edits cannot silently drift from source.
|
||||
|
||||
## Verify, rotate, and disable
|
||||
|
||||
After startup, verify the dashboard shows `UP`, store `AVAILABLE`, a nonzero
|
||||
signing window, and changing request/UDP panels during a smoke run. Confirm an
|
||||
unauthenticated `/metrics` request returns `404`, the bearer-authenticated
|
||||
collector target is healthy, Prometheus is not bound on a host port, Grafana is
|
||||
not anonymously accessible, and dashboard query labels contain no identifiers.
|
||||
|
||||
Rotate metrics access by writing a new 32-128 character token to the secret file
|
||||
with private permissions and restarting Rendezvous and Prometheus together.
|
||||
Rotate the Grafana administrator password through the same protected secret
|
||||
workflow. Delete both secret files after a disposable local run.
|
||||
|
||||
To disable aggregate observability independently, stop/remove the overlay and
|
||||
set `Rendezvous:Metrics:Enabled` to `false`; `/metrics` returns `404` and the
|
||||
core service continues. To disable only the public session diagnostic, leave the
|
||||
overlay running and set `Rendezvous:Diagnostics:Enabled` to `false`.
|
||||
@@ -2,9 +2,10 @@
|
||||
|
||||
This runbook defines the production signals and privileged controls for the
|
||||
Rendezvous service. The service emits `System.Diagnostics.Metrics` instruments
|
||||
from the `FinalFactory.Rendezvous` meter and distributed-tracing activities from
|
||||
`FinalFactory.Rendezvous.Server`. Connect those sources to the deployment's
|
||||
OpenTelemetry or equivalent collector. Do not add identifiers to metric labels.
|
||||
from the `FinalFactory.Rendezvous` meter, distributed-tracing activities from
|
||||
`FinalFactory.Rendezvous.Server`, and an optional bearer-protected Prometheus
|
||||
endpoint. Connect only a private collector network. Do not add identifiers to
|
||||
metric labels.
|
||||
|
||||
Concrete detect/contain/recover/verify procedures for abuse, key compromise,
|
||||
targeted revocation, restart, rollback, saturation, privacy incidents, and
|
||||
@@ -38,6 +39,25 @@ dependency upgrades are in the [incident and change runbooks](incident-runbooks.
|
||||
| `rendezvous.store.expiry_churn` | Cumulative natural expiry activity | none |
|
||||
| `rendezvous.store.available` | Store health (`1` available, `0` unavailable) | none |
|
||||
|
||||
The Prometheus exporter additionally exposes active/fresh/awaiting store state,
|
||||
drain state, SSE subscriber/tenant/replay gauges, bounded HTTP and UDP
|
||||
histograms, UDP ingress and conservative admitted-response budgets, signing-key
|
||||
state/window gauges, and process/.NET pressure. Its only labels are the fixed
|
||||
operation, status, result, transport, partition, action, outcome, elapsed-bucket,
|
||||
key-state, and GC-generation dimensions. Unknown or unsafe values normalize to
|
||||
`other`; identifiers, metadata, addresses, endpoints, tokens, and capabilities
|
||||
are never labels.
|
||||
|
||||
The exporter is disabled by default. Enabling `Rendezvous:Metrics:Enabled`
|
||||
requires `BearerTokenSecretReference` to be an external `env:` or absolute
|
||||
`file:` secret containing 32-128 visible ASCII bytes. Unauthorized requests get
|
||||
the same `404` as a disabled endpoint, and accepted responses are `no-store`.
|
||||
Expose `/metrics` only to the private collector network, rotate its token as a
|
||||
deployment secret, and never place the token in a URL, Compose environment
|
||||
value, dashboard, log, or issue. The checked-in Prometheus/Grafana provisioning
|
||||
and its verification procedure are in
|
||||
[diagnostic dashboards](diagnostic-dashboards.md).
|
||||
|
||||
HTTP responses include `X-Rendezvous-Correlation-ID`. It is a generated trace ID
|
||||
or random value, never a caller-supplied session or player identifier. UDP and
|
||||
HTTP activities contain operation-level data only. Logs and traces must not add
|
||||
|
||||
Reference in New Issue
Block a user