HeiKyu/Rendezvous
2
0
Fork 0
Code Issues 27 Pull Requests Actions Packages Projects Releases Wiki Activity

Add privacy-safe observability, health, audit, and operator controls #16

New Issue
Open
opened 2026-07-15 10:04:26 +02:00 by Kyuubi · 2 comments
Kyuubi commented 2026-07-15 10:04:26 +02:00
Owner
Copy Link
Copy Source

Parent: #1
Depends on: #15

Outcome

Let operators understand and control the service without turning sensitive or high-cardinality data into telemetry.

Scope

  • Add liveness/readiness, HTTP and UDP listener checks, and dependency/configuration readiness.
  • Emit structured logs, metrics, traces, and audit events with safe correlation across HTTP issuance and UDP attempts.
  • Track low-cardinality request/outcome latency, active leases/attempts, expiry churn, queue depth, limiter drops, pairing latency, store health, and reported direct outcomes.
  • Never use tokens, session/room IDs, metadata, or raw IPs as metric labels.
  • Enforce endpoint/attempt/log retention and redaction policies from #2.
  • Provide authenticated operator API/CLI for aggregate inspection, tenant/key status, listing revocation, credential revocation, and graceful drain—no web admin UI.
  • Add alert/dashboard and privacy-operation guidance.

Acceptance criteria

  • Readiness proves required configuration and HTTP/UDP binds; liveness does not depend on optional external systems.
  • Secret-log tests cover success, rejection, exceptions, debug logging, and operator actions.
  • Operator authentication is distinct from game credentials.
  • Revocations are audited and take effect within a documented bound.
  • Destructive operations require explicit identifiers/confirmation.
  • Routine inspection does not reveal raw endpoint or player data.
  • Key expiry, auth spikes, success-rate regressions, saturation, and store degradation have recommended alerts.
Parent: #1 Depends on: #15 ## Outcome Let operators understand and control the service without turning sensitive or high-cardinality data into telemetry. ## Scope - Add liveness/readiness, HTTP and UDP listener checks, and dependency/configuration readiness. - Emit structured logs, metrics, traces, and audit events with safe correlation across HTTP issuance and UDP attempts. - Track low-cardinality request/outcome latency, active leases/attempts, expiry churn, queue depth, limiter drops, pairing latency, store health, and reported direct outcomes. - Never use tokens, session/room IDs, metadata, or raw IPs as metric labels. - Enforce endpoint/attempt/log retention and redaction policies from #2. - Provide authenticated operator API/CLI for aggregate inspection, tenant/key status, listing revocation, credential revocation, and graceful drain—no web admin UI. - Add alert/dashboard and privacy-operation guidance. ## Acceptance criteria - Readiness proves required configuration and HTTP/UDP binds; liveness does not depend on optional external systems. - Secret-log tests cover success, rejection, exceptions, debug logging, and operator actions. - Operator authentication is distinct from game credentials. - Revocations are audited and take effect within a documented bound. - Destructive operations require explicit identifiers/confirmation. - Routine inspection does not reveal raw endpoint or player data. - Key expiry, auth spikes, success-rate regressions, saturation, and store degradation have recommended alerts.
Kyuubi added reference codex/issue-16-observability-operator-controls 2026-07-16 12:38:32 +02:00
Kyuubi self-assigned this 2026-07-16 12:38:32 +02:00
Kyuubi commented 2026-07-16 12:38:38 +02:00
Author
Owner
Copy Link
Copy Source

Starting issue #16 on codex/issue-16-observability-operator-controls, stacked on verified issue #15 (88ef946). I’ll implement privacy-safe telemetry, readiness/listener checks, distinct operator authentication and aggregate/read/revoke/drain controls, redaction/audit evidence, and alert/runbook guidance, then run adversarial review plus Debug/Release and real process gates before pushing.

Starting issue #16 on `codex/issue-16-observability-operator-controls`, stacked on verified issue #15 (`88ef946`). I’ll implement privacy-safe telemetry, readiness/listener checks, distinct operator authentication and aggregate/read/revoke/drain controls, redaction/audit evidence, and alert/runbook guidance, then run adversarial review plus Debug/Release and real process gates before pushing.
Kyuubi commented 2026-07-16 13:23:03 +02:00
Author
Owner
Copy Link
Copy Source

Implemented and verified on codex/issue-16-observability-operator-controls in commit be732de (feat(server): add observability and operator controls (#16)). The branch is pushed; the issue remains open until the dependency stack is landed.

Delivered:

  • readiness/liveness split covering HTTP, required IPv4 UDP, configured IPv6 UDP, provisioning, store availability, and graceful drain;
  • low-cardinality Meter/ActivitySource telemetry for HTTP/UDP latency and outcomes, active leases/attempts, queue depth, expiry churn, store health, limiter drops, pairing latency, connection outcomes, operator authentication, and audit events;
  • generated safe correlation IDs on every HTTP response and documented them in the generated OpenAPI contract;
  • bounded 30-day in-memory audit trail with fingerprinted actor/target identifiers, purge-on-read/write retention, safe structured export logs, and no tokens/endpoints/metadata/raw player IDs;
  • separate OperatorBearer OpenAPI surface with ReadPolicy, RevokePublisher, RotateKeys, and ManagePolicy authorization;
  • aggregate status, targeted listing revocation, bounded principal/grant revocation, idempotent signing-key revocation, and confirmed graceful drain;
  • exact repeated confirmations for every destructive action and complete affected-resource counts;
  • default-deny exact-source operator network policy plus a dedicated bounded operator admission partition/reserve so public overload cannot lock out incident response;
  • alert/dashboard/privacy/key-rotation/drain/incident guidance in docs/operations/observability-and-operator-runbook.md.

Adversarial review:

  • three independent API/architecture/privacy, correctness/security, and tests/performance passes are clean at P0-P2;
  • review fixes included denied-source admission, operator reserve isolation, stale idle metrics, limiter callbacks outside hot locks, purge-on-read audit retention, retry-safe key revocation, standard Bearer challenge, complete revocation impact, safe durable audit logging, and OpenAPI correlation headers.

Verification:

  • dotnet format Rendezvous.slnx --no-restore --verify-no-changes clean;
  • Debug build clean and full suite 243/243;
  • Release build clean and full suite 243/243;
  • real development server run with ephemeral HTTP/UDP binds: /health/live 200, /health/ready 200, unauthenticated /v1/operator/status 401 with Bearer realm="operator"; every response carried X-Rendezvous-Correlation-ID;
  • git diff --check clean.
Implemented and verified on `codex/issue-16-observability-operator-controls` in commit `be732de` (`feat(server): add observability and operator controls (#16)`). The branch is pushed; the issue remains open until the dependency stack is landed. Delivered: - readiness/liveness split covering HTTP, required IPv4 UDP, configured IPv6 UDP, provisioning, store availability, and graceful drain; - low-cardinality `Meter`/`ActivitySource` telemetry for HTTP/UDP latency and outcomes, active leases/attempts, queue depth, expiry churn, store health, limiter drops, pairing latency, connection outcomes, operator authentication, and audit events; - generated safe correlation IDs on every HTTP response and documented them in the generated OpenAPI contract; - bounded 30-day in-memory audit trail with fingerprinted actor/target identifiers, purge-on-read/write retention, safe structured export logs, and no tokens/endpoints/metadata/raw player IDs; - separate `OperatorBearer` OpenAPI surface with `ReadPolicy`, `RevokePublisher`, `RotateKeys`, and `ManagePolicy` authorization; - aggregate status, targeted listing revocation, bounded principal/grant revocation, idempotent signing-key revocation, and confirmed graceful drain; - exact repeated confirmations for every destructive action and complete affected-resource counts; - default-deny exact-source operator network policy plus a dedicated bounded operator admission partition/reserve so public overload cannot lock out incident response; - alert/dashboard/privacy/key-rotation/drain/incident guidance in `docs/operations/observability-and-operator-runbook.md`. Adversarial review: - three independent API/architecture/privacy, correctness/security, and tests/performance passes are clean at P0-P2; - review fixes included denied-source admission, operator reserve isolation, stale idle metrics, limiter callbacks outside hot locks, purge-on-read audit retention, retry-safe key revocation, standard Bearer challenge, complete revocation impact, safe durable audit logging, and OpenAPI correlation headers. Verification: - `dotnet format Rendezvous.slnx --no-restore --verify-no-changes` clean; - Debug build clean and full suite 243/243; - Release build clean and full suite 243/243; - real development server run with ephemeral HTTP/UDP binds: `/health/live` 200, `/health/ready` 200, unauthenticated `/v1/operator/status` 401 with `Bearer realm="operator"`; every response carried `X-Rendezvous-Correlation-ID`; - `git diff --check` clean.
Sign in to join this conversation.
codex/issue-16-observability-operator-controls
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Kyuubi schmidt.florian (Florian Schmidt)
No Assignees Kyuubi
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: HeiKyu/Rendezvous#16
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?