Let operators understand and control the service without turning sensitive or high-cardinality data into telemetry.
Scope
Add liveness/readiness, HTTP and UDP listener checks, and dependency/configuration readiness.
Emit structured logs, metrics, traces, and audit events with safe correlation across HTTP issuance and UDP attempts.
Track low-cardinality request/outcome latency, active leases/attempts, expiry churn, queue depth, limiter drops, pairing latency, store health, and reported direct outcomes.
Never use tokens, session/room IDs, metadata, or raw IPs as metric labels.
Enforce endpoint/attempt/log retention and redaction policies from #2.
Provide authenticated operator API/CLI for aggregate inspection, tenant/key status, listing revocation, credential revocation, and graceful drain—no web admin UI.
Add alert/dashboard and privacy-operation guidance.
Acceptance criteria
Readiness proves required configuration and HTTP/UDP binds; liveness does not depend on optional external systems.
Routine inspection does not reveal raw endpoint or player data.
Key expiry, auth spikes, success-rate regressions, saturation, and store degradation have recommended alerts.
Parent: #1
Depends on: #15
## Outcome
Let operators understand and control the service without turning sensitive or high-cardinality data into telemetry.
## Scope
- Add liveness/readiness, HTTP and UDP listener checks, and dependency/configuration readiness.
- Emit structured logs, metrics, traces, and audit events with safe correlation across HTTP issuance and UDP attempts.
- Track low-cardinality request/outcome latency, active leases/attempts, expiry churn, queue depth, limiter drops, pairing latency, store health, and reported direct outcomes.
- Never use tokens, session/room IDs, metadata, or raw IPs as metric labels.
- Enforce endpoint/attempt/log retention and redaction policies from #2.
- Provide authenticated operator API/CLI for aggregate inspection, tenant/key status, listing revocation, credential revocation, and graceful drain—no web admin UI.
- Add alert/dashboard and privacy-operation guidance.
## Acceptance criteria
- Readiness proves required configuration and HTTP/UDP binds; liveness does not depend on optional external systems.
- Secret-log tests cover success, rejection, exceptions, debug logging, and operator actions.
- Operator authentication is distinct from game credentials.
- Revocations are audited and take effect within a documented bound.
- Destructive operations require explicit identifiers/confirmation.
- Routine inspection does not reveal raw endpoint or player data.
- Key expiry, auth spikes, success-rate regressions, saturation, and store degradation have recommended alerts.
Starting issue #16 on codex/issue-16-observability-operator-controls, stacked on verified issue #15 (88ef946). I’ll implement privacy-safe telemetry, readiness/listener checks, distinct operator authentication and aggregate/read/revoke/drain controls, redaction/audit evidence, and alert/runbook guidance, then run adversarial review plus Debug/Release and real process gates before pushing.
Starting issue #16 on `codex/issue-16-observability-operator-controls`, stacked on verified issue #15 (`88ef946`). I’ll implement privacy-safe telemetry, readiness/listener checks, distinct operator authentication and aggregate/read/revoke/drain controls, redaction/audit evidence, and alert/runbook guidance, then run adversarial review plus Debug/Release and real process gates before pushing.
Implemented and verified on codex/issue-16-observability-operator-controls in commit be732de (feat(server): add observability and operator controls (#16)). The branch is pushed; the issue remains open until the dependency stack is landed.
Delivered:
readiness/liveness split covering HTTP, required IPv4 UDP, configured IPv6 UDP, provisioning, store availability, and graceful drain;
low-cardinality Meter/ActivitySource telemetry for HTTP/UDP latency and outcomes, active leases/attempts, queue depth, expiry churn, store health, limiter drops, pairing latency, connection outcomes, operator authentication, and audit events;
generated safe correlation IDs on every HTTP response and documented them in the generated OpenAPI contract;
bounded 30-day in-memory audit trail with fingerprinted actor/target identifiers, purge-on-read/write retention, safe structured export logs, and no tokens/endpoints/metadata/raw player IDs;
separate OperatorBearer OpenAPI surface with ReadPolicy, RevokePublisher, RotateKeys, and ManagePolicy authorization;
exact repeated confirmations for every destructive action and complete affected-resource counts;
default-deny exact-source operator network policy plus a dedicated bounded operator admission partition/reserve so public overload cannot lock out incident response;
alert/dashboard/privacy/key-rotation/drain/incident guidance in docs/operations/observability-and-operator-runbook.md.
Adversarial review:
three independent API/architecture/privacy, correctness/security, and tests/performance passes are clean at P0-P2;
review fixes included denied-source admission, operator reserve isolation, stale idle metrics, limiter callbacks outside hot locks, purge-on-read audit retention, retry-safe key revocation, standard Bearer challenge, complete revocation impact, safe durable audit logging, and OpenAPI correlation headers.
Verification:
dotnet format Rendezvous.slnx --no-restore --verify-no-changes clean;
Debug build clean and full suite 243/243;
Release build clean and full suite 243/243;
real development server run with ephemeral HTTP/UDP binds: /health/live 200, /health/ready 200, unauthenticated /v1/operator/status 401 with Bearer realm="operator"; every response carried X-Rendezvous-Correlation-ID;
git diff --check clean.
Implemented and verified on `codex/issue-16-observability-operator-controls` in commit `be732de` (`feat(server): add observability and operator controls (#16)`). The branch is pushed; the issue remains open until the dependency stack is landed.
Delivered:
- readiness/liveness split covering HTTP, required IPv4 UDP, configured IPv6 UDP, provisioning, store availability, and graceful drain;
- low-cardinality `Meter`/`ActivitySource` telemetry for HTTP/UDP latency and outcomes, active leases/attempts, queue depth, expiry churn, store health, limiter drops, pairing latency, connection outcomes, operator authentication, and audit events;
- generated safe correlation IDs on every HTTP response and documented them in the generated OpenAPI contract;
- bounded 30-day in-memory audit trail with fingerprinted actor/target identifiers, purge-on-read/write retention, safe structured export logs, and no tokens/endpoints/metadata/raw player IDs;
- separate `OperatorBearer` OpenAPI surface with `ReadPolicy`, `RevokePublisher`, `RotateKeys`, and `ManagePolicy` authorization;
- aggregate status, targeted listing revocation, bounded principal/grant revocation, idempotent signing-key revocation, and confirmed graceful drain;
- exact repeated confirmations for every destructive action and complete affected-resource counts;
- default-deny exact-source operator network policy plus a dedicated bounded operator admission partition/reserve so public overload cannot lock out incident response;
- alert/dashboard/privacy/key-rotation/drain/incident guidance in `docs/operations/observability-and-operator-runbook.md`.
Adversarial review:
- three independent API/architecture/privacy, correctness/security, and tests/performance passes are clean at P0-P2;
- review fixes included denied-source admission, operator reserve isolation, stale idle metrics, limiter callbacks outside hot locks, purge-on-read audit retention, retry-safe key revocation, standard Bearer challenge, complete revocation impact, safe durable audit logging, and OpenAPI correlation headers.
Verification:
- `dotnet format Rendezvous.slnx --no-restore --verify-no-changes` clean;
- Debug build clean and full suite 243/243;
- Release build clean and full suite 243/243;
- real development server run with ephemeral HTTP/UDP binds: `/health/live` 200, `/health/ready` 200, unauthenticated `/v1/operator/status` 401 with `Bearer realm="operator"`; every response carried `X-Rendezvous-Correlation-ID`;
- `git diff --check` clean.
Shipped in v1.0.0: health endpoints, token-gated metrics, audit events, operator controls, and the optional diagnostics dashboard are live on the foxden test deployment (dashboard enabled 2026-08-22). One telemetry gap found by the security review is tracked in #33.
Shipped in [v1.0.0](https://git.finalfactory.de/HeiKyu/Rendezvous/releases/tag/v1.0.0): health endpoints, token-gated metrics, audit events, operator controls, and the optional diagnostics dashboard are live on the foxden test deployment (dashboard enabled 2026-08-22). One telemetry gap found by the security review is tracked in #33.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Parent: #1
Depends on: #15
Outcome
Let operators understand and control the service without turning sensitive or high-cardinality data into telemetry.
Scope
Acceptance criteria
Starting issue #16 on
codex/issue-16-observability-operator-controls, stacked on verified issue #15 (88ef946). I’ll implement privacy-safe telemetry, readiness/listener checks, distinct operator authentication and aggregate/read/revoke/drain controls, redaction/audit evidence, and alert/runbook guidance, then run adversarial review plus Debug/Release and real process gates before pushing.Implemented and verified on
codex/issue-16-observability-operator-controlsin commitbe732de(feat(server): add observability and operator controls (#16)). The branch is pushed; the issue remains open until the dependency stack is landed.Delivered:
Meter/ActivitySourcetelemetry for HTTP/UDP latency and outcomes, active leases/attempts, queue depth, expiry churn, store health, limiter drops, pairing latency, connection outcomes, operator authentication, and audit events;OperatorBearerOpenAPI surface withReadPolicy,RevokePublisher,RotateKeys, andManagePolicyauthorization;docs/operations/observability-and-operator-runbook.md.Adversarial review:
Verification:
dotnet format Rendezvous.slnx --no-restore --verify-no-changesclean;/health/live200,/health/ready200, unauthenticated/v1/operator/status401 withBearer realm="operator"; every response carriedX-Rendezvous-Correlation-ID;git diff --checkclean.Shipped in v1.0.0: health endpoints, token-gated metrics, audit events, operator controls, and the optional diagnostics dashboard are live on the foxden test deployment (dashboard enabled 2026-08-22). One telemetry gap found by the security review is tracked in #33.