Rendezvous/docs/deployment/linux.md

# Secure single-active Linux deployment

Tracking: #17

Rendezvous v1 stores listings, observed endpoints, join attempts, replay markers,
and runtime revocations only in the process that accepted them. Deploy exactly
one active instance. A second live replica would have a different directory and
replay boundary; `SingleActiveInstance=false` is therefore rejected rather than
presented as high availability.

## Pinned container

The root `Dockerfile` uses a multi-stage .NET 10 build and pins both Microsoft
base images by multi-architecture manifest digest. The runtime is the chiseled
ASP.NET image, contains only the published server, runs as UID/GID 1654, exposes
TCP 8080 and UDP 9050 explicitly, and does not require a writable application
directory. Supply a small writable `/tmp` tmpfs because runtime libraries can
legitimately need temporary space; keep the root filesystem read-only.

From a clean checkout:

```bash
docker build --pull=false --tag finalfactory/rendezvous:local .
docker inspect --format '{{.Config.User}}' finalfactory/rendezvous:local
```

The reported user must be `1654:1654`. Digest pins make a rebuild reproducible;
updating .NET is an explicit reviewed change to the tag, digest, SDK pin, and
lock files together. Do not replace the digest with `latest` in production.

The local Compose example applies a read-only root, non-root user, no Linux
capabilities, `no-new-privileges`, bounded PIDs/files/memory/CPU, and a shutdown
grace period longer than the service drain deadline:

```bash
install -d -m 0700 deploy/compose/secrets
umask 077
openssl rand -out deploy/compose/secrets/signing-key 32
export RENDEZVOUS_UID="$(id -u)"
export RENDEZVOUS_GID="$(id -g)"
test "$RENDEZVOUS_UID" -ne 0
docker compose -f deploy/compose/compose.yaml up --build --detach
```

`deploy/compose/appsettings.Production.json` is an isolated loopback smoke
profile, not an Internet template: it deliberately opts into private advertised
endpoints and has no TLS proxy. Its random key is ignored by Git and must be
deleted after use. Its deliberately long key window only keeps this disposable
local fixture usable; production keys require short, reviewed rotation windows.
Production configuration must use its real public names and must leave
`AllowPrivatePublicEndpoints` false.

## Production topology

Use one active service behind a source-preserving edge:

```text
clients -- HTTPS/443 --> TLS reverse proxy -- HTTP/8080 --> Rendezvous
clients -- UDP/9050 -------------------------------------> Rendezvous
```

- Give the HTTPS origin and UDP endpoint stable DNS names. Set
  `PublicHttpBaseUrl` to the exact external HTTPS origin and `PublicUdpHost` /
  `PublicUdpPort` to the endpoint given to game clients.
- Terminate TLS 1.2 or newer at a maintained reverse proxy. Bind internal HTTP
  only to the private proxy network. Restrict `AllowedHosts` to the public HTTP
  host; wildcard host filtering is rejected.
- Put only the proxy's exact literal addresses in
  `Rendezvous:AbuseProtection:TrustedProxyAddresses`. Rendezvous ignores
  forwarded headers from every other source. Keep the last proxy from replacing
  the original client address and prevent direct access to TCP 8080.
- Forward UDP as UDP, without an HTTP proxy. NAT, load balancer, firewall, and
  return routing must preserve the client's source IP/port and must send replies
  from the same advertised IP/port. Many HTTP load balancers, Kubernetes ingress
  controllers, rootless container port proxies, anycast products, and generic
  L7 services cannot guarantee this. Do not deploy through one unless an actual
  host/join smoke proves both observed source and reply path. A load balancer
  must have exactly one healthy Rendezvous target.
- Permit inbound TCP 443 to the TLS proxy and UDP 9050 to Rendezvous. Permit the
  proxy to reach TCP 8080. Permit DNS, time synchronization, image/telemetry
  destinations as required by local policy, and UDP replies to client endpoints.
  Deny public TCP 8080 and every unused inbound port.

Readiness is the load-balancer gate; liveness is only a process-health signal.
Remove a draining instance from new traffic when `/health/ready` becomes 503.
Do not use liveness failure to start a second active process while the old one
still owns the public UDP address.

## Required production configuration

Production startup validates all of these before binding listeners:

- an absolute path-free HTTPS `PublicHttpBaseUrl`;
- an unambiguous public `PublicUdpHost` and port;
- `SingleActiveInstance=true`, an explicit non-wildcard `AllowedHosts`, and at
  least one exact trusted TLS-proxy address;
- a 1-30 second drain deadline whose minimum observation interval is shorter;
- at least one enabled game policy and an active scoped signing key.

Missing values produce an actionable startup error. The checked-in base file is
intentionally unsafe for Production so an accidental bare launch fails closed.

Signing keys support two external references:

- `env:NAME` reads 1-4096 bytes encoded as base64 from `NAME`;
- `file:/absolute/path` reads 1-4096 raw bytes from a non-symlink file.

Prefer a read-only container secret owned by the configured container identity.
For systemd, use a root-owned, `rendezvous`-group-owned `0440` file (or an
equivalent narrow ACL) so the non-root process can read but not replace it. A
signing key must contain at least 32 random bytes. Never put the key, publisher/operator
credential, or secret value in JSON, a command argument, an image layer, Compose
environment, logs, metrics, or source control. Configuration contains only the
reference and non-secret lifecycle metadata. A vault/KMS adapter can replace the
provider where local policy requires it.

Keep the host clock synchronized with authenticated NTP. Credential and key
windows use wall time; lease, timeout, drain, and rate-limit deadlines use a
monotonic clock. Alert on clock synchronization loss before rotating keys.

The checked-in Compose limits (one CPU and 512 MiB) are for its isolated smoke
profile, not a production capacity claim. The measured core-state candidate
uses 2 vCPU and 2 GiB with the same 128-PID/4096-descriptor ceilings; see
the [capacity and resilience gate](../operations/capacity-and-resilience.md).
Measure real traffic, then change resource limits and server budgets together.
Memory pressure or CPU throttling must not extend orchestrator termination past
`DrainDeadlineSeconds` plus five seconds.

## Graceful shutdown

SIGTERM and the authenticated operator drain both stop new registrations and
join attempts immediately. On process shutdown, HTTP and UDP remain available
long enough for existing join attempts to finish. The service exits as soon as
the minimum drain interval has elapsed and no attempts remain, or forcibly
clears all ephemeral state at the configured deadline. It then stops UDP and
HTTP listeners and exits. Configure Docker/systemd/Kubernetes termination grace
strictly longer than the service deadline; the examples use 40 seconds for a
30-second drain.

Never use SIGKILL for a normal rollout. After stopping, verify the process is
gone and neither `8080/tcp` nor `9050/udp` is bound before starting its
replacement on the same host. A crashed or force-killed process cannot drain;
clients recover through bounded retries and hosts re-register.

## systemd alternative

Publish the server for Linux, install the immutable output at `/opt/rendezvous`,
place production configuration beside the application read-only, place key
files below `/etc/rendezvous`, and install `deploy/systemd/rendezvous.service`:

```bash
dotnet publish src/FinalFactory.Rendezvous.Server \
  --configuration Release --runtime linux-x64 --self-contained false \
  --output publish/rendezvous
systemd-analyze verify deploy/systemd/rendezvous.service
sudo systemctl daemon-reload
sudo systemctl enable --now rendezvous.service
```

Create the dedicated `rendezvous` user without a login shell. Keep
`/opt/rendezvous` and `/etc/rendezvous` root-owned and non-writable by that user;
install each required key with `root:rendezvous` ownership and mode `0440`. The
unit applies the measured 2-vCPU/2-GiB core-state candidate profile plus the
same filesystem, privilege, network-family, and shutdown hardening as Compose.

## HTTP and UDP smoke

Build the diagnostic once, then exercise the actual published HTTP and UDP
paths. The test creates a public listing, sends authenticated presence and punch
traffic through UDP 9050, establishes peer-to-peer traffic, reports the outcome,
and deregisters cleanly:

```bash
dotnet build src/FinalFactory.Rendezvous.TestClient --configuration Release
./scripts/smoke-deployment.sh
```

For the local Compose profile, the script derives a ten-minute diagnostic
publisher credential from the ignored local key without printing either secret.
For production, do not copy the signing key to the smoke host. Instead inject a
short-lived, region-scoped credential through
`RENDEZVOUS_PUBLISHER_CREDENTIAL`, and set the external endpoints:

```bash
export RENDEZVOUS_PUBLISHER_CREDENTIAL='<short-lived deployment credential>'
export RENDEZVOUS_SMOKE_HTTP_URL='https://rendezvous.your-company.tld/'
export RENDEZVOUS_SMOKE_UDP_ENDPOINT='rendezvous-udp.your-company.tld:9050'
export RENDEZVOUS_SMOKE_GAME_ID='<credential game ID>'
export RENDEZVOUS_SMOKE_ENVIRONMENT_ID='<credential environment ID>'
export RENDEZVOUS_SMOKE_REGION='<credential region>'
export RENDEZVOUS_SMOKE_PROTOCOL_VERSION='<enabled protocol version>'
./scripts/smoke-deployment.sh
```

Those four scope values must match both the short-lived credential and an
enabled server policy. The defaults (`space-game`, `smoke`, `local`, protocol
`1`) are only for the checked-in local Compose profile.

The smoke fails unless both health endpoints and the complete authenticated UDP
mediation/direct-traffic flow succeed. It does not prove every consumer NAT;
run the topology harness and representative external-network tests as well.

## Restart, upgrade, rollback, and backup

Rendezvous has no durable runtime database. Restarting intentionally loses all
listings, observed endpoints, attempts, replay markers, and runtime-only
revocations. Hosts must treat registration as a renewable lease and re-register
after service recovery. Clients must re-browse and start a new bounded attempt.

Back up only reviewed configuration, policy, secret references, key material and
its custody/lifecycle records, deployment manifests, and image digest. Never
claim a backup contains live sessions or endpoints. Restore keys only through the
secret system, not into the image or repository.

For an upgrade:

1. Build and test the new pinned digest; validate configuration without starting
   a second active instance.
2. Drain and stop the current process, verify both sockets are released, then
   start the replacement on the same public endpoints.
3. Require live/readiness and HTTP+UDP smoke success; monitor host
   re-registration, error rate, and direct-connect outcomes.

For rollback, repeat the same stop-before-start sequence with the previously
recorded image digest and compatible configuration/key set. Never run old and
new versions concurrently to avoid split ephemeral state. If a wire-incompatible
change ever becomes necessary, use a new API/protocol version rather than a
rolling two-version replica set.