feat(deployment): add secure Linux runtime (#17)
This commit is contained in:
@@ -0,0 +1,225 @@
|
||||
# Secure single-active Linux deployment
|
||||
|
||||
Tracking: #17
|
||||
|
||||
Rendezvous v1 stores listings, observed endpoints, join attempts, replay markers,
|
||||
and runtime revocations only in the process that accepted them. Deploy exactly
|
||||
one active instance. A second live replica would have a different directory and
|
||||
replay boundary; `SingleActiveInstance=false` is therefore rejected rather than
|
||||
presented as high availability.
|
||||
|
||||
## Pinned container
|
||||
|
||||
The root `Dockerfile` uses a multi-stage .NET 10 build and pins both Microsoft
|
||||
base images by multi-architecture manifest digest. The runtime is the chiseled
|
||||
ASP.NET image, contains only the published server, runs as UID/GID 1654, exposes
|
||||
TCP 8080 and UDP 9050 explicitly, and does not require a writable application
|
||||
directory. Supply a small writable `/tmp` tmpfs because runtime libraries can
|
||||
legitimately need temporary space; keep the root filesystem read-only.
|
||||
|
||||
From a clean checkout:
|
||||
|
||||
```bash
|
||||
docker build --pull=false --tag finalfactory/rendezvous:local .
|
||||
docker inspect --format '{{.Config.User}}' finalfactory/rendezvous:local
|
||||
```
|
||||
|
||||
The reported user must be `1654:1654`. Digest pins make a rebuild reproducible;
|
||||
updating .NET is an explicit reviewed change to the tag, digest, SDK pin, and
|
||||
lock files together. Do not replace the digest with `latest` in production.
|
||||
|
||||
The local Compose example applies a read-only root, non-root user, no Linux
|
||||
capabilities, `no-new-privileges`, bounded PIDs/files/memory/CPU, and a shutdown
|
||||
grace period longer than the service drain deadline:
|
||||
|
||||
```bash
|
||||
install -d -m 0700 deploy/compose/secrets
|
||||
umask 077
|
||||
openssl rand -out deploy/compose/secrets/signing-key 32
|
||||
export RENDEZVOUS_UID="$(id -u)"
|
||||
export RENDEZVOUS_GID="$(id -g)"
|
||||
test "$RENDEZVOUS_UID" -ne 0
|
||||
docker compose -f deploy/compose/compose.yaml up --build --detach
|
||||
```
|
||||
|
||||
`deploy/compose/appsettings.Production.json` is an isolated loopback smoke
|
||||
profile, not an Internet template: it deliberately opts into private advertised
|
||||
endpoints and has no TLS proxy. Its random key is ignored by Git and must be
|
||||
deleted after use. Its deliberately long key window only keeps this disposable
|
||||
local fixture usable; production keys require short, reviewed rotation windows.
|
||||
Production configuration must use its real public names and must leave
|
||||
`AllowPrivatePublicEndpoints` false.
|
||||
|
||||
## Production topology
|
||||
|
||||
Use one active service behind a source-preserving edge:
|
||||
|
||||
```text
|
||||
clients -- HTTPS/443 --> TLS reverse proxy -- HTTP/8080 --> Rendezvous
|
||||
clients -- UDP/9050 -------------------------------------> Rendezvous
|
||||
```
|
||||
|
||||
- Give the HTTPS origin and UDP endpoint stable DNS names. Set
|
||||
`PublicHttpBaseUrl` to the exact external HTTPS origin and `PublicUdpHost` /
|
||||
`PublicUdpPort` to the endpoint given to game clients.
|
||||
- Terminate TLS 1.2 or newer at a maintained reverse proxy. Bind internal HTTP
|
||||
only to the private proxy network. Restrict `AllowedHosts` to the public HTTP
|
||||
host; wildcard host filtering is rejected.
|
||||
- Put only the proxy's exact literal addresses in
|
||||
`Rendezvous:AbuseProtection:TrustedProxyAddresses`. Rendezvous ignores
|
||||
forwarded headers from every other source. Keep the last proxy from replacing
|
||||
the original client address and prevent direct access to TCP 8080.
|
||||
- Forward UDP as UDP, without an HTTP proxy. NAT, load balancer, firewall, and
|
||||
return routing must preserve the client's source IP/port and must send replies
|
||||
from the same advertised IP/port. Many HTTP load balancers, Kubernetes ingress
|
||||
controllers, rootless container port proxies, anycast products, and generic
|
||||
L7 services cannot guarantee this. Do not deploy through one unless an actual
|
||||
host/join smoke proves both observed source and reply path. A load balancer
|
||||
must have exactly one healthy Rendezvous target.
|
||||
- Permit inbound TCP 443 to the TLS proxy and UDP 9050 to Rendezvous. Permit the
|
||||
proxy to reach TCP 8080. Permit DNS, time synchronization, image/telemetry
|
||||
destinations as required by local policy, and UDP replies to client endpoints.
|
||||
Deny public TCP 8080 and every unused inbound port.
|
||||
|
||||
Readiness is the load-balancer gate; liveness is only a process-health signal.
|
||||
Remove a draining instance from new traffic when `/health/ready` becomes 503.
|
||||
Do not use liveness failure to start a second active process while the old one
|
||||
still owns the public UDP address.
|
||||
|
||||
## Required production configuration
|
||||
|
||||
Production startup validates all of these before binding listeners:
|
||||
|
||||
- an absolute path-free HTTPS `PublicHttpBaseUrl`;
|
||||
- an unambiguous public `PublicUdpHost` and port;
|
||||
- `SingleActiveInstance=true`, an explicit non-wildcard `AllowedHosts`, and at
|
||||
least one exact trusted TLS-proxy address;
|
||||
- a 1-30 second drain deadline whose minimum observation interval is shorter;
|
||||
- at least one enabled game policy and an active scoped signing key.
|
||||
|
||||
Missing values produce an actionable startup error. The checked-in base file is
|
||||
intentionally unsafe for Production so an accidental bare launch fails closed.
|
||||
|
||||
Signing keys support two external references:
|
||||
|
||||
- `env:NAME` reads 1-4096 bytes encoded as base64 from `NAME`;
|
||||
- `file:/absolute/path` reads 1-4096 raw bytes from a non-symlink file.
|
||||
|
||||
Prefer a read-only container secret owned by the configured container identity.
|
||||
For systemd, use a root-owned, `rendezvous`-group-owned `0440` file (or an
|
||||
equivalent narrow ACL) so the non-root process can read but not replace it. A
|
||||
signing key must contain at least 32 random bytes. Never put the key, publisher/operator
|
||||
credential, or secret value in JSON, a command argument, an image layer, Compose
|
||||
environment, logs, metrics, or source control. Configuration contains only the
|
||||
reference and non-secret lifecycle metadata. A vault/KMS adapter can replace the
|
||||
provider where local policy requires it.
|
||||
|
||||
Keep the host clock synchronized with authenticated NTP. Credential and key
|
||||
windows use wall time; lease, timeout, drain, and rate-limit deadlines use a
|
||||
monotonic clock. Alert on clock synchronization loss before rotating keys.
|
||||
|
||||
Start with the Compose limits (one CPU, 512 MiB, 128 PIDs, 4096 descriptors),
|
||||
measure real traffic, then change the limits and the server budgets together.
|
||||
Memory pressure or CPU throttling must not extend orchestrator termination past
|
||||
`DrainDeadlineSeconds` plus five seconds.
|
||||
|
||||
## Graceful shutdown
|
||||
|
||||
SIGTERM and the authenticated operator drain both stop new registrations and
|
||||
join attempts immediately. On process shutdown, HTTP and UDP remain available
|
||||
long enough for existing join attempts to finish. The service exits as soon as
|
||||
the minimum drain interval has elapsed and no attempts remain, or forcibly
|
||||
clears all ephemeral state at the configured deadline. It then stops UDP and
|
||||
HTTP listeners and exits. Configure Docker/systemd/Kubernetes termination grace
|
||||
strictly longer than the service deadline; the examples use 40 seconds for a
|
||||
30-second drain.
|
||||
|
||||
Never use SIGKILL for a normal rollout. After stopping, verify the process is
|
||||
gone and neither `8080/tcp` nor `9050/udp` is bound before starting its
|
||||
replacement on the same host. A crashed or force-killed process cannot drain;
|
||||
clients recover through bounded retries and hosts re-register.
|
||||
|
||||
## systemd alternative
|
||||
|
||||
Publish the server for Linux, install the immutable output at `/opt/rendezvous`,
|
||||
place production configuration beside the application read-only, place key
|
||||
files below `/etc/rendezvous`, and install `deploy/systemd/rendezvous.service`:
|
||||
|
||||
```bash
|
||||
dotnet publish src/FinalFactory.Rendezvous.Server \
|
||||
--configuration Release --runtime linux-x64 --self-contained false \
|
||||
--output publish/rendezvous
|
||||
systemd-analyze verify deploy/systemd/rendezvous.service
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now rendezvous.service
|
||||
```
|
||||
|
||||
Create the dedicated `rendezvous` user without a login shell. Keep
|
||||
`/opt/rendezvous` and `/etc/rendezvous` root-owned and non-writable by that user;
|
||||
install each required key with `root:rendezvous` ownership and mode `0440`. The
|
||||
unit applies the same resource, filesystem, privilege, network-family, and
|
||||
shutdown constraints as Compose.
|
||||
|
||||
## HTTP and UDP smoke
|
||||
|
||||
Build the diagnostic once, then exercise the actual published HTTP and UDP
|
||||
paths. The test creates a public listing, sends authenticated presence and punch
|
||||
traffic through UDP 9050, establishes peer-to-peer traffic, reports the outcome,
|
||||
and deregisters cleanly:
|
||||
|
||||
```bash
|
||||
dotnet build src/FinalFactory.Rendezvous.TestClient --configuration Release
|
||||
./scripts/smoke-deployment.sh
|
||||
```
|
||||
|
||||
For the local Compose profile, the script derives a ten-minute diagnostic
|
||||
publisher credential from the ignored local key without printing either secret.
|
||||
For production, do not copy the signing key to the smoke host. Instead inject a
|
||||
short-lived, region-scoped credential through
|
||||
`RENDEZVOUS_PUBLISHER_CREDENTIAL`, and set the external endpoints:
|
||||
|
||||
```bash
|
||||
export RENDEZVOUS_PUBLISHER_CREDENTIAL='<short-lived deployment credential>'
|
||||
export RENDEZVOUS_SMOKE_HTTP_URL='https://rendezvous.your-company.tld/'
|
||||
export RENDEZVOUS_SMOKE_UDP_ENDPOINT='rendezvous-udp.your-company.tld:9050'
|
||||
export RENDEZVOUS_SMOKE_GAME_ID='<credential game ID>'
|
||||
export RENDEZVOUS_SMOKE_ENVIRONMENT_ID='<credential environment ID>'
|
||||
export RENDEZVOUS_SMOKE_REGION='<credential region>'
|
||||
export RENDEZVOUS_SMOKE_PROTOCOL_VERSION='<enabled protocol version>'
|
||||
./scripts/smoke-deployment.sh
|
||||
```
|
||||
|
||||
Those four scope values must match both the short-lived credential and an
|
||||
enabled server policy. The defaults (`space-game`, `smoke`, `local`, protocol
|
||||
`1`) are only for the checked-in local Compose profile.
|
||||
|
||||
The smoke fails unless both health endpoints and the complete authenticated UDP
|
||||
mediation/direct-traffic flow succeed. It does not prove every consumer NAT;
|
||||
run the topology harness and representative external-network tests as well.
|
||||
|
||||
## Restart, upgrade, rollback, and backup
|
||||
|
||||
Rendezvous has no durable runtime database. Restarting intentionally loses all
|
||||
listings, observed endpoints, attempts, replay markers, and runtime-only
|
||||
revocations. Hosts must treat registration as a renewable lease and re-register
|
||||
after service recovery. Clients must re-browse and start a new bounded attempt.
|
||||
|
||||
Back up only reviewed configuration, policy, secret references, key material and
|
||||
its custody/lifecycle records, deployment manifests, and image digest. Never
|
||||
claim a backup contains live sessions or endpoints. Restore keys only through the
|
||||
secret system, not into the image or repository.
|
||||
|
||||
For an upgrade:
|
||||
|
||||
1. Build and test the new pinned digest; validate configuration without starting
|
||||
a second active instance.
|
||||
2. Drain and stop the current process, verify both sockets are released, then
|
||||
start the replacement on the same public endpoints.
|
||||
3. Require live/readiness and HTTP+UDP smoke success; monitor host
|
||||
re-registration, error rate, and direct-connect outcomes.
|
||||
|
||||
For rollback, repeat the same stop-before-start sequence with the previously
|
||||
recorded image digest and compatible configuration/key set. Never run old and
|
||||
new versions concurrently to avoid split ephemeral state. If a wire-incompatible
|
||||
change ever becomes necessary, use a new API/protocol version rather than a
|
||||
rolling two-version replica set.
|
||||
@@ -63,8 +63,9 @@ only its public key ID/lifecycle metadata and does not require retired secret
|
||||
material to remain available.
|
||||
|
||||
Key IDs are non-secret base64url identifiers. Secret references are resolved
|
||||
through `ISecretProvider`; production supports `env:<VARIABLE>` references and
|
||||
the interface is replaceable by a deployment-specific vault/KMS adapter. The
|
||||
through `ISecretProvider`; production supports base64 `env:<VARIABLE>` and raw
|
||||
`file:/absolute/path` references to bounded non-symlink files. The interface is
|
||||
replaceable by a deployment-specific vault/KMS adapter. The
|
||||
committed development profile uses an in-memory random key identified by a
|
||||
`development:ephemeral/...` reference. It never writes key material to disk and
|
||||
all credentials become invalid when the process exits.
|
||||
@@ -74,8 +75,10 @@ all credentials become invalid when the process exits.
|
||||
`Rendezvous:Provisioning` supplies issuer, audience, clock skew, signing-key
|
||||
descriptors, and game policies. A production key reference such as
|
||||
`env:RENDEZVOUS_SIGNING_KEY_2026_01` expects that environment variable to hold at
|
||||
least 32 random bytes encoded as base64. Missing, malformed, short, inactive, or
|
||||
duplicate keys stop startup with a key-ID-only diagnostic. No game-wide secret
|
||||
least 32 random bytes encoded as base64. `file:/run/secrets/rendezvous-signing`
|
||||
expects the raw bytes in a read-only, absolute, non-symlink file. Missing,
|
||||
malformed, short, inactive, or duplicate keys stop startup with a key-ID-only
|
||||
diagnostic. No game-wide secret
|
||||
belongs in `appsettings`, source control, examples, the Client package, URLs,
|
||||
responses, logs, metrics, exceptions, or diagnostic dumps.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user