Production deployment

This deployment is for a single Linux host with rootful Docker Engine and the Docker Compose v2 plugin. Caddy terminates public TLS and proxies to the API; PostgreSQL has no published host port. Keep the host patched, restrict Docker socket access to administrators, and place database backups on storage that is separate from the Docker host.

Prerequisites#

• A DNS name for the API with working A and, if published, AAAA records.
• Inbound TCP ports 80 and 443 open to the host. UDP 443 is optional and enables HTTP/3.
• Docker Engine, Docker Compose v2, OpenSSL, and the age / age-keygen utilities on the host.
• A secure, offline location for the deployment secret bundle, backup age identity, and signing-key recovery material. Do not store private keys with the PostgreSQL backup files.

First deployment#

From the repository root, create the non-secret Compose settings:

cp infra/deployment/env.example .env
chmod 0600 .env

Set SAFEPASS_API_DOMAIN to the DNS name used by clients and SAFEPASS_TLS_EMAIL to an operations contact. Keep the API issuer and audience stable after clients have registered. The .env file contains no passwords or private keys.

Generate the secret files and OPAQUE server setup once:

sudo bash scripts/bootstrap-production-secrets.sh

The script builds the API image and creates unique PostgreSQL administrator, application, and migration credentials; a JWT signing key; an OPAQUE session encryption key; and the OPAQUE server setup. It fails if a generated secret file already exists. Preserve the complete infra/deployment/secrets directory in a separately protected secret manager or encrypted offline backup. In particular, losing or replacing OpaqueAuth__ServerSetup invalidates existing OPAQUE accounts. Do not run the bootstrap script again on an initialized deployment.

Compose file secrets are mounted as files under /run/secrets; the API reads the double-underscore file names through .NET's key-per-file configuration provider. The bootstrap script sets host file ownership for the non-root API user and the PostgreSQL initialization user. This uses the numeric IDs in the pinned .NET and PostgreSQL images and requires rootful Linux Docker. Docker Compose file-backed secrets are bind mounts, so host file permissions are the protection at rest; Docker does not encrypt those source files.

Validate and start the stack:

sudo docker compose --env-file .env -f infra/deployment/compose.yaml config --quiet
sudo docker compose --env-file .env -f infra/deployment/compose.yaml up -d --build
sudo docker compose --env-file .env -f infra/deployment/compose.yaml ps
curl --fail --show-error https://api.example.com/health/ready

Caddy obtains and renews the public certificate automatically and redirects HTTP to HTTPS. Its persistent /data volume contains certificate state. The API listens only on the private Compose networks. PostgreSQL uses a separate internal network and is not reachable from the host or Caddy. The database administrator role is available only inside the database container; the API uses a restricted application role, and migrations use a separate role. The edge network assigns Caddy a stable private address so the API trusts forwarded client IP and HTTPS headers from that proxy alone. Check that the default 172.30.0.0/24 subnet does not overlap the host LAN or another Docker network; update the subnet and both service addresses together if needed.

Use https://${SAFEPASS_API_DOMAIN} as the API URL in clients. The health endpoints are public and reveal only live or ready status. Request bodies, authorization headers, and access logs are not enabled in Caddy.

Encrypted backups#

Create an age identity on a secure machine or offline storage and keep the private identity off the SafePass host:

age-keygen -o /secure/safepass-backup-identity.txt
chmod 0600 /secure/safepass-backup-identity.txt
age-keygen -y /secure/safepass-backup-identity.txt

Use the printed age1... public key as SAFEPASS_BACKUP_AGE_RECIPIENT. Set SAFEPASS_BACKUP_DIR to a mounted backup volume or another location that is replicated off-host. The age identity is used only for encryption and decryption; it is separate from the Ed25519 signing key described below.

Create a separate Ed25519 signing key on the SafePass host. The backup service needs the private key to sign each archive, so keep it root-owned and readable only by root. Store a separately protected offline recovery copy of the private key. The public key is not secret: keep a trusted offline copy with recovery instructions, and install that known-good copy on restore hosts. Never accept a verification public key from the backup directory or from the archive itself. When rotating the signing key, retain each previous public key until all backups made with its private key have expired, and select the matching public key when verifying those archives.

sudo install -d -o root -g root -m 0700 /etc/safepass
sudo bash -c '
  set -euo pipefail
  umask 077
  test ! -e /etc/safepass/backup-signing-key.pem
  test ! -e /etc/safepass/backup-signing-public.pem
  openssl genpkey -algorithm ED25519 -out /etc/safepass/backup-signing-key.pem
  openssl pkey -in /etc/safepass/backup-signing-key.pem -pubout -out /etc/safepass/backup-signing-public.pem
  chmod 0600 /etc/safepass/backup-signing-key.pem
  chmod 0644 /etc/safepass/backup-signing-public.pem
'

Record the public-key fingerprint in offline recovery notes and verify it when installing the public key on another host:

openssl pkey -pubin -in /etc/safepass/backup-signing-public.pem -outform DER | sha256sum

The signature is detached at <archive>.sig. It is Ed25519 over the fixed domain string SafePass PostgreSQL backup signature v1, one NUL byte, and the SHA-256 digest of the encrypted archive. Hashing is streamed, so signing and verification do not load a large dump into memory. Keep each .age archive and its .age.sig sidecar together; retention removes both. The signature proves which signing key produced the archive, while age provides encryption and ciphertext integrity.

For a one-time backup:

sudo env \
  SAFEPASS_BACKUP_AGE_RECIPIENT='age1replace-with-your-public-key' \
  SAFEPASS_BACKUP_SIGNING_KEY=/etc/safepass/backup-signing-key.pem \
  SAFEPASS_BACKUP_DIR=/mnt/offsite/safepass \
  SAFEPASS_BACKUP_RETENTION_DAYS=35 \
  scripts/backup-postgres.sh

To run it daily, install the systemd timer. Replace /opt/safepass in the service file if the repository lives elsewhere, and set the age recipient, signing-key path, and backup directory in a root-owned environment file. The unit imports the private-key path from this file and exposes the key read-only inside its filesystem namespace:

sudo install -d -o root -g root -m 0700 /etc/safepass
sudo install -o root -g root -m 0600 infra/deployment/backup.env.example /etc/safepass/backup.env
sudoedit /etc/safepass/backup.env
sudo install -o root -g root -m 0644 infra/deployment/systemd/safepass-backup.service /etc/systemd/system/
sudo install -o root -g root -m 0644 infra/deployment/systemd/safepass-backup.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now safepass-backup.timer

If SAFEPASS_BACKUP_DIR is changed from /var/backups/safepass, also add the new path to ReadWritePaths in the systemd service before enabling the timer. If SAFEPASS_BACKUP_SIGNING_KEY points somewhere other than /etc/safepass/backup-signing-key.pem, update ReadOnlyPaths in the service unit to the same path before enabling the timer.

The timer runs daily at 03:17 UTC with up to 20 minutes of randomized delay. Check executions with journalctl -u safepass-backup.service and timer state with systemctl list-timers safepass-backup.timer.

Off-host copy#

An archive that lives only on the database host is not a backup. The same unit runs scripts/push-backup-offsite.sh as ExecStartPost, which copies the newest .age archive and its .sig sidecar to an rclone destination, compares SHA-1 sums, and deletes remote archives older than SAFEPASS_BACKUP_RETENTION_DAYS. Configure the destination in /etc/safepass/backup.env:

SAFEPASS_BACKUP_OFFSITE_REMOTE=safepassb2:safepass-backups

An empty value skips the step, which keeps a host without remote storage working but leaves the deployment without disaster protection. rclone keeps its credentials in ~/.config/rclone/rclone.conf (0600, root); for Backblaze B2 a bucket-scoped application key with readFiles, writeFiles, listFiles and deleteFiles is enough. The copy fails the unit when the checksum does not match, so a silent remote failure cannot leave the deployment unprotected.

sudo systemctl start safepass-backup.service
sudo journalctl -u safepass-backup.service -n 5
sudo -i rclone lsf safepassb2:safepass-backups

Missed-backup monitoring#

The backup timer cannot notice that it stopped running, that the job keeps failing, or that the backup volume is not mounted. A separate hourly check does: scripts/check-backup-freshness.sh fails when the backup directory is missing, when it holds no backup, when the newest backup is older than SAFEPASS_BACKUP_MAX_AGE_HOURS (default 26: one daily run plus the timer's randomized delay and a margin), when that archive is empty or has no .sig file, or when SAFEPASS_BACKUP_SIGNING_PUBLIC_KEY is set and the signature does not verify. It takes the backup time from the UTC timestamp in the file name, because copying to off-host storage can reset file times. It only reads file names, sizes and signatures; it never decrypts an archive.

On failure it writes one line to the journal and, if SAFEPASS_BACKUP_ALERT_COMMAND names an executable, runs it with that line as its only argument, for example a small script that posts to a chat webhook or sends mail. The message contains the host name, the backup file name and its age, never key material. Configure both settings in /etc/safepass/backup.env and install the units:

sudo install -o root -g root -m 0644 infra/deployment/systemd/safepass-backup-check.service /etc/systemd/system/
sudo install -o root -g root -m 0644 infra/deployment/systemd/safepass-backup-check.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now safepass-backup-check.timer
sudo systemctl start safepass-backup-check.service && journalctl -u safepass-backup-check.service -n 5

The check unit mounts /var/backups/safepass and /etc/safepass read-only; if the backup directory or alert hook lives elsewhere, add it to ReadOnlyPaths. Exit status 1 means a missing, stale or incomplete backup, and 2 means invalid settings. Point an external monitor at the unit's failed state too, so a host that is down entirely is also noticed.

Verify a backup can be restored#

Run this regularly with the private age identity available only to the administrator and a trusted copy of the signing public key. The script verifies the detached signature before creating a database or decrypting the archive. It then creates a randomly named temporary database, streams decryption into pg_restore (no plaintext dump file is written), checks the migration version and required SafePass tables, then drops the temporary database. Keep the public key in trusted recovery material independent of the backup storage.

sudo env SAFEPASS_BACKUP_AGE_IDENTITY=/secure/safepass-backup-identity.txt \
  SAFEPASS_BACKUP_SIGNING_PUBLIC_KEY=/etc/safepass/backup-signing-public.pem \
  scripts/verify-postgres-restore.sh /mnt/offsite/safepass/safepass-postgres-YYYYMMDDTHHMMSSZ.dump.age

The production database is not modified by this verification. It requires the PostgreSQL service to be healthy and the original deployment roles to exist.

Disaster-recovery drill#

Run a drill at least quarterly and after changes to the backup, schema or secret handling. It proves that a real backup restores into a working service, not only that its tables exist:

1. Pick the newest off-host archive and run scripts/check-backup-freshness.sh with SAFEPASS_BACKUP_SIGNING_PUBLIC_KEY set to the trusted public key.
2. Run scripts/verify-postgres-restore.sh on it (signature, decryption, schema version, required tables).
3. On a separate host or project, restore the archive as described in Disaster recovery, start the API with the original secret bundle, and confirm with a test account that sign-in works and that sync returns records which the client can decrypt.
4. Record the date, archive name, restore time and any deviation, and fix gaps in this guide.

Drill log:

Disaster recovery#

Restore onto a clean host using the original .env values and the exact secret bundle from the same deployment. Do not generate a new OPAQUE setup. Initialize the PostgreSQL volume and roles first:

sudo docker compose --env-file .env -f infra/deployment/compose.yaml up -d postgres
sudo docker compose --env-file .env -f infra/deployment/compose.yaml exec -T postgres \
  pg_isready -U safepass_admin -d safepass

Verify the selected archive with the trusted signing public key before age decryption or database restore. The public key should match the offline recovery copy and recorded fingerprint. First copy the encrypted archive into a private root-owned temporary directory; verification and decryption must use that same copy. Ensure the temporary filesystem has enough free space for the encrypted archive. The temporary files contain no plaintext database data:

sudo bash <<'SAFEPASS_RESTORE'
set -euo pipefail
umask 077
backup_file=/mnt/offsite/safepass/safepass-postgres-YYYYMMDDTHHMMSSZ.dump.age
snapshot_dir="$(mktemp -d /var/lib/safepass-restore.XXXXXXXXXX)"
trap 'rm -rf -- "$snapshot_dir"' EXIT
snapshot_file="$snapshot_dir/backup.dump.age"
signature_input="$snapshot_dir/signature-input"
cp -- "$backup_file" "$snapshot_file"
printf '%s\0' 'SafePass PostgreSQL backup signature v1' > "$signature_input"
openssl dgst -sha256 -binary "$snapshot_file" >> "$signature_input"
openssl pkeyutl -verify -rawin -pubin -inkey /etc/safepass/backup-signing-public.pem -in "$signature_input" -sigfile "$backup_file.sig"
 
set -o pipefail
age -d -i /secure/safepass-backup-identity.txt \
  "$snapshot_file" \
  | docker compose --env-file .env -f infra/deployment/compose.yaml exec -T postgres \
      pg_restore --username=safepass_admin --dbname=safepass --exit-on-error --single-transaction
SAFEPASS_RESTORE

Start the application and validate readiness:

sudo docker compose --env-file .env -f infra/deployment/compose.yaml up -d --build
curl --fail --show-error https://api.example.com/health/ready

For an in-place recovery, stop the API and proxy, preserve the existing database under a different name, and restore into a newly created empty safepass database. Keep the preserved database until login, sync, and recovery have been checked. Never restore a dump over live tables.

Website#

safe-pass.app serves the static Next.js export from apps/website. Build and publish it with:

bash scripts/deploy-site.sh

The script builds the crypto WASM binding when it is missing, runs next build in a Node container, and replaces the contents of infra/deployment/site, which Caddy mounts read-only at /srv/site. The directory is emptied in place: a bind mount follows the inode, so deleting and recreating the directory left the container serving an empty stale mount until it restarted.

Website account portal (optional)#

The account page on the SafePass website calls this API from the browser. It is off by default. To allow it, set SAFEPASS_WEB_ORIGINS in .env to the exact website origin, for example SAFEPASS_WEB_ORIGINS=https://safepass.example; separate several origins with commas. Each entry must be a bare HTTPS origin without a path; plain HTTP is accepted only for localhost, 127.0.0.1 and [::1], and any other value stops the API at startup. Browsers then receive CORS headers for those origins only, without credentials: tokens travel in the Authorization header and the API sets no cookies.

Updates and operations#

Review and test new PostgreSQL patch images before changing SAFEPASS_POSTGRES_IMAGE; PostgreSQL major-version changes require a planned database upgrade. Deploy API updates with docker compose up -d --build and verify health and client sync. Preserve the PostgreSQL data volume, Caddy /data volume, .env, and the full secret bundle according to the host's recovery policy. Rotate credentials through a reviewed database-role and application secret update; changing the OPAQUE setup is a protocol-level account migration, not an ordinary key rotation.