sodimo

Chapter 20 / 40

The vault

The vault is the Sodimo account-and-key source of truth. Every SaaS password, every SSH private key, every service-account JSON, every third-party API token — one vault, one recovery path, one handoff artefact. If the team ever asks “where is the credential for X?”, the answer is always the vault. If the answer is anything else, the vault is wrong and needs to be updated.

Status: Planned — activates the day the Framework Desktop is racked. Until then, credentials live in Tom’s 1Password and on paper; the migration into Vaultwarden is part of the harness bring-up checklist.

Why Vaultwarden

The choice is Vaultwarden — the Rust reimplementation of the Bitwarden server, pinned to a specific upstream tag and run as a Podman Quadlet on the harness. It is adopted, not forked, consistent with Pivot 2 (commodity tooling is adopt-as-pinned-quadlet).

Three candidates were considered:

  • Bitwarden-server (upstream). Heavy — Microsoft SQL Server, multiple services, non-trivial quadlet. Overkill for a team of five. Rejected on weight.
  • 1Password for Teams. The path of least resistance in the short term, but it is a SaaS account that would live outside the harness, outside Cloudflare Access, and outside the backup story. At handoff, it would be the one credential the team could not self-serve. Rejected on sovereignty — the vault itself cannot be a SaaS account Sodimo can be locked out of.
  • Vaultwarden. Single Rust binary, SQLite or Postgres backend, clean environment-variable configuration, first-class Bitwarden-client compatibility (web, browser extension, iOS, Android, CLI). Fits the one-repo-of-quadlets pattern exactly. Selected.

Version pin per Principle 1: a specific vaultwarden/server:1.x.y tag, committed into the sodimo/harness repo. The team’s upgrade schedule is Sodimo’s upgrade schedule, not upstream’s.

Deployment

A single quadlet plus its data volume. The deployment follows the same pattern as every other service in chapter The harness.

  • vaultwarden — the server container, pinned tag. Environment file at /etc/sodimo/secrets/vaultwarden.env carries the admin token, the SMTP relay credentials for password-reset emails, and the signups-disabled flag.
  • vaultwarden-data — a named volume on the LUKS-encrypted data LVM. Holds the SQLite database, attachment blobs, and the RSA keys Vaultwarden uses to sign client tokens.

Backing store is SQLite. A team of five with modest vault traffic does not justify Postgres; SQLite’s single-file database simplifies the backup story (one file to snapshot) and removes a moving part. If team size grows past twenty or attachment volume gets noisy, the migration to the existing openwebui-db Postgres is documented in the Vaultwarden upstream migration guide — a one-time operation, not a day-one concern.

Caddy fronts Vaultwarden with TLS using the Cloudflare-issued certificate, same pattern as OpenWebUI and Piler. The URL is vault.sodimo.eu.

Reach into the service has two paths:

  • Employees reach vault.sodimo.eu over Cloudflare Tunnel + Cloudflare Access, authenticated via Google Workspace IdP. Same shape as every other employee-facing on-prem service (see chapters Network and Cloudflare). No Tailscale required; a laptop with Google login is enough.
  • Tom reaches Vaultwarden the same way employees do — CF Tunnel + CF Access — and also has a Tailscale path for administrative recovery when CF is unreachable. Tailscale is the break-glass path, not the daily-driver path.

What’s stored

Every credential Sodimo depends on goes into one of four Vaultwarden collections:

SaaS passwords. Cloudflare admin, Google Workspace admin, GitHub organisation owner, Anthropic console, the Framework Desktop’s Fedora-login, ISP portal, Synology NAS admin, Sodiwin VPN, WhatsApp Business manager, Stripe (if ever enabled). Each entry has the URL, username, password, TOTP seed if applicable, and a short free-text note explaining what the account is for and who set it up.

SSH private keys. Tom’s keys for the harness, Rani’s key (when issued), Paul’s key (when issued), a break-glass key for the Framework Desktop stored as an attachment in a sealed recovery item. Public keys live in the relevant authorized_keys files in the harness repo — only private keys are in the vault.

Service-account JSONs. Google Workspace service-account JSON for Cloudflare Access, Cloudflare API tokens (scoped per use — one for cloudflared, one for CI, one for manual admin), GitHub Actions deploy keys, WhatsApp Cloud API credentials, any future third-party integration keys. Each JSON is stored as an attachment on an item named svc-<service>-<purpose>.

API keys for third-party integrations. Anthropic API key, OpenAI key if ever provisioned, Tavily key, Mistral OCR key, any other LLM-adjacent provider. These get their own collection because they rotate more frequently than SaaS passwords and need per-key rotation tracking.

Nothing lives outside the vault except the three credentials needed to reach the vault in the first place — see the Recovery section below.

Rotation schedule

Rotation is per-class, driven by the class of credential rather than calendar date. A reminder runs against the vault monthly and surfaces any item past its class cadence.

  • SSH keys — 90 days. Tom, Rani, Paul. Emergency rotation on laptop loss. Public keys in the harness repo are updated in the same commit.
  • Admin passwords (Cloudflare, Google Workspace, GitHub org) — 180 days. Each with a TOTP bound to the work-phone authenticator. Rotation is scheduled, not reactive.
  • Service tokens (Cloudflare API tokens, GitHub deploy keys, Anthropic API key) — on employee offboarding, and every 365 days otherwise. The annual rotation is a forcing function to audit whether the token is still in use.
  • Vaultwarden admin token — 365 days or on any suspicion of compromise.
  • Break-glass items (recovery key, emergency SSH, safe-combination) — never rotated automatically. Rotated only on a specific incident — loss of the office safe, loss of the recovery envelope, departure of the sole master-vault operator.

Rotation events are logged as a Cloudflare dashboard row — run_ledger has a surface=vault_rotation entry per rotation, so the audit trail lives in the same D1 table as every other AI run.

Access model

Four roles, distinct powers.

  • Master operator (Tom, transferring to Michel at handoff). Holds the Vaultwarden admin-token in a paper envelope in the office safe. Can create, delete, and re-grant collections. One human at a time.
  • Writers (Tom, Paul at handoff). Full read and write on all collections. Can add, edit, and delete items but cannot change collection membership.
  • Readers (Rani, Jack, future employees). Per-collection read-only access, scoped to what their role needs. Rani gets the CRM-related SaaS entries; Jack gets the networking and ISP entries; the LLM API keys are writers-only.
  • Service principals. No human Vaultwarden login. Services that need a credential at runtime read it from /etc/sodimo/secrets/<service>.env on the harness — those env files are hydrated at deploy time from the vault via a Vaultwarden-CLI (bw) script that runs once at bring-up and once after each rotation. No long-lived Vaultwarden session on the box.

Authentication is e-mail + master password + TOTP. E-mails are Google Workspace addresses, so the Google Workspace SSO posture inherits — a disabled Workspace account cannot log into Vaultwarden either.

Recovery

The vault is the one system that cannot be recovered from the vault. The recovery path is paper, deliberate, and rehearsed.

Three items live in a single sealed envelope in the office safe:

  1. Vaultwarden admin token — the bootstrap token that grants master access even if the admin account’s TOTP device is lost.
  2. Recovery phrase for the master-operator account — printed at Vaultwarden-setup time, never stored digitally.
  3. Safe combination + Framework Desktop bios password + LUKS passphrase — the physical-layer credentials needed to reach the bits the vault is stored in.

Opening the envelope is a logged event — a Signal message to Michel and Tom before and after, logged in the same run_ledger with surface=vault_recovery. The envelope is resealed and re-stored after use; the items are rotated within seven days of opening.

An annual dry-run rehearses the recovery: pretend Tom is unreachable, have Michel open the envelope, bring up Vaultwarden from backup on a clean laptop, verify access. If the dry-run fails, the rehearsal is the incident and the fix ships that week.

Backup

Two-tier backup, same pattern as the rest of the harness data.

  • Nightly, to the NAS. The Vaultwarden SQLite file and the attachments/ directory are snapshotted to SAUVE_DISK on the 192.168.0.x network at 02:15, fifteen minutes before the Sodiwin dump. Retention: 30 daily snapshots, rolling.
  • Weekly, to Cloudflare R2. Every Sunday 03:00, the latest NAS snapshot is encrypted with an age-based symmetric key (the key itself is in the vault — the backup is unreadable without the vault, which is recovered from paper) and uploaded to the sodimo-backups R2 bucket. Retention: 12 weekly snapshots, rolling.

The loop is deliberate: backups are encrypted with a key from the vault, the vault is recovered from paper, the paper is in the safe. No single medium loss is terminal.

Handoff note

When the engagement ends, the next operator finds the vault by reading this chapter and the operator-handover document in the safe. The handover document lists:

  • The physical location of the office safe and its combination (sealed envelope — opened only on handoff).
  • The Vaultwarden admin token and recovery phrase (second sealed envelope, inside the safe).
  • The URL vault.sodimo.eu and the Cloudflare Access policy that gates it.
  • The step-by-step first-login ritual: open the envelope, log in with the master account, verify access to all four collections, rotate the admin token, re-seal and re-store the envelope with the new token.

The handover is not a file on a laptop — it is a physical ritual. The harness is reproducible from a git repo; the vault is reproducible from the safe. Both are required; neither alone is sufficient.

If the next operator cannot find the safe, the envelope, or the master password — the engagement failed the handoff test. That is the explicit success criterion for this chapter.