sodimo

Chapter 21 / 40

Quadlet reference

Every service on the harness is a Podman Quadlet under sodimo/dotfiles at home/dot_config/containers/systemd/. Chezmoi applies the tree into ~/.config/containers/systemd/; systemctl --user enable --now sodimo.target brings the stack up. This chapter is a per-quadlet lookup table — for the architecture picture, read The harness.

Env caveat · .env files under home/dot_config/sodimo/ are gitignored but required for quadlet boot; .env.tmpl companions are the vendored shape. Tracked as sodimo/dotfiles#15.

Routing posture

Two HTTP paths reach a quadlet. Every stanza below names which applies.

  • cloudflared — one named Cloudflare Tunnel runs as a host-level systemd unit (not a quadlet; pinned by the bootc image). It terminates all *.sodimo.eu hostnames and forwards plain HTTP to Caddy on 127.0.0.1:80. CF Access gates every hostname at the edge against Google Workspace.
  • Caddy — reverse-proxies from 127.0.0.1:80 into the container network by container name. TLS is terminated at Cloudflare; Caddy speaks plain HTTP internally. Per-service route files live at home/dot_config/caddy/routes/*.caddy.
  • Tailscale — break-glass only. Tom’s dev path; never an employee tool.

Every quadlet attaches to sodimo.network (10.89.0.0/24). Container-name DNS is implicit — litellm resolves from any container on the network.

Caddy

  • Upstream · caddyserver/caddy
  • Image · docker.io/library/caddy:2-alpine
  • Role · reverse proxy; fans out to containers on sodimo.network by name
  • Ports · 127.0.0.1:{80,443}
  • Routing · internal — receives from cloudflared on :80
  • Env · CADDY_ADMIN=off
  • Secrets · none (TLS terminated at CF)
  • Depends on · sodimo.network
  • Source · caddy.container

Cockpit

  • Upstream · cockpit-project/cockpit
  • Image · quay.io/cockpit/ws:latest (only :latest in the stack; slow cadence, strong compat)
  • Role · web admin UI; Paul’s day-one surface for services, journals, storage
  • Ports · 127.0.0.1:9090
  • Routing · cockpit.sodimo.eu via cloudflared → Caddy → host.containers.internal:9090
  • Env · none
  • Secrets · none (Cockpit auths against host PAM)
  • Depends on · host podman socket + systemd (privileged, --pid=host)
  • Source · cockpit.container

OpenWebUI

  • Upstream · open-webui/open-webui
  • Image · ghcr.io/open-webui/open-webui:main (pin by digest per Principle 1)
  • Role · chat UI for local + cloud-escalation models; interactive only, agents use Paperclip
  • Ports · 127.0.0.1:3000:8080
  • Routing · chat.sodimo.eu via cloudflared → Caddy → openwebui:8080
  • Env · openwebui.env — OPENAI_API_BASE_URL=http://litellm:4000 (LiteLLM is the only model backend, never llama-swap directly); WEBUI_AUTH=false, ENABLE_SIGNUP=false, DEFAULT_USER_ROLE=admin; audio STT/TTS engines blanked (D-097)
  • Secrets · WEBUI_SECRET_KEY, LiteLLM master key, Postgres password
  • Depends on · openwebui-db (pgvector:pg16), openwebui-redis (redis:7-alpine), litellm
  • Source · openwebui.container

LiteLLM

  • Upstream · BerriAI/litellm
  • Image · ghcr.io/berriai/litellm:main-stable
  • Role · single gateway from every AI caller (OpenWebUI, Paperclip, Worker MCP) to every model backend (llama-swap local, Anthropic cloud); token-accounting + fallback + cost caps in one place
  • Ports · 127.0.0.1:4000
  • Routing · internal-only
  • Env · litellm.env (master key, DB/Redis URLs, cloud API keys); litellm.yaml maps local-task/local-heavy/local-embeddings → llama-swap, claude-opus-4-7/gpt-5 → cloud
  • Secrets · LITELLM_MASTER_KEY, LITELLM_SALT_KEY, ANTHROPIC_API_KEY, OPENAI_API_KEY
  • Depends on · litellm-db (pgvector:pg16), litellm-redis, llama-swap
  • Source · litellm.container

llama-swap

  • Upstream · mostlygeek/llama-swap
  • Image · ghcr.io/mostlygeek/llama-swap:vulkan — Vulkan, not ROCm (D-166). Strix Halo’s gfx1151 iGPU is a ROCm moving target; RADV is the production-stable posture. AMDVLK for prompt-heavy / long-context is aspirational (bundled image ships only the Mesa RADV ICD).
  • Role · OpenAI-compatible front-end that hot-swaps llama.cpp instances by model name
  • Runtime · raw /app/llama-server invoked per-model from each cmd: block. The image carries no ramalama binary, so earlier ramalama --runtime llama.cpp run … invocations were latently broken; rewritten against the kyuz0 Strix-Halo reference (kyuz0/amd-strix-halo-toolboxes, see docs/kyuz0-toolbox.md).
  • Mandatory flag group · every model’s cmd: carries --no-mmap -fa on -ngl 999 --batch-size 4096 --ubatch-size 512 --cache-type-k q8_0 --cache-type-v q8_0 --jinja --direct-io --cache-prompt --cache-reuse 256 --threads 12. Sourced from kyuz0 docs/models.ini.example; deviation requires re-validating against the resync runbook.
  • Self-proxy rule · proxy: targets must be http://127.0.0.1:NNNN, never http://llama-swap:NNNN. llama-swap and the spawned llama-server share a netns; container-self DNS forces a netavark round-trip that 502s under the adguard DNAT hijack.
  • Ports · 127.0.0.1:9292:8080
  • Routing · internal-only
  • Rootless posture · GroupAdd=video only. Do not use GroupAdd=keep-groups or --group-add=render — incompatible under rootless podman 5.8.2.
  • Env · HSA_OVERRIDE_GFX_VERSION=11.0.0 (gfx1151 → gfx1100); GPU passthrough /dev/dri + /dev/kfd
  • Volume · %h/.local/share/llama-models:/models:ro — raw GGUF layout on the harness filesystem (multi-shard GGUFs like gpt-oss-120b chain automatically from shard 1). Not ramalama’s nested cache.
  • Models · llama-swap.yaml defines four:
    • qwen3-4b / local-task — Vulkan-compiled GGUF, TTL=0, group task.
    • gpt-oss-120b / local-heavy — unsloth UD-Q8_K_XL, 2-shard GGUF, 65 k ctx, reasoning_effort=high via --chat-template-kwargs. Flipped from gpt-oss-20b on e400703. Group heavy (TTL=600 s, exclusive: true).
    • qwen3-30b / qwen3-30b-coder — TTL=600 s, group heavy.
    • qwen3-embedding / local-embeddings — TTL=0, group embeddings.
  • Smoke verdict (2026-04-22) · qwen3-4b end-to-end PASS via OpenWebUI → LiteLLM → llama-swap → llama-server — 11.5 s cold, 0.08–0.20 s warm, 53 tok/s on RADV Vulkan. gpt-oss-120b config-path validated transitively; first real run gated on 100 GB GGUF download + sodimo/harness#11 kernel cmdline (iommu=pt amdgpu.gttsize=126976 ttm.pages_limit=32505856).
  • Vestiges · /dev/kfd + HSA_OVERRIDE_GFX_VERSION=11.0.0 + host podman.sock mount are unused on the Vulkan path; kept for ROCm forward-compat and tracked as investigation-only in sodimo/dotfiles#14.
  • Secrets · none
  • Depends on · sodimo.network
  • Source · llama-swap.container · llama-swap.yaml

Twenty

  • Upstream · twentyhq/twenty
  • Image · docker.io/twentycrm/twenty:v2.0.0 — fully-qualified registry path, not short-name. Short-name pulls fail for non-TTY auto-updates (bootc auto-update, podman auto-update) under short-name policy.
  • Role · Sodimo’s CRM — pipeline kanban, contacts, companies, email-thread linkage, native workflow engine. The MCP wrapper calls the Twenty REST + GraphQL API directly — no turbular (D-165).
  • Ports · 127.0.0.1:3333:3000
  • Routing · crm.sodimo.eu via cloudflared → Caddy → twenty:3000
  • Env · twenty.env — AUTH_GOOGLE_ENABLED=true (Google SSO), EMAIL_SMTP_HOST=postfix (IMAP/SMTP via Sodimo Postfix), BullMQ on twenty-redis
  • Secrets · APP_SECRET, Postgres password, Google OAuth client ID/secret, SMTP password
  • Depends on · twenty-worker (same image, yarn worker:prod), twenty-db (postgres:16), twenty-redis. First-boot migrations complete in ~18 s (not the minutes an operator may expect).
  • Source · twenty.container

Vaultwarden

  • Upstream · dani-garcia/vaultwarden
  • Image · docker.io/vaultwarden/server:1.32.7
  • Role · Bitwarden-compatible password vault (see The vault)
  • Ports · 127.0.0.1:8222:80
  • Routing · vault.sodimo.eu via cloudflared → Caddy → vaultwarden:80 (+ WebSocket route to vaultwarden:3012)
  • Env · vaultwarden.env — SIGNUPS_ALLOWED=false, SMTP via Sodimo Postfix for password-reset
  • Secrets · ADMIN_TOKEN (64-char random), SMTP password
  • Depends on · sodimo.network; SQLite backup nightly to NAS → weekly R2
  • Source · vaultwarden.container

Postfix

  • Upstream · bokysan/docker-postfix (vendored-upstream Postfix 3.x; Principle 1 counter-example)
  • Image · docker.io/boky/postfix:4.3.0
  • Role · outbound SMTP + inbound MX for sodimo.eu (+ sodimonet.fr, yallafood.eu, cavisteduliban.fr)
  • Ports · 25, 587, 465 (direct; no Caddy, no cloudflared)
  • Routing · direct — Port 25 opened on the ISP side (HB2 ticket)
  • Env · postfix.env (hostname, allowed sender domains, rspamd milter wiring)
  • Config state · postfix-main.cf + postfix-master.cf Volume= lines are temporarily commented out in postfix.container pending config authorship (sodimo/dotfiles#18). Postfix currently boots env-only on POSTFIX_* variables; full config lands with #18.
  • Secrets · DKIM signing key (via rspamd), SMTP relay credentials if ever added
  • Depends on · rspamd (milter), dovecot (LMTP delivery). CF Queue email_outbox → sodimo-email-drain → local sendmail -t (Principle 3).
  • Source · postfix.container

Dovecot

  • Upstream · dovecot/dovecot
  • Image · docker.io/dovecot/dovecot:2.3.21.1
  • Role · IMAP + LMTP for 33 Sodimo mailboxes (migrated from Strato)
  • Ports · 143, 993; LMTP on 24 internal-only
  • Routing · IMAP clients reach mail.sodimo.eu:993. No web UI.
  • Env · dovecot.conf bind-mounted; per-user passwd entries age-encrypted by chezmoi into users.d/
  • Secrets · per-user password hashes in users.d/
  • Depends on · dovecot-mail volume (maildir, nightly rsync’d to NAS)
  • Source · dovecot.container

rspamd

  • Upstream · rspamd/rspamd
  • Image · docker.io/rspamd/rspamd:3.9.1
  • Role · spam filter + DKIM signing for outbound; Postfix connects via milter on rspamd:11332
  • Ports · internal-only; web UI on 11334 reachable via Tailscale for Tom
  • Routing · internal-only
  • Env · none
  • Secrets · DKIM private keys in rspamd-config volume
  • Depends on · sodimo.network
  • Source · rspamd.container

Piler

  • Upstream · jsuto/piler
  • Image · docker.io/sutoj/piler:1.4.5
  • Role · full-text-searchable email archive; BCC of every inbound + outbound from Postfix; indefinite retention (D-106)
  • Ports · 127.0.0.1:8091:80
  • Routing · archive.sodimo.eu via cloudflared → Caddy → piler:80
  • Env · piler.env (MySQL credentials)
  • Secrets · MySQL root + app password
  • Depends on · piler-data, piler-mysql volumes. CNIL employee-transparency notice required before activation.
  • State · known-broken · the Sutoj image expects external MySQL. Missing piler-mysql.container sidecar + MYSQL_HOSTNAME env; currently stalls at boot on MYSQL localhost not ready. Tracked as sodimo/dotfiles#17 (sidecar + companion env still to draft).
  • Source · piler.container

sodiwin-etl

  • Type · systemd user timer + one-shot service (not a quadlet)
  • Image · ghcr.io/sodimo/etl:latest (pulled at dispatch)
  • Role · nightly ETL of Sodiwin SFTP CSV dump → Cloudflare D1. Fires 03:00 Europe/Paris (Florian’s dump lands 02:30).
  • Dispatch · sodiwin-etl.timer → sodiwin-etl.service → %h/.local/bin/sodiwin-etl → podman run
  • Env · %h/.config/sodimo/sodiwin-etl.env — D1 account ID, database ID, API token
  • Secrets · Cloudflare API token (D1 write)
  • Black-box boundary · reads from NAS mount /mnt/sodiwin-dumps/ populated by SFTP from Florian; never touches Sodiwin directly
  • Source · sodimo/etl (unit files land via harness bootc)

Paperclip

  • Upstream · paperclipai/paperclip · docs.paperclip.ing · see Paperclip for the full chapter
  • Image · ghcr.io/sodimo/paperclip:v2026.416.0-pi0.x.y — derived from ghcr.io/paperclipai/paperclip:sha-b8725c5 (v2026.416.0) with @mariozechner/pi layered on top. Upstream tags commit SHAs, not semver; our derived tag pins both pieces. Claude Code and Codex CLIs are pre-installed upstream, pi-mono is added in the derived image (see docker/paperclip/Dockerfile). claude and pi are the two canonical CLI harnesses for Paperclip agents.
  • Ports · 127.0.0.1:3100:3100 (UI + API on a single port, SERVE_UI=true default)
  • Routing · paperclip.sodimo.eu via cloudflared → Caddy → paperclip:3100 (WebSocket-preserving proxy), CF Access gated
  • Env · %h/.config/sodimo/paperclip.env (from paperclip.env.tmpl) — default PAPERCLIP_DEPLOYMENT_MODE=authenticated, PAPERCLIP_PUBLIC_URL=https://paperclip.sodimo.eu, telemetry off. All Anthropic + OpenAI traffic routed through http://litellm:4000 (LiteLLM master key as bearer); no direct cloud creds. Agent-runtime vars (PAPERCLIP_AGENT_ID, PAPERCLIP_API_URL, …) are auto-injected per spawn.
  • Secrets · PAPERCLIP_SECRETS_MASTER_KEY (32-byte hex), BETTER_AUTH_SECRET (32-byte hex, BetterAuth session signing; Paperclip refuses to start without it — added on e400703), Postgres password, LiteLLM master key. First admin bootstrapped out-of-band with paperclipai auth bootstrap-ceo --data-dir /paperclip --base-url https://paperclip.sodimo.eu.
  • Depends on · paperclip-db (postgres:16), litellm. Persistent data at /paperclip (volume paperclip-data); pi model config bind-mounted at /paperclip/.pi/agent/models.json.
  • Rootless caveat · upstream image uses gosu node … as entrypoint; rootless podman 5.8.2 blocks it with operation not permitted. Validated workaround in smoke: --user 1000:1000 --entrypoint '["node",…]'. Tracked as sodimo/dotfiles#16 (quadlet-level override vs image-level entrypoint swap — design call open).
  • Symmetric-application rule · Paperclip agents enqueue to CF Queue email_outbox via the Worker email_send tool, never local sendmail directly (Principle 3, see 15-design-principles.md).
  • Source · paperclip.container

sodimo-email-drain

  • Type · systemd user service (not a quadlet)
  • Role · pull consumer for CF Queue email_outbox; validates against ALLOWED_SENDERS and pipes to local sendmail -t. Principle 3 in concrete form: no inbound port on Fedora.
  • Dispatch · %h/.local/bin/sodimo-email-drain (~130 lines Python, single urllib dep)
  • Env · %h/.config/sodimo/email-drain.env — CF_ACCOUNT_ID, CF_QUEUE_ID, CF_API_TOKEN, ALLOWED_SENDERS
  • Secrets · Cloudflare API token (queue consumer)
  • Failure modes · sendmail failure → message lease → retry → 3 attempts → DLQ to email_dlq; weekly CF Worker cron summarises DLQ to admin@
  • Symmetric-application rule · Paperclip agents on the same Fedora box still enqueue to email_outbox; they do not call local sendmail directly (see 15-design-principles.md)
  • Source · sodimo/dotfiles: bin/sodimo-email-drain

Upgrade discipline

One loop, verified on every bump:

  1. Bump the pinned tag (or digest, per D-167) in home/dot_config/containers/systemd/<service>.container.
  2. chezmoi apply on the harness.
  3. systemctl --user daemon-reload && systemctl --user restart <service>.service.
  4. Verify from Cockpit: unit active (running), Caddy route answers 200, CF Access policy still attached, and for LLM-adjacent services the run_ledger shows fresh rows.

A failed step-4 rolls back via chezmoi forget + the prior tag. Never edit the running container in place.