Chapter 26 / 40

Paperclip — where background agents live

Paperclip is where Sodimo’s background agents live. When the morning-brief runs, when the delta-alert fires, when the Monday commentary composes itself overnight — Paperclip is what wakes them, tracks what they did, and shows the result to whoever cares.

Status: Planned — activates once the Framework Desktop is on-site and the harness is up. Paperclip is the one service Sodimo ships as a derived image rather than a pinned upstream quadlet, because the agent runtime needs a third CLI harness (pi-mono) layered on top of the two (claude, codex) that ship with upstream. The only Sodimo-owned surface otherwise is the integration glue that attributes run_ledger entries back to Sodimo’s MCP tool calls.

The operational card — image, env, ports, dependencies — lives in 39-quadlet-reference.md#paperclip. This chapter is the story: why the derived image, why Restart=on-failure, why the hostname allowlist, why the env file diverges.

When to look at it

Three scenarios bring a team member to Paperclip:

An agent produced something wrong. The morning brief named the wrong customer; the collection letter cited the wrong balance. Paperclip’s Issue detail shows the exact prompt, the exact output, the tool calls in between, and the model used.
The monthly cost looks high. The Costs page breaks spend down by agent and by model. A runaway scheduled job is visible at a glance.
Is the agent scheduled for X still running? The Routines page lists every scheduled agent — what it does, when it fires, when it last ran, whether that run succeeded.

What you see

Paperclip has five pages that matter day-to-day:

Activity. The live feed — every run as it fires, with status and duration.
Agent detail. One agent, its full run history, its token spend, its routines.
Issue detail. One work unit — the task, the agent that picked it up, the run that executed it, the output, the provenance chain if the task came from another agent.
Routines. The schedule — cron-like rules that wake an agent on a timer. “Every weekday 07:30, morning brief for Rani.” “Every Monday 09:00, weekly commentary for Paul.”
Costs. Monthly spend per agent, separating local runs (no per-request cost) from cloud escalations.

The feed is CLI-agnostic. Some agents run under pi-mono against the local models; others run under Claude Code against the cloud tier. Same dashboard, same schema, same cost view. The choice of CLI is an agent setting.

Why a derived image

Paperclip is unusual in the Sodimo stack: it is the only service that ships as a derived image rather than a pinned upstream quadlet. The derivation is minimal — a five-line Dockerfile at docker/paperclip/Dockerfile that layers @mariozechner/pi@latest onto ghcr.io/paperclipai/paperclip:sha-b8725c5 — but it forces two pieces of policy that are worth knowing.

First use of the docker/ top-level. Until Paperclip, sodimo/dotfiles held only chezmoi-managed config; no Sodimo-built images existed. docker/paperclip/ establishes the pattern for any future derived image the stack needs.

Upstream tags commit SHAs, not semver. The sha-b8725c5 tag resolves to what upstream calls v2026.416.0 — the two are interchangeable but only the SHA is stable. The derived tag scheme is ghcr.io/sodimo/paperclip:v2026.416.0-pi0.x.y — upstream calver plus the resolved @mariozechner/pi version at build time. The 0.x.y placeholder must be replaced with the actual resolved version at publish; until then it signals “built but not published.” Tracked at sodimo/dotfiles#12.

Why pi at all. Paperclip ships with claude and codex pre-installed as the default CLI harnesses for agent spawns. pi (pi-mono) is the third; it is the CLI Sodimo standardizes on for local-model agents, and it has to be added to the image because upstream does not ship it. The layering is deliberately minimal — one npm install -g and a /tmp + npm-cache cleanup — so a future upstream bump is a one-line change to the FROM.

The image was built locally on 2026-04-22 (2.55 GB, ~5 min build) and registered in podman images on the dev box. It is not yet pushed to ghcr.io — blocked on org-level write:packages auth. The quadlet references ghcr.io/sodimo/paperclip:… eagerly; the first production pull will happen once the publish path is unblocked.

Env, secrets, and the `.tmpl` exception

Paperclip’s env file lives at home/dot_config/sodimo/paperclip.env.tmpl — the first chezmoi .tmpl in sodimo/dotfiles. Every other service’s env file is a flat *.env next to its .container. The divergence is handoff-authorized: the Paperclip env carries multiple 32-byte-hex secrets that benefit from chezmoi’s template-driven secret injection, and grouping them under dot_config/sodimo/ makes the “find the secrets file” story uniform across future services with the same shape.

Four secrets are required for Paperclip to boot:

BETTER_AUTH_SECRET — BetterAuth session signing key. Paperclip refuses to start without it; the boot failure is loud and early, validated in the Wednesday 2026-04-22 smoke. The template ships a CHANGEME-BETTER-AUTH-32-BYTE-HEX placeholder; rotate to a real 32-byte hex at commissioning. Rotation invalidates all active sessions.
PAPERCLIP_SECRETS_MASTER_KEY — 32-byte hex. Loss of this key means loss of every encrypted secret stored inside Paperclip’s own DB.
Postgres password — for the paperclip-db sidecar (postgres:16).
LiteLLM master key — used as the bearer when Paperclip calls LiteLLM.

The defaults in the template are deliberately conservative:

PAPERCLIP_DEPLOYMENT_MODE=authenticated — the UI is gated behind Paperclip’s own auth; CF Access gates the hostname at the edge on top of that.
PAPERCLIP_PUBLIC_URL=https://paperclip.sodimo.eu.
Telemetry (PAPERCLIP_TELEMETRY_DISABLED=1, DO_NOT_TRACK=1) off.

All Anthropic and OpenAI traffic from Paperclip agents — whether spawned under claude or pi — is routed through http://litellm:4000 as the ANTHROPIC_BASE_URL and OPENAI_BASE_URL, with the LiteLLM master key as bearer. No direct cloud credentials live in Paperclip. This is Principle 3 in concrete form: every outbound AI call crosses the gateway, and every crossing is captured in run_ledger.

Rootless posture and the `gosu` workaround

Upstream’s ghcr.io/paperclipai/paperclip image uses gosu node … as its entrypoint — an unprivileged-drop pattern that assumes a rootful container runtime. Under rootless podman 5.8.2, that pattern fails at boot with:

error: failed switching to "node": operation not permitted

The Wednesday 2026-04-22 apps-stack smoke validated a workaround at the quadlet level: override the entrypoint to skip gosu and run node directly, with an explicit --user 1000:1000. That workaround is not yet in-tree; two fix paths are documented on sodimo/dotfiles#16:

Quadlet-level override (recommended) — local, reversible, no image rebuild. PodmanArgs=--user 1000:1000 --entrypoint '["node",…]' inside paperclip.container.
Image-level entrypoint swap — flip the entrypoint in docker/paperclip/Dockerfile. Permanent, but diverges harder from upstream and complicates future upstream bumps.

The design call is pending. Until it lands, Paperclip boots cleanly on the test rig with the quadlet override and 200s on /api/health.

The hostname allowlist

Paperclip has a runtime PAPERCLIP_ALLOWED_HOSTNAMES allowlist that is strictly tighter than the HTTP bind. A request with a Host header not on the allowlist gets a self-describing 403 instructing operators to run:

paperclipai allowed-hostname <host>

This is not a bug. Production paperclip.sodimo.eu is already on the allowlist in the template. The behavior surfaces during LAN-exposure smoke tests — when the stack is temporarily bound to 0.0.0.0 and hit via a bare IP — and the 403 message is self-instructing enough that the operator response is one command.

Restart posture

Paperclip is the one service in the stack that uses Restart=on-failure rather than the stack-wide Restart=always. The divergence is handoff-explicit: Paperclip failures are usually config-level (missing secret, bad DB URL, upstream auth drift) and quiet systemd retry-loops hide more than they help. on-failure produces a clean non-zero exit surfaced by systemctl --user status rather than a noisy loop.

This is revisit-able. After the first week of production runtime on the harness, if on-failure turns out to mask transient DB reconnects that always would have recovered from, the policy moves. For now: loud-fail over silent-recover.

Routing

Traffic path at production: cloudflared terminates paperclip.sodimo.eu → Caddy (WebSocket-preserving proxy) → paperclip:3100. CF Access gates the hostname at the edge, on top of Paperclip’s own authenticated deployment mode.

Inside the harness, PublishPort=127.0.0.1:3100:3100 binds to the host loopback only — never 0.0.0.0. No LAN-direct access to Paperclip; every reach goes through the Cloudflare path. The host-loopback bind is what lets Caddy (running on the same host) proxy to Paperclip without the port being exposed to the broader network.

Smoke verdict — 2026-04-22

Wednesday evening’s apps-stack smoke brought up Paperclip end-to-end on the dev box:

Stack composed cleanly after the BETTER_AUTH_SECRET fix landed in e400703 and the rootless --user 1000:1000 workaround was applied at the quadlet level.
/api/health returned HTTP 200.
Postgres migrations against the paperclip-db sidecar completed cleanly on first boot.
Hostname allowlist behaved exactly as documented — 403s on non-allowlisted hostnames, 200s on paperclip.sodimo.eu.

Next session’s target: wire CF Access onto the hostname, validate the cloudflared → Caddy → Paperclip chain end-to-end with a real Google Workspace sign-in.

How Paperclip fits the rest of the stack

Paperclip does not replace OpenWebUI and does not replace the CRM. The split:

Surface	What lives there
OpenWebUI (chapter OpenWebUI)	Interactive chat with the local models. Ad-hoc.
Paperclip	Scheduled or background agent runs. Audited.
CRM (chapter CRM)	Humans + customers. Kanban, records, notes.
Piler (chapter Mail)	The mail archive. Search and retrieval.

Agents running under Paperclip call the same tools any other AI surface calls — see chapter What the AI can access. The tool calls land on the Cloudflare MCP endpoint exactly as they would from a Claude.ai session. This is deliberate: every side-effect an agent causes routes through the same endpoint a human would route through, so the audit trail is unified. Nothing in Paperclip shortcuts to the local box.

Symmetric-application rule

Paperclip agents emit email through the Cloudflare Queue email_outbox via the Worker’s email_send tool — never through local sendmail on the Fedora host, even though sendmail is physically reachable. This is Principle 3 — the AI must use the same pathway a human API caller would. The rule matters because Paperclip is the most tempting place to break it: the agent is on the same box as sendmail, and a one-line bash tool call would “work.” It would also silently cut the run_ledger emission for that send. See 15-design-principles.md#principle-3.

Approvals

Paperclip has an approvals page for agent outputs that need a human to sign off before they send. During this engagement the page stays unrouted — every agent output lands as a draft in the relevant queue (Paul’s collection-letter queue, for example) rather than through Paperclip’s own approval flow. The feature is present in the codebase; turning it on is a post-handoff decision.

When to look at it, restated

Paperclip is the place a team member goes when the question is “what did the AI do last night, and did it cost anything.” The place a team member goes when the question is “what should the AI do next” is the skills library and OpenWebUI. One is the audit surface; the other two are the authoring and chat surfaces. They do not overlap.