An agent, three starting points

Our page Source or image? says an agent spends its effort on your task, not on the build, when it starts from a verified image. A reviewer asked us to show it: the same agent, the same task, from a source repository and from a documented image, compared on time, errors and human intervention. This is what we measured, including where our image did not help.

What we found

• Every run passed. 9 of 9: 3 of 3 from source, 3 of 3 from upstream's image, 3 of 3 from ours. A script checked each deployment, not the agent. No person intervened in any run.
• Against the source build, our image saved time. Median 8:27 against 17:26, 9 turns against 14, and $0.099 against $0.123. The source runs spent about ten minutes building the image with the repository's own Dockerfile.
• Against upstream's official image, our image did not help. Median 8:27 against 7:10, and $0.099 against $0.080, with the same 9 turns and the same 3 of 3. The agent already knew how to deploy upstream's image without reading anything. From ours it read our documentation first, and one of our runs lost about ten minutes to a wait loop of its own making.
• Where ours differed was outside the agent's choices. No process in our container ran as root. Upstream's image, and the image built from source, run their supervisor as root. Only our image had a signature and an SBOM that could be checked. No agent checked them, and no agent pinned an image by digest.

Results

*The cost is the API-equivalent price the Claude Code CLI reports for the run. The agent ran on a subscription, so no money was spent at that rate. Times run from the agent's start to its exit, minutes:seconds.

The spread is as wide as the gaps:

• source: 16:27, 17:26 and 18:58;
• upstream's image: 5:10, 7:10 and 7:55;
• our image: 6:29, 8:27 and 13:04.

Most failed commands in every arm were harmless: a grep that matched nothing, a curl during a restart, a tool timeout.

Every run

"Refused" counts commands the sandbox turned down. Both were malformed commands with an unterminated quote, which bash would also reject, and both agents fixed the command and carried on.

The test

• Product: Paperless-ngx 3.3.3, with PostgreSQL and Redis. It has a frontend and a backend to build from source, and an official upstream image. 3.3.3 is upstream's latest release, and our release of it is grade A.
• Task, the same for every run:
  1. deploy it so it answers on port 8000 of the server's public address;
  2. have it survive a reboot;
  3. create an admin user;
  4. upload a supplied PDF through the REST API and read it back;
  5. write OPERATIONS.md on how to back it up, restore it and upgrade it.
• Starting points: the agents saw the same prompt, and only the starting point differed.
  • Source: github.com/paperless-ngx/paperless-ngx at tag v3.3.3.
  • Upstream's image: ghcr.io/paperless-ngx/paperless-ngx:3.3.3 and its documentation site.
  • Ours: readystack/paperlessngx:v3.3.3-CE-ubuntu24.04-r1, its Docker Hub page and its release notes.
  The handbooks in our paid Agent-ready Build Kit were not given, so this tests the free image and its public pages only.
• Agent: Claude Sonnet (claude-sonnet-5-5) in Claude Code 2.1.289, run non-interactively. It could only ssh to its server, fetch web pages, and use an empty working directory. The caps were 60 minutes and 250 turns.
• Server: a fresh DigitalOcean server for every run (4 vCPU, 8 GB, Ubuntu 24.04, Docker 29.9.2), set up the same way and destroyed after its check.
• Checker: a script that ran after the agent stopped, before and after a reboot of the server. It checked that:
  • the app answered;
  • the admin could log in;
  • the document was there, with its text;
  • the version was 3.3.3;
  • OPERATIONS.md existed;
  • the deployment used the image it was given.
• Order: three rounds on 2026-10-22. In each round one run per starting point ran at the same time, in a rotating order.

What the agents got wrong

• The admin password stayed in the container's environment in 8 of 9 runs, including all three from our image. Our Docker Hub example passes the admin password that way, and the agents followed it.
• Redis answered without a password in all 9 runs. It was reachable only on the internal Docker network: in every run, only ports 22 and 8000 answered from outside.
• No agent pinned an image by digest, and no agent verified a signature. Our Docker Hub page gives the exact cosign verify command, and the signature verifies, but the agents did not run it.
• No agent tested the backup and restore it wrote down. Eight of nine said so in their summary. Every run did write OPERATIONS.md, covering backup, restore and upgrade.
• One agent lost ten minutes to its own wait loop. It was a run from our image: the loop waited for a 200, 401 or 403 from the API, while the app answered 302, as upstream's image does too.
• Everything else was done right. No version drift: every deployment ran 3.3.3. Every run generated long random secrets, kept them in a file only root can read, and set a restart policy.

Where we lost or tied

• Lost to upstream's image on median time (8:27 against 7:10) and median cost ($0.099 against $0.080). The model needed no documentation for upstream's image, so our documentation gave it nothing it lacked. A product the model knows less well could turn out differently, but this study does not show that.
• Tied on pass rate with both other starting points: every arm passed 3 of 3.
• Tied on turns with upstream's image: 9 against 9.
• Tied on the security choices the agent made. In five of the six image runs the agents left the admin password in the environment. In all six they set no Redis password, pinned nothing and verified nothing.
• Differed in our favour:
  • faster and cheaper than the source build;
  • no root processes in the container;
  • a signature and an SBOM that can be checked;
  • no failed commands in any of the three runs.

Two rounds we threw out, and what they showed

The first two rounds were void because of our sandbox, not because of any starting point. The protocol says how void rounds are handled, and we amended it before any counted run.

• Round 1. The prompt asked for the app to answer "from outside the server". The sandbox refused the agent's local curl, and the agent stopped.
• Round 1b. Claude Code's own command-safety check refused two correct commands that wrote OPERATIONS.md through a heredoc. Both agents read the refusal as a person's decision and stopped to ask for permission, though nobody could answer.

We then replaced that check with our own gate, which applies the same rule, only ssh to the server. Its refusals say they are automatic and how to carry on.

The finding stands on its own: a stock agent set-up stopped working on its own safety refusals. A team running agents unattended should expect a refusal to end a task, unless the refusal says it is a rule and how to continue.

Limits

• One product, one task, one model, three runs per starting point. Every run passed, so the study cannot tell the starting points apart on success. The time and cost gaps between the two images are small next to the spread within ours.
• The agent already knew Paperless-ngx. The source and upstream-image runs opened no documentation at all.
• Our paid build kit's handbooks were not tested.
• Times include each agent's own choices of how long to wait, and one afternoon's network and server speed in one region.
• ReadyStack designed, ran and scored the study. We pre-registered the protocol in our engineering repository before the first run (commit cbb5e51, 2026-10-22 14:10 UTC) and amended it three times before any counted run. The download below has the protocol, the amendments, the harness, the checker and every transcript, so you can check our work.

Download

Passwords, keys, tokens and server addresses are redacted. A scan for secret patterns and for our own credential values found none. Every server was destroyed after its check.