How AI is used, and fenced in
What AI models do in our build pipeline, what they never decide, and how a model that is wrong, or is fooled, is kept from reaching a published release. Written 2026-11-12. Questions or holes we missed: tell us on GitHub.
What AI does here
AI models do much of the work of packaging a release:
- researching an upstream release and writing its build strategy (what to build from, which version, which settings);
- writing and correcting the admin, database and API handbooks;
- proposing fixes when a check fails, such as a dependency floor for a new vulnerability or a corrected handbook line;
- reviewing a held release independently, before it is published.
Which models. Today this runs on hosted AI model subscriptions. Earlier, a locally run open model did part of the work: about a third of it by late August 2026, until October 2026. Our plan is to move back to locally run models as they improve, so that the pipeline depends less on any outside service.
What a model never decides
A model never decides what is published. Code does, and every one of these checks must pass, each with a recorded result:
- No network during the build. The release is built with
--network=none, so nothing a model wrote can fetch anything while it builds. - Every input pinned. Everything the build uses is fetched beforehand and pinned by digest or commit. The packaging step refuses a recipe with an unpinned fetch, and, since 2026-11-12, one that turns off TLS or signature checks (
curl -k,--no-check-certificate,pip --trusted-host,GIT_SSL_NO_VERIFY, an apt source marked trusted, and similar). - Non-root by default. The few products that need extra privileges are named, each with its reason (the non-root rule).
- Our own vulnerability scan. A release with a fixable critical or high vulnerability is not published as grade A.
- Handbooks checked against the running image. Every claim in a handbook is checked by code against recorded facts and against the image itself. The check errs toward holding a release.
- Independent review, then a clean-room rebuild. A held release is reviewed separately, then rebuilt from its sold archive on a second build host, with the network off, before it is published.
- Code changes to the pipeline go through the full test suite, which must pass on the merged result.
No person reviews each release before it is published: these checks decide. That is a choice, so that a release depends on evidence anyone can check, not on someone's opinion.
Threat model: prompt injection and model mistakes
Prompt injection is text planted where a model will read it, written to make the model do something else. A model working on a release reads upstream material: READMEs, release notes, issues, changelogs, package metadata. Anyone can write those. So we assume some of what a model reads is hostile, and that a model can also simply be wrong.
| What could go wrong | What stands in the way |
|---|---|
| A planted instruction makes a model add a download from a URL it chose. | The build has no network. The fetch must happen beforehand and be pinned by digest or commit, and the packaging step refuses anything unpinned. The recipe shows every fetch, with its pin. |
| A model turns off a certificate or signature check "to get it working". | The packaging step refuses the known ways of doing that, listed above. A justified exception has to say why, in the recipe, where you can read it. |
| A model runs the service as root, or adds privileges. | The non-root rule. An exception is named on the product's page with its reason. |
| A model picks an old, vulnerable version. | The vulnerability scan, and the version policy: a release tracks upstream's latest version of its line. |
| A model writes a false or dangerous instruction in a handbook. | The handbook check runs the commands it can against the image, and compares claims with recorded facts. The independent review reads the rest. |
| A model weakens a check in the pipeline itself. | Pipeline changes go through the full test suite, and the tests pin what each check must refuse. |
Limits. These are the defences as designed. We have not run an adversarial test of them, such as a planted instruction in a test upstream project. The independent reviewer is also an AI model, so a trick that fools one model may fool both. A misleading handbook sentence that none of the checks can test could still be published. That is why every release's recipe, build kit and evidence are public, and why we want to hear about anything you find.
What you can do
- Read the recipe of any release: its public part is open to everyone, and every file is open to any registered account.
- Rebuild the release from its build kit with the network off, and compare the digest (what each piece of evidence proves).
- Scan the image yourself, by digest, with the scanner you trust.
- Report anything that looks wrong on GitHub. See also the FAQ.