OpenSearch 3.9.0-CE-ubuntu24.04-r3 build trail
How ReadyStack built readystack/opensearch:3.9.0-CE-ubuntu24.04-r3 and its build kit, copied record by record: why the release exists, every chain, drive and stage it went through, the model work, the review rounds, the defects, the checks that verified it and when it was published. Each line names the record it comes from in brackets; a record that does not exist for this release says so.
This trail
As JSON- Image
readystack/opensearch:3.9.0-CE-ubuntu24.04-r3 - Index digestsha256:0c4efd41417222379711827de5334e992c8686eba61872bf3d77a9ba81dc9098
- linux/amd64 digestsha256:cfeedc2623aba884b8fe03dee8146d237c801f68f985935b344eeaace425136b
- Archive
ReadyStack-opensearch-3.9.0-CE-ubuntu24.04-r3.tar.gz - Archive SHA-256 nowef57d427d80c803ad80e9c1fb1d71d34d92cb2c0752380d6b6964ac312614ce0
- Generated2026-11-02T23:33:35Z by KitForge release/build_trail.py
- Sections9 sections, 4 fully recorded; 10 items not recorded
Signed with the release key The trail's signature is a Sigstore bundle beside it. Check it is ours before you rely on it:
cosign verify-blob --key https://readystack.dev/keys/cosign.pub \ --bundle ReadyStack-opensearch-3.9.0-CE-ubuntu24.04-r3.trail.json.sigstore ReadyStack-opensearch-3.9.0-CE-ubuntu24.04-r3.trail.json
IdentityWhy this release existsThe path through the harnessModel work per stageReviewDefectsVerificationPublicationPackages and CPU level
Identity
1 item not recorded- ProductOpenSearch (opensearch) [MANIFEST.json]
- Release tag3.9.0-CE-ubuntu24.04-r3 [MANIFEST.json]
- Imagereadystack/opensearch:3.9.0-CE-ubuntu24.04-r3 [MANIFEST.json]
- Image index digestsha256:0c4efd41417222379711827de5334e992c8686eba61872bf3d77a9ba81dc9098 [docs/provenance/LOCKED_DIGEST.txt]
- linux/amd64 image digestsha256:cfeedc2623aba884b8fe03dee8146d237c801f68f985935b344eeaace425136b [ReadyStack-opensearch-3.9.0-CE-ubuntu24.04-r3.rebuild.json, subject.image_manifest_digest]
- ArchiveReadyStack-opensearch-3.9.0-CE-ubuntu24.04-r3.tar.gz (bytes 1150500355) [MANIFEST.json]
- Archive sha256 at publish54c3c08eabb7e29b46567b59a74e69533b057a24beb5a02142e7d3196582b421 [MANIFEST.json, repack_history[0].previous_tarball_sha256]
- Archive sha256 nowef57d427d80c803ad80e9c1fb1d71d34d92cb2c0752380d6b6964ac312614ce0 [MANIFEST.json, tarball_sha256]
- Archive records agreeyes [MANIFEST.json, PUBLISH_MANIFEST.json and .cutover.json name the same sha256]
- Repack history
- at 2026-11-01 · label kitfix · reason customer kit files brought up to the current engine: BUILD_OFFLINE.sh (kit root, docs/offline, docs/docker); the header of BUILD_OFFLINE.sh no longer says a --provenance rebuild meets a SLSA build level, comment lines only; not image inputs; image and every build input unchanged · previous sha256 54c3c08eabb7e29b46567b59a74e69533b057a24beb5a02142e7d3196582b421 · sha256 ef57d427d80c803ad80e9c1fb1d71d34d92cb2c0752380d6b6964ac312614ce0 · replaced BUILD_OFFLINE.sh, docs/docker/BUILD_OFFLINE.sh, docs/offline/BUILD_OFFLINE.sh, rebuild.json
- Upstream repositoryhttps://github.com/opensearch-project/OpenSearch [docs/provenance/SOURCE_TREE_HASH.json]
- Upstream ref3.9.0 [docs/provenance/SOURCE_TREE_HASH.json]
- Upstream commit4ee42a94e87f66fbf1e62a9871b1b87f91e02472 [docs/provenance/SOURCE_TREE_HASH.json]
- Source tree hash2ae217a7cfe15e67871dcb85116f6032694851802f619e60e9921326333b0902 (files 19243) [docs/provenance/SOURCE_TREE_HASH.json, tree_listing_sha256 (sha256 of the sorted file listing)]
- Kit tree hashsha256:abddd68d64866d1fed86db40af58405ee24398e25c96c06779284168a2537bf8 [ReadyStack-opensearch-3.9.0-CE-ubuntu24.04-r3.rebuild.json, inputs.tree_hash]
- SOURCE_DATE_EPOCH1790105837 (utc 2026-09-22T19:37:17Z) [ReadyStack-opensearch-3.9.0-CE-ubuntu24.04-r3.rebuild.json, inputs.source_date_epoch]
- Base images
- ref ubuntu:24.04@sha256:24737cef17f4e22b1977df7d9a05e77c4e28b7da5b8911f8d8ccda136bf3573c · platform digest sha256:78d4744ead1f5b19b04d0df558ab182a5300903900755e515b2ca4d10eb2044d
- BuildKitmoby/buildkit:v0.31.2 (image digest sha256:e0e67ea9c4bb07bdef683e29f70f3b623d1eb9c82274a9ec04e6bb0511449fb0) [ReadyStack-opensearch-3.9.0-CE-ubuntu24.04-r3.rebuild.json, engine]
- Not recorded: the engine commit(s) that built it (drive_runs records them for drives from 2026-11-02; this release's drives were rebuilt from line logs, which do not name it)
Why this release exists
Fully recorded- Chain714 (created 2026-10-27T18:12:38Z · trigger customer_request · upstream version 3.9.0) [chains, chains #714]
- Requestbuild request #400 (rebuild) (requested at 2026-10-27T17:58:28Z · requested by kitforge-agent · version 3.9.0) [build_requests, build_requests #400]
- Reasonrevision 3.9.0-CE-ubuntu24.04-r3: same upstream version rebuilt (retarget.py --rev-bump) [build_requests, build_requests #400 request_reason]
- Supersedesreadystack/opensearch:3.9.0-CE-ubuntu24.04-r2 (chain 673) [chains, chains #673]
The path through the harness
2 items not recorded1 chain · 3 drives · 18 stages · 1 rewind
Chain 714 release · shipped
Created 2026-10-27T18:12:38Z, upstream 3.9.0, build request 400 [chains, chains #714]
| Drive | Started | Minutes | Build host | How it stopped |
|---|---|---|---|---|
| 447 | 2026-10-27T18:13:12Z | 26.8 | build host 2 withhold-publish | stage_failed at package_archive [drive_runs, drive_runs #447] rebuilt from a build line's log |
| 461 | 2026-10-28T00:56:59Z | 17.4 | build host 2 withhold-publish | ready_to_ship [drive_runs, drive_runs #461] rebuilt from a build line's log |
| 468 | 2026-10-28T02:48:35Z | 15.8 | build host 2 publish-held | shipped [drive_runs, drive_runs #468] rebuilt from a build line's log |
Stages, in order
| Stage | First start | Passed | Attempts | Gate checks passed | Artifacts (SHA-256) |
|---|---|---|---|---|---|
intake |
2026-10-27T18:13:18Z | 2026-10-27T18:13:18Z | 1 | 4
|
— |
acquire |
2026-10-27T18:13:32Z | 2026-10-27T18:13:33Z | 1 | 12
|
— |
plan |
2026-10-27T18:13:52Z | 2026-10-27T18:13:53Z | 1 | 10
|
— |
assemble_kit |
2026-10-27T18:20:40Z | 2026-10-27T18:20:41Z | 1 | 27
|
— |
remediate |
2026-10-27T18:21:08Z | 2026-10-27T18:21:09Z | 1 | 8
|
— |
build_kit |
2026-10-27T18:30:05Z | 2026-10-27T18:30:06Z | 1 | 9
|
— |
verdict |
2026-10-27T18:34:15Z | 2026-10-27T18:34:15Z | 1 | 8
|
— |
publish_image |
2026-10-27T18:36:00Z | 2026-10-27T18:36:01Z | 1 | 7
|
— |
boot_instance |
2026-10-27T18:36:36Z | 2026-10-27T18:36:38Z | 2 | 3
|
— |
sbom |
2026-10-27T18:37:40Z | 2026-10-27T18:37:40Z | 1 | 5
|
— |
schema_discovery |
2026-10-27T18:37:42Z | 2026-10-27T18:37:43Z | 1 | 7
|
— |
admin_handbook |
2026-10-27T18:37:51Z | 2026-10-27T18:37:52Z | 1 | 7
|
— |
db_handbook |
2026-10-27T18:38:02Z | 2026-10-27T18:38:03Z | 1 | 8
|
— |
seed_data |
2026-10-27T18:38:09Z | 2026-10-27T18:38:09Z | 1 | 6
|
— |
seed_validate |
2026-10-27T18:38:11Z | 2026-10-27T18:38:12Z | 1 | 3
|
— |
usecase_validate |
2026-10-27T18:38:41Z | 2026-10-27T18:38:42Z | 1 | 10
|
— |
finalize_docs |
2026-10-27T18:38:47Z | 2026-10-27T18:38:47Z | 1 | 9
|
— |
package_archive |
2026-10-27T18:39:59Z | 2026-10-28T03:04:17Z | 3 | 33 (final gate PASS)
|
— |
[stage_runs]
Rewinds
- 2026-10-28T02:47:53Z:
package_archivetopackage_archiveon docs_redo (scripts/redo-docs.py) [stage_events, stage_events #547]
- Not recorded: each drive's ticks and first failed check (drive_runs records them for drives from 2026-11-02; these drives were rebuilt from line logs)
- Not recorded: the sha256 of each stage's artifacts (stage_events records them for stages passed from 2026-11-02; this chain's events were rebuilt from stage_runs and logs) — the archive's CHECKSUMS.sha256 lists the sha256 of every file it ships
Model work per stage
3 items not recordedRole: writer model. Calls up to 2026-10-28T03:04:17Z.
15 calls over 4 stages
| Stage | Calls | Refused | Errors | Max attempt | Forced commits | Real tokens | Lesson rules shown |
|---|---|---|---|---|---|---|---|
plan | 2 | 0 | 0 | 1 | not recorded | not recorded | — |
remediate | 2 | 0 | 0 | 1 | not recorded | not recorded | — |
usecase_validate | 1 | 0 | 0 | 1 | not recorded | not recorded | — |
finalize_docs | 10 | 0 | 0 | 1 | not recorded | not recorded | — |
[micro_step_runs]
- Not recorded: forced commits (micro_step_runs records them for calls from 2026-11-02)
- Not recorded: real token counts (recorded per call from 2026-11-02; earlier rows hold only an estimate, which is not shown)
- Not recorded: the lesson rules shown to the writer model (lesson_recalls holds no rows for this chain: its recorder was silent from 2026-06-06 to 2026-11-02)
Review
2 items not recordedRole: review model.
1 round, 1 counted, $0.52 recorded cost
| Round | Started | Critical | Major | Minor | Verdict | Cost | Cap in force | Archive reviewed |
|---|---|---|---|---|---|---|---|---|
| 1 | 2026-10-28T01:16:17Z | 0 | 0 | 2 | clean to publish [held_rounds, held_rounds #138] rebuilt from a held-review state file | $0.52 | not recorded | not recorded |
- Not recorded: the sha256 of the archive each round reviewed (held_rounds records it for rounds from 2026-11-02; these rounds were rebuilt from state files)
- Not recorded: the round cap in force at each round (the state files did not keep it)
Defects
Fully recorded- Ledger entries4 [defects, defects where product_slug=opensearch and this tag or chain]
- Disclosed in the archive2 [docs/defects/defects.json, entries]
4 ledger entries: by severity low 3, high 1; by status fixed-in-release 1, fixed-before-release 2, open 1.
| Defect | Severity | Status | Found by | Fixed in |
|---|---|---|---|---|
RSD-opensearch-4e2750036cRelease notes: The Known issues entry reports an invalid test request as a problem, so a customer may think the health endpoint is faulty. [defects, defects #124537] | low | fixed-in-release | held review (review model) | 3.9.0-CE-ubuntu24.04-r3 |
RSD-opensearch-3ab357ef2cAPI guide: API_INTEGRATION_GUIDE.md:1033-1047 print a literal backslash-u2014 in all 12 endpoint-list lines (grep 'u2014'); the shipped text used an em dash; every other list in the guide uses '—'. [defects, defects #125919] | low | fixed-before-release | held review (review model) | — |
RSD-opensearch-80021ae0f2API guide: A GET only reads settings; changing them takes PUT /_cluster/settings, which the guide does not document. [defects, defects #125920] | low | fixed-before-release | held review (review model) | — |
RSD-opensearch-3f07421e9eThe k-NN plugin's nmslib library needs AVX-512, and neither the image nor its handbook says so [defects, defects #152629] | high | open | image scan | — |
Verification
2 items not recorded- Reproduced before publishingyes (rebuilt digest sha256:cfeedc2623aba884b8fe03dee8146d237c801f68f985935b344eeaace425136b · published digest sha256:cfeedc2623aba884b8fe03dee8146d237c801f68f985935b344eeaace425136b · digest match yes · builder moby/buildkit:v0.31.2 · build seconds 228) [docs/provenance/REPRO_VERIFICATION.json, verified]
- Final contract gate (G8)PASS (checks cosign, scan_grade, tag_base_os, vex_shipped, scout_grade_a, script_copies, three_way_sha, offline_rebuild, rebuild_manifest, artifact_coverage, assetchain_coverage) [stage_runs, stage_runs chain 714 package_archive gate_result.g8.result]
- Grade at releaseA (percent 94 · tool docker scout policy 1.26.0 · config docker-default · org vex no · at 2026-10-28T02:49:25Z · failing Docker Scout counts 104 packages with copyleft licences (GPL, LGPL, MPL), all of them operating-system packages (it counts only packages whose licence it can read; the SBOM lists every component).) [SCOUT_EVIDENCE.json, docker_scout_policy]
- First own scan of the releaseA (at 2026-10-28T03:04:37Z · score 100 · rubric cerodeo-v1 · fixable critical 0 · fixable high 0 · syft 1.54.1 · grype 0.120.1 · grype db v6.1.10@2026-10-27T06:30:41Z) [image_scans, image_scans #1363]
- Image signatureverified on the index and the linux/amd64 manifest (key fingerprint 04daa92344a52dbc · at 2026-10-28T02:48:45Z) [docs/provenance/IMAGE_EVIDENCE.json, signed]
- Image signature transparency-log entries
- log index 3522886071 · transparency-log entry
- log index 3522887818 · transparency-log entry
- rebuild.json signatureSigstore bundle, release key (key fingerprint 04daa92344a52dbc · log index 3627434213 · transparency-log entry · integrated 2026-11-01T10:53:00Z) [ReadyStack-opensearch-3.9.0-CE-ubuntu24.04-r3.rebuild.json.sigstore, verificationMaterial.tlogEntries[0]]
- Signed attestations
- kind provenance · digest sha256:0c4efd41417222379711827de5334e992c8686eba61872bf3d77a9ba81dc9098 · state attached · log index 3532660924 · transparency-log entry
- kind sbom · digest sha256:0c4efd41417222379711827de5334e992c8686eba61872bf3d77a9ba81dc9098 · state attached · log index 3532666107 · transparency-log entry
- kind vsa · digest sha256:0c4efd41417222379711827de5334e992c8686eba61872bf3d77a9ba81dc9098 · state attached · log index 3532668749 · transparency-log entry
- kind provenance · digest sha256:cfeedc2623aba884b8fe03dee8146d237c801f68f985935b344eeaace425136b · state attached · log index 3532671245 · transparency-log entry
- kind sbom · digest sha256:cfeedc2623aba884b8fe03dee8146d237c801f68f985935b344eeaace425136b · state attached · log index 3532673673 · transparency-log entry
- kind vsa · digest sha256:cfeedc2623aba884b8fe03dee8146d237c801f68f985935b344eeaace425136b · state attached · log index 3532676422 · transparency-log entry
Offline clean-room rebuilds
- 2026-10-28T01:16:21Z on build host 2 (Docker-in-Docker daemon with --network none): bit for bit yes, digest match yes, customer verify yes
rebuilt sha256:cfeedc2623aba884b8fe03dee8146d237c801f68f985935b344eeaace425136b [clean-room rebuild ledger, runs[503]]
- Not recorded: the policy tool's --result-file itself (its per-policy results are copied into SCOUT_EVIDENCE.json and the archive's IMAGE_EVIDENCE.json; the file is not kept)
- Not recorded: the pins record (docs/PINS.md is not in this archive as generated; a repack that adds it is appended to this trail's history)
Publication
Fully recorded- Image pushed (internal tag), publish_image passed2026-10-27T18:36:01Z [stage_runs, stage_runs chain 714 stage publish_image]
- Release tag promoted2026-10-28T03:02:26Z (method imagetools_create_retag_by_digest_no_rebuild · digest sha256:0c4efd41417222379711827de5334e992c8686eba61872bf3d77a9ba81dc9098 · tag readystack/opensearch:3.9.0-CE-ubuntu24.04-r3) [deliverables/provenance/PROMOTION_RECORD.json, promoted_at]
- Archive manifests written2026-10-28T02:53:37Z [PUBLISH_MANIFEST.json, published_at]
- Archive cut over on prodasset, package_archive passed2026-10-28T03:04:17Z [stage_runs, stage_runs chain 714 stage package_archive]
- Cut-over markerpresent (chain 714) [.cutover.json]
- Release checklist and site event2026-10-28T03:04:15Z (status complete · site event 240 · steps grade a verified done · release tag promoted done · cosign signed done · release evidence recorded done · contract gate passed done · archive published done · hub readme live done · reproducibility recorded done · defects ledger published done · catalog available done · build request fulfilled done · seo regenerated skipped · site catalogue refreshed done · site grade recorded done · site queue synced done · customer requests charged done · owners notified done) [deliverables/provenance/RELEASE_CHECKLIST.json, run_at]
Packages and CPU level
Fully recorded- Packages in the image1003 (scanned 2026-11-02T06:30:04Z · source backfill:daily-scan) [release_images, package_count]
- x86-64 level the image needsv4 (used v4 · baseline v2 · elf files 816 · probed 2026-11-02T17:04:09Z · source backfill) [release_images, isa_level_hard]
History
1 event- 2026-11-02T23:33:35Z generated: by KitForge release/build_trail.py · generator commit 916780cc218cf013bd623a8c085aaf73d3b053d0 · note backfill: built after the release from its records
Records
25 records| Record | Where it is | Rows |
|---|---|---|
docs/defects/defects.json | inside the archive (ReadyStack-opensearch-3.9.0-CE-ubuntu24.04-r3.tar.gz) | — |
docs/provenance/IMAGE_EVIDENCE.json | inside the archive (ReadyStack-opensearch-3.9.0-CE-ubuntu24.04-r3.tar.gz) | — |
docs/provenance/LOCKED_DIGEST.txt | inside the archive (ReadyStack-opensearch-3.9.0-CE-ubuntu24.04-r3.tar.gz) | — |
docs/provenance/REPRO_VERIFICATION.json | inside the archive (ReadyStack-opensearch-3.9.0-CE-ubuntu24.04-r3.tar.gz) | — |
docs/provenance/SOURCE_TREE_HASH.json | inside the archive (ReadyStack-opensearch-3.9.0-CE-ubuntu24.04-r3.tar.gz) | — |
.cutover.json | beside the archive (prodasset /data/production/opensearch/3.9.0-CE-ubuntu24.04-r3/) | — |
MANIFEST.json | beside the archive (prodasset /data/production/opensearch/3.9.0-CE-ubuntu24.04-r3/) | — |
PUBLISH_MANIFEST.json | beside the archive (prodasset /data/production/opensearch/3.9.0-CE-ubuntu24.04-r3/) | — |
ReadyStack-opensearch-3.9.0-CE-ubuntu24.04-r3.rebuild.json | beside the archive (prodasset /data/production/opensearch/3.9.0-CE-ubuntu24.04-r3/) | — |
ReadyStack-opensearch-3.9.0-CE-ubuntu24.04-r3.rebuild.json.sigstore | beside the archive (prodasset /data/production/opensearch/3.9.0-CE-ubuntu24.04-r3/) | — |
SCOUT_EVIDENCE.json | beside the archive (prodasset /data/production/opensearch/3.9.0-CE-ubuntu24.04-r3/) | — |
clean-room rebuild ledger | the clean-room rebuild ledger (runstate/audit-2026-09/bit4bit.json, not published) | — |
build_requests | sadb, ReadyStack's engine database (not published) | 400 |
chains | sadb, ReadyStack's engine database (not published) | 714 |
defects | sadb, ReadyStack's engine database (not published) | opensearch 3.9.0-CE-ubuntu24.04-r3 |
drive_runs | sadb, ReadyStack's engine database (not published) | chain 714 |
held_rounds | sadb, ReadyStack's engine database (not published) | chain 714 |
image_scans | sadb, ReadyStack's engine database (not published) | 1363 |
micro_step_runs | sadb, ReadyStack's engine database (not published) | chain 714 |
release_images | sadb, ReadyStack's engine database (not published) | opensearch 3.9.0-CE-ubuntu24.04-r3 |
stage_events | sadb, ReadyStack's engine database (not published) | chain 714 |
stage_runs | sadb, ReadyStack's engine database (not published) | chain 714 |
attestations.json | the chain's workspace on the controller (not published) | — |
deliverables/provenance/PROMOTION_RECORD.json | the chain's workspace on the controller (not published) | — |
deliverables/provenance/RELEASE_CHECKLIST.json | the chain's workspace on the controller (not published) | — |