ci: rebuild only Docker variants whose bases changed and make images reproducible#2545
Open
xavierleune wants to merge 1 commit into
Open
ci: rebuild only Docker variants whose bases changed and make images reproducible#2545xavierleune wants to merge 1 commit into
xavierleune wants to merge 1 commit into
Conversation
…reproducible Scheduled rebuilds used a single fingerprint covering the 15 base images of all variants: any single base update (e.g. php:8.3-zts-trixie or golang:1.26-trixie) rebuilt and re-pushed all 24 images, moving every generic tag (php8.4, 1-php8.4, ...) roughly every other day. Comparing two such rebuilds of the same release showed byte-identical content (same dpkg versions, same frankenphp binary) but different digests, caused only by build artifacts: timestamp() label, layer mtimes and embedded provenance. - Compute one fingerprint per variant (its php-base + golang-base digests) and only rebuild/push the variants whose fingerprint changed. The label is read from the versioned builder tag; the legacy all-variants fingerprint is still accepted so the transition (old bake file in the latest release tag) keeps the current behavior until the next release. - Store the fingerprint label on builder images only, so runner digests depend solely on their content. - Make images reproducible: SOURCE_DATE_EPOCH and created/revision labels pinned to the built commit, rewrite-timestamp=true on push, BuildKit provenance disabled (its timestamps made every digest unique; the signed GitHub attestations pushed by actions/attest are kept), and removal of files with non-deterministic content (apt logs, ldconfig aux-cache) from Debian runner layers. Two independent no-cache builds of the 8.4-trixie runner now produce the same digest. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
xavierleune
marked this pull request as ready for review
July 21, 2026 08:08
Member
|
I like the idea, but we need to double check that this won't break anything. It's a tricky part of the CI! |
Contributor
Author
|
yep that's super tricky. LMK if I can help |
Contributor
|
That's why I use my packages in ubi 10 docker containers 🤣. Rob referred to that time as Deskation, but I'm far too impatient. I'll review later, but it's a good idea. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The nightly scheduled rebuild uses a single fingerprint covering the base images of all variants (
php:{8.2,8.3,8.4,8.5}-zts-{trixie,bookworm,alpine}+golang:1.26-*). When any single one of them moves, all 24 images are rebuilt and re-pushed, so every generic tag (php8.4,1-php8.4,latest, …) gets a new digest roughly every other day (13 full rebuilds in the last 30 scheduled runs).Comparing two such rebuilds of the same release (July 16 vs July 18,
1-php8.4, both triggered by base images unrelated to the 8.4 variant) shows byte-identical content — same dpkg package versions, samefrankenphpbinary sha256, sameinstall-php-extensions, same watcher libs — yet different digests. The churn came only from build artifacts:org.opencontainers.image.created = timestamp()(guarantees a new config digest on every build),WORKDIR /applayer differed),org.opencontainers.image.revisionpointing atmain's HEAD instead of the built release tag.This causes spurious pulls/deploys for users who track digests (watchtower, renovate, digest pinning, CI caches).
Changes
Per-variant fingerprint
docker-compute-fingerprints.shnow computes one fingerprint per variant (the digests of itsphp-base+golang-basecontexts) and outputs arebuild_variantslist; the build/push matrix only includes variants whose bases actually changed.docker-bake.hclis the pre-change one, so behavior stays exactly as today until the next release ships the new bake file, then per-variant skipping kicks in automatically.main()function: it replaces itself on disk viagit checkoutwhile running, and bash reads scripts lazily.preparejob (the per-variant check reads 12 labels via skopeo).Reproducible images
SOURCE_DATE_EPOCHand thecreatedlabel are pinned to the built commit's date;revisionnow points at the release tag's commit on scheduled runs.rewrite-timestamp=trueon the push output clamps layer file mtimes.*.attest=type=provenance,disabled=true): its embedded timestamps made every digest unique. The signed GitHub attestations pushed byactions/attest(push-to-registry: true) are unchanged./var/log/dpkg.log,/var/log/apt,/var/log/alternatives.log, ldconfig'saux-cachewhich embeds mtimes).With this, a rebuild of unchanged inputs converges to the same digest, so tags only move when content actually changes.
Verification
--no-cachebuilds ofrunner-php-8-4-23-trixie(linux/arm64, BuildKit v0.31.2) produce the same manifest digest (sha256:516a940f…) — the Go build itself is already deterministic.ff8d5d61…), so the transition is seamless.skip=true; simulating aphp:8.3-zts-trixiedigest change post-transition rebuilds onlyphp-8-3-32-trixie.shellcheck,actionlintanddocker buildx bake --print(default variables, local usage) all pass.Behavioral changes worth maintainers' attention
org.opencontainers.image.created(and the OCI configcreatedfield) now reflect the built commit's date instead of the build date.dev.frankenphp.base.fingerprintlabel (builder images keep it).Out of scope (accepted): builder images (Go build cache) and Alpine (apk db not double-build-tested) are not guaranteed reproducible; with per-variant skipping they are only rebuilt when their own bases change anyway.
🤖 Generated with Claude Code