Skip to content

ci: rebuild only Docker variants whose bases changed and make images reproducible#2545

Open
xavierleune wants to merge 1 commit into
php:mainfrom
alketech:fix/docker-per-variant-fingerprint-reproducible
Open

ci: rebuild only Docker variants whose bases changed and make images reproducible#2545
xavierleune wants to merge 1 commit into
php:mainfrom
alketech:fix/docker-per-variant-fingerprint-reproducible

Conversation

@xavierleune

Copy link
Copy Markdown
Contributor

Problem

The nightly scheduled rebuild uses a single fingerprint covering the base images of all variants (php:{8.2,8.3,8.4,8.5}-zts-{trixie,bookworm,alpine} + golang:1.26-*). When any single one of them moves, all 24 images are rebuilt and re-pushed, so every generic tag (php8.4, 1-php8.4, latest, …) gets a new digest roughly every other day (13 full rebuilds in the last 30 scheduled runs).

Comparing two such rebuilds of the same release (July 16 vs July 18, 1-php8.4, both triggered by base images unrelated to the 8.4 variant) shows byte-identical content — same dpkg package versions, same frankenphp binary sha256, same install-php-extensions, same watcher libs — yet different digests. The churn came only from build artifacts:

  • org.opencontainers.image.created = timestamp() (guarantees a new config digest on every build),
  • fresh layer mtimes (push builds run without cache; even the WORKDIR /app layer differed),
  • embedded BuildKit provenance (contains build timestamps),
  • org.opencontainers.image.revision pointing at main's HEAD instead of the built release tag.

This causes spurious pulls/deploys for users who track digests (watchtower, renovate, digest pinning, CI caches).

Changes

Per-variant fingerprint

  • docker-compute-fingerprints.sh now computes one fingerprint per variant (the digests of its php-base + golang-base contexts) and outputs a rebuild_variants list; the build/push matrix only includes variants whose bases actually changed.
  • The fingerprint label is stored on builder images only (both targets are always built and pushed together), so runner digests depend solely on their content.
  • The legacy all-variants fingerprint is still computed, passed and accepted when comparing: scheduled runs build the latest release tag, whose docker-bake.hcl is the pre-change one, so behavior stays exactly as today until the next release ships the new bake file, then per-variant skipping kicks in automatically.
  • The script body moved into a main() function: it replaces itself on disk via git checkout while running, and bash reads scripts lazily.
  • DockerHub login added to the prepare job (the per-variant check reads 12 labels via skopeo).

Reproducible images

  • SOURCE_DATE_EPOCH and the created label are pinned to the built commit's date; revision now points at the release tag's commit on scheduled runs.
  • rewrite-timestamp=true on the push output clamps layer file mtimes.
  • BuildKit provenance is disabled (*.attest=type=provenance,disabled=true): its embedded timestamps made every digest unique. The signed GitHub attestations pushed by actions/attest (push-to-registry: true) are unchanged.
  • The Debian runner layers no longer keep files with non-deterministic content (/var/log/dpkg.log, /var/log/apt, /var/log/alternatives.log, ldconfig's aux-cache which embeds mtimes).

With this, a rebuild of unchanged inputs converges to the same digest, so tags only move when content actually changes.

Verification

  • Two independent --no-cache builds of runner-php-8-4-23-trixie (linux/arm64, BuildKit v0.31.2) produce the same manifest digest (sha256:516a940f…) — the Go build itself is already deterministic.
  • The legacy fingerprint computed by the new script is bit-identical to the label on the images currently on Docker Hub (ff8d5d61…), so the transition is seamless.
  • Dry-run of the schedule path against the real Docker Hub state (with v1.12.5's old bake file, i.e. the exact transition scenario) correctly yields skip=true; simulating a php:8.3-zts-trixie digest change post-transition rebuilds only php-8-3-32-trixie.
  • shellcheck, actionlint and docker buildx bake --print (default variables, local usage) all pass.

Behavioral changes worth maintainers' attention

  • Embedded BuildKit provenance is no longer attached to the pushed images (GitHub-signed attestations remain).
  • org.opencontainers.image.created (and the OCI config created field) now reflect the built commit's date instead of the build date.
  • Runner images no longer carry the dev.frankenphp.base.fingerprint label (builder images keep it).

Out of scope (accepted): builder images (Go build cache) and Alpine (apk db not double-build-tested) are not guaranteed reproducible; with per-variant skipping they are only rebuilt when their own bases change anyway.

🤖 Generated with Claude Code

…reproducible

Scheduled rebuilds used a single fingerprint covering the 15 base images of
all variants: any single base update (e.g. php:8.3-zts-trixie or
golang:1.26-trixie) rebuilt and re-pushed all 24 images, moving every generic
tag (php8.4, 1-php8.4, ...) roughly every other day. Comparing two such
rebuilds of the same release showed byte-identical content (same dpkg
versions, same frankenphp binary) but different digests, caused only by build
artifacts: timestamp() label, layer mtimes and embedded provenance.

- Compute one fingerprint per variant (its php-base + golang-base digests)
  and only rebuild/push the variants whose fingerprint changed. The label is
  read from the versioned builder tag; the legacy all-variants fingerprint is
  still accepted so the transition (old bake file in the latest release tag)
  keeps the current behavior until the next release.
- Store the fingerprint label on builder images only, so runner digests
  depend solely on their content.
- Make images reproducible: SOURCE_DATE_EPOCH and created/revision labels
  pinned to the built commit, rewrite-timestamp=true on push, BuildKit
  provenance disabled (its timestamps made every digest unique; the signed
  GitHub attestations pushed by actions/attest are kept), and removal of
  files with non-deterministic content (apt logs, ldconfig aux-cache) from
  Debian runner layers. Two independent no-cache builds of the 8.4-trixie
  runner now produce the same digest.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@xavierleune
xavierleune marked this pull request as ready for review July 21, 2026 08:08
@dunglas

dunglas commented Jul 21, 2026

Copy link
Copy Markdown
Member

I like the idea, but we need to double check that this won't break anything. It's a tricky part of the CI!

@xavierleune

Copy link
Copy Markdown
Contributor Author

yep that's super tricky.
I got into this because I was tired to have a my CI to do a full rebuild of my extensions every other day. It may improve things a lot to help keeping deps fresh but rebuild only when needed. The climate will thank us 😄

LMK if I can help

@henderkes

Copy link
Copy Markdown
Contributor

That's why I use my packages in ubi 10 docker containers 🤣. Rob referred to that time as Deskation, but I'm far too impatient.

I'll review later, but it's a good idea.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants