Skip to content

fix: bypass expired Debian 11 security repo in Dockerfile.build - #7782

Merged
ycombinator merged 1 commit into
elastic:mainfrom
ycombinator:fix/debian11-expired-security-repo
Sep 9, 2026
Merged

ycombinator merged 1 commit into
elastic:mainfrom
ycombinator:fix/debian11-expired-security-repo

Conversation

@ycombinator

Copy link
Copy Markdown
Contributor

⚠️ TEMPORARY WORKAROUND — Revert once elastic/golang-crossbuild#754 is merged and new images are published

What is the problem this PR solves?

The Buildkite Package x86_64 and Package aarch64 jobs are failing (e.g. build #16689) because Debian 11 (Bullseye) reached EOL and its security repository's InRelease file has expired. apt-get update in Dockerfile.build fails with:

E: Release file for http://deb.debian.org/debian-security/dists/bullseye-security/InRelease is expired

The root cause fix is in the upstream base image: elastic/golang-crossbuild#754, which switches Debian 11 sources to the archive and snapshot mirrors. This PR should be reverted once that PR is merged and new golang-crossbuild images are published.

How does this PR solve the problem?

Passes -o Acquire::Check-Valid-Until=false to apt-get update in Dockerfile.build to bypass the expiry check. This does not change the Debian version being built for — the base image remains golang-crossbuild:*-debian11. Since Debian 11 is EOL, the security repo has no new updates regardless, so bypassing the validity check has no practical security impact.

How to test this PR locally

Trigger a packaging build or run locally:

docker build -t fleet-server-builder:test --build-arg GO_VERSION=$(cat .go-version) --build-arg SUFFIX=main-debian11 -f Dockerfile.build .

Design Checklist

  • I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
  • I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
  • I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

  • I have commented my code, particularly in hard-to-understand areas

Related issues

Temporary workaround for the Buildkite package build failures caused by
the Debian 11 (Bullseye) security repository's InRelease file expiring
after EOL. The root cause fix is upstream in golang-crossbuild:
elastic/golang-crossbuild#754

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 8, 2026 16:40
@ycombinator
ycombinator requested a review from a team as a code owner September 8, 2026 16:40
@ycombinator ycombinator added bug Something isn't working Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team labels Sep 8, 2026
@ycombinator ycombinator added the backport-active-all Automated backport with mergify to all the active branches label Sep 8, 2026
@ycombinator
ycombinator requested a review from samuelvl September 8, 2026 16:40
@ycombinator
ycombinator requested a review from ebeahan September 8, 2026 16:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is narrowly scoped to the stated CI breakage, is clearly documented as temporary, and preserves existing build behavior aside from bypassing the expired metadata check.

Pull request overview

This PR applies a temporary workaround to keep Fleet Server’s packaging/build image functional now that Debian 11 (Bullseye) security repository metadata has expired, unblocking CI packaging jobs that run apt-get update inside Dockerfile.build.

Changes:

  • Add an explanatory comment documenting the temporary nature of the workaround and linking the upstream fix to revert to.
  • Pass -o Acquire::Check-Valid-Until=false to apt-get update so APT proceeds despite expired InRelease metadata.
File summaries
File Description
Dockerfile.build Bypasses Debian 11 repo metadata “Valid-Until” expiry during apt-get update, with clear revert guidance tied to the upstream base image fix.
Review details
  • Files reviewed: 1/1 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Caution

agentic threat detected
Threat detection flagged this output in warn mode. Manual review is REQUIRED before any follow-up automation.

Details

Potential security threats were detected in the agent output.

Review the workflow run logs for details.

TL;DR

The failing Package x86_64 FIPS Buildkite job is not a source-code regression in this PR; it failed during artifact upload to GCS due to a transient Google auth/backend outage (503 UNAVAILABLE during token introspection). Re-run the build/job.

Remediation

  • Re-run Buildkite build #16691 (or just the Package x86_64 FIPS job); no code changes are indicated by this failure.
  • If it recurs, escalate to CI/platform owners for GCS/OAuth service health and retry policy review.
Investigation details

Root Cause

The only hard failure in the provided job log is from the upload step in .buildkite/scripts/release_test.sh, where gsutil/GCS auth fails with a service-side 503 during token introspection.

PR change scope is limited to Dockerfile.build and successfully passes through build/package stages before upload. The log shows distribution artifacts were produced and Buildkite artifact upload succeeded, which further indicates the failure is external to repository code.

Relevant changed file in PR:

  • Dockerfile.build (single-file change; apt update flag adjustment)

Evidence

  • Build: https://buildkite.com/elastic/fleet-server/builds/16691
  • Job/step: Package x86_64 FIPS → .buildkite/scripts/release_test.sh
  • Key log excerpt:
    ERROR: Task '(fleetserverciinternal/redacted) failed: TokenIntrospectionError('{\n  "error": {\n    "code": 503,\n    "message": "The service is currently unavailable.",\n    "status": "UNAVAILABLE"\n  }\n}\n')
    
  • Additional supporting log signals:
    • Artifacts were created/copied before the auth error.
    • Buildkite artifact upload at end succeeded (Artifact uploads completed successfully).

Verification

  • Not run locally in this environment; analysis is based on the provided Buildkite failure summary and full job log.

Follow-up

  • If retries keep failing, add retry/backoff around the GCS upload path in CI orchestration and investigate upstream auth service incidents around build time.

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

@ycombinator
ycombinator merged commit bc44a05 into elastic:main Sep 9, 2026
17 checks passed
@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

@Mergifyio backport 9.5 9.4 8.19

@mergify

mergify Bot commented Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

@ycombinator

Copy link
Copy Markdown
Contributor Author

No longer needed as elastic/golang-crossbuild#754 was merged. Will revert.

@ycombinator
ycombinator deleted the fix/debian11-expired-security-repo branch September 9, 2026 10:32
ycombinator added a commit that referenced this pull request Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-active-all Automated backport with mergify to all the active branches bug Something isn't working skip-changelog Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants