feat: add archive infrastructure for EOL distribution packages by abtreece · Pull Request #47 · fullstaq-ruby/infra

abtreece · 2026-04-25T02:57:36Z

Summary

Add GCS buckets for APT/YUM archive repositories (public-read, no CI write access)
Add Azure DNS zones, NS delegation, and A/AAAA records for apt-archive.fullstaqruby.org and yum-archive.fullstaqruby.org
Add Caddy server blocks to redirect archive subdomains to the versioned GCS bucket paths
Update query-latest-repo-versions.rb to query archive bucket versions, with graceful fallback (version 0) before first migration

Context

Addresses fullstaq-ruby/server-edition#190 — CI disk space exhaustion from the growing Aptly state archive.

Archive repos follow the PostgreSQL (apt-archive.postgresql.org) and HashiCorp (archive.releases.hashicorp.com) pattern: separate archive repositories, served as static redirects to GCS. The archive holds every package whose Ruby version or distribution is EOL; the live repos hold only packages where both are supported, so the two sets never overlap. The archive grows as Ruby versions and distributions reach EOL.

Companion PR: fullstaq-ruby/server-edition#191 (migration scripts, build config, runbook)
Implementation plan: context/plans/EOL-ARCHIVE-MIGRATION.md (Tasks 1-3)

Design notes

Archive buckets intentionally lack CI write access. The live apt/yum buckets grant roles/storage.objectAdmin to the github-ci-deploy workload identity; the archive buckets deliberately omit this binding. Migration happens out-of-band via the companion server-edition#191 scripts, so packages enter the archive only through a deliberate migration — enforced in IAM, not by convention.
NS delegation timing. Each archive subdomain is its own azurerm_dns_zone with an NS record in the apex zone. Caddy's ACME DNS-01 challenge for the new subdomains will fail until the apex NS records propagate, so the Caddy restart in step 3 must happen after propagation completes (verified by step 2's dig check), not concurrent with terraform apply.
Archive buckets set force_destroy = false, unlike the live repo buckets. The live buckets can be rebuilt by CI; the archives cannot, so a bucket replacement or terraform destroy fails on a non-empty archive bucket instead of deleting its contents.
Caddy reads repo versions only at startup. Migration runs outside CI, so nothing restarts Caddy afterwards, and /admin/restart_web_server only accepts deploy-environment OIDC tokens. Caddy must be restarted by hand after each migration, or the archive hosts keep redirecting to versions/0/ and return 404.

Deployment sequence

terraform apply in terraform/ to create buckets and DNS
Wait for DNS propagation (verify with dig per test plan)
Restart Caddy (via Ansible or SSH) to pick up new Caddyfile and env vars
Run migration scripts from the server-edition companion PR
Restart Caddy again (sudo systemctl restart caddy) so it picks up the new archive versions

Test plan

terraform init -backend=false && terraform validate && terraform fmt -check -diff -recursive passes
DNS resolves: dig apt-archive.fullstaqruby.org A and dig yum-archive.fullstaqruby.org A return backend server IP
Caddy starts without error even before first migration (archive version defaults to 0)
After migration: curl -I https://apt-archive.fullstaqruby.org/dists/ returns 302 to GCS

noahssarcastic

Looks good, reviewed in tandem with fullstaq-ruby/server-edition#191.

FooBarWidget

Also looks good to me, agree with @noahssarcastic: query_repo_version could use some code clarity improvements but not blocking.

Also, the infra overview docs need to be updated. Since #57 is not yet merged, I suggest updating #57 with the changes made in this PR #47.

Feel free to deploy @abtreece.

Documents the new APT/YUM archive infrastructure introduced in this PR: the two new public-read GCS buckets, the deliberate absence of CI write access (frozen-mirror invariant enforced in IAM), the Azure DNS zones and apex NS delegation for the archive subdomains, and the 404 fallback behavior in query-latest-repo-versions.rb that lets the web server start cleanly before the first migration runs. Addresses FooBarWidget's note on PR fullstaq-ruby#57 that fullstaq-ruby#47's changes should ship with their own documentation.

@noahssarcastic

Address review feedback on PR fullstaq-ruby#47 from @noahssarcastic and @FooBarWidget: the awkward `suffix:` parameter in query-latest-repo-versions.rb was a downstream symptom of inconsistent bucket naming (`-repo-archive` broke the `-repo` terminal-suffix convention used by the live buckets). Rename: <prefix>-server-edition-apt-repo-archive -> -apt-archive-repo <prefix>-server-edition-yum-repo-archive -> -yum-archive-repo Safe to do now -- the buckets don't exist yet (PR not deployed). With consistent naming, query_repo_version becomes a direct lookup: `type` is the bucket-name slot, and the only special-casing for archives is a semantic `allow_missing:` flag for the pre-migration bootstrap window. No string manipulation, no latent type/suffix mismatch footgun. Coordination: server-edition#191's CI workflow sets ARCHIVE_REPO_BUCKET_NAME -- that value will need to match the new bucket name before the first migration runs.

abtreece · 2026-05-23T02:21:52Z

For reviewers tracking both PRs: c1814eb renames the archive buckets from -repo-archive to -archive-repo for naming consistency with the live buckets. Companion update to #191's runbook + script docstrings is in fullstaq-ruby/server-edition@0f8fc5d (no script logic changes — they only reference ENV['ARCHIVE_REPO_BUCKET_NAME']).

FooBarWidget

Feel free to deploy.

abtreece · 2026-05-30T15:04:04Z

Feel free to deploy.

@FooBarWidget I've been deep in some home repair things this past week. I'll try to get it deployed this weekend.

The previous code degraded any non-2xx response (auth errors, 5xx, redirects) to LATEST_VERSION=0 whenever the suffix indicated an archive bucket. That silently turns a transient outage or misconfiguration into Caddy redirecting clients to /versions/0/... 404s — broken behavior masquerading as valid config. Only treat HTTP 404 as the legitimate "archive not yet populated" case; surface every other failure.

Documents the new APT/YUM archive infrastructure introduced in this PR: the two new public-read GCS buckets, the deliberate absence of CI write access (frozen-mirror invariant enforced in IAM), the Azure DNS zones and apex NS delegation for the archive subdomains, and the 404 fallback behavior in query-latest-repo-versions.rb that lets the web server start cleanly before the first migration runs. Addresses FooBarWidget's note on PR fullstaq-ruby#57 that fullstaq-ruby#47's changes should ship with their own documentation.

@noahssarcastic

Address review feedback on PR fullstaq-ruby#47 from @noahssarcastic and @FooBarWidget: the awkward `suffix:` parameter in query-latest-repo-versions.rb was a downstream symptom of inconsistent bucket naming (`-repo-archive` broke the `-repo` terminal-suffix convention used by the live buckets). Rename: <prefix>-server-edition-apt-repo-archive -> -apt-archive-repo <prefix>-server-edition-yum-repo-archive -> -yum-archive-repo Safe to do now -- the buckets don't exist yet (PR not deployed). With consistent naming, query_repo_version becomes a direct lookup: `type` is the bucket-name slot, and the only special-casing for archives is a semantic `allow_missing:` flag for the pre-migration bootstrap window. No string manipulation, no latent type/suffix mismatch footgun. Coordination: server-edition#191's CI workflow sets ARCHIVE_REPO_BUCKET_NAME -- that value will need to match the new bucket name before the first migration runs.

Archive buckets hold frozen EOL packages that CI cannot regenerate, so a bucket replacement or destroy must fail rather than silently delete their contents.

Copilot

Copilot review overview

🟡 Changes recommended

The companion tooling deletes some EOL packages instead of archiving them, contradicting the documented archive invariant.

Review effort: Balanced
Findings: 1 · 1

Open (2)

Archive EOL Ruby packages before pruning supported distributions · New
Reconcile contradictory web-server and DNS architecture descriptions · New

What changed in this PR

Adds infrastructure for serving archived EOL APT/YUM packages separately from live repositories.

Changes:

Adds public archive buckets and delegated Azure DNS zones.
Adds Caddy redirects with pre-migration version fallback.
Documents archive operation and migration requirements.

File	Description
`terraform/repo_buckets.tf`	Defines protected, publicly readable archive buckets.
`terraform/dns.tf`	Adds archive DNS zones and records.
`ansible/files/Caddyfile`	Redirects archive domains to versioned GCS paths.
`ansible/files/query-latest-repo-versions.rb`	Queries archive versions with 404 fallback.
`docs/infrastructure-overview.md`	Documents archive architecture and operations.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+
+- Administered by role: Infra Maintainers
+
+The Server Edition's APT and YUM archive repositories hold every package whose Ruby version *or* distribution has reached end-of-life. The live repos hold only packages where both are still supported, so the two sets never overlap. The archive only grows: packages move into it as Ruby versions and distributions reach end-of-life.


+
+Both buckets are publicly readable. Unlike the live APT/YUM repo buckets, the archive buckets deliberately have **no CI write access** — packages enter the archive only through a deliberate migration, and that is enforced in IAM rather than by convention. Migration into these buckets happens out-of-band via scripts in the [server-edition repository](https://github.com/fullstaq-ruby/server-edition).
+
+Users access these archives via `apt-archive.fullstaqruby.org` and `yum-archive.fullstaqruby.org`, which redirect to the bucket contents. Each archive subdomain has its own Azure DNS zone, delegated via NS records in the `fullstaqruby.org` apex zone, with A/AAAA records pointing at the backend server.


abtreece · 2026-09-30T02:00:47Z

@FooBarWidget @noahssarcastic could one of you re-approve? The earlier approvals were dismissed by new pushes. Since the last approval (8aea6f4):

ace1c57 sets force_destroy = false on the archive buckets. CI can't regenerate their contents, so Terraform should refuse to delete a non-empty archive bucket.
d1c1eb5 documents that Caddy must be restarted by hand after each archive migration. It only reads repo versions at startup, and migration runs outside CI.
de41bb6 corrects the docs and PR description: the archive holds packages where the Ruby version or the distro is EOL (per the clarification on feat: add EOL package archival tooling server-edition#191). It's no longer described as a frozen EOL-distro mirror.
Rebased onto current main (74081fd), which also drops the two merge commits. The PR diff is otherwise unchanged, and CI is green on de41bb6.

No Terraform, Caddyfile or Ruby logic changed since your approval apart from the force_destroy flip.

abtreece mentioned this pull request Apr 25, 2026

feat: add EOL package archival tooling fullstaq-ruby/server-edition#191

Open

4 of 6 tasks

abtreece requested a review from FooBarWidget May 15, 2026 05:06

noahssarcastic previously approved these changes May 19, 2026

View reviewed changes

Comment thread ansible/files/query-latest-repo-versions.rb Outdated

FooBarWidget previously approved these changes May 21, 2026

View reviewed changes

Comment thread ansible/files/query-latest-repo-versions.rb Outdated

FooBarWidget mentioned this pull request May 21, 2026

docs: refresh infra docs for post-Hetzner architecture #57

Open

abtreece dismissed stale reviews from FooBarWidget and noahssarcastic via b58dae6 May 22, 2026 20:03

abtreece requested review from FooBarWidget and noahssarcastic May 23, 2026 02:22

FooBarWidget previously approved these changes May 28, 2026

View reviewed changes

noahssarcastic previously approved these changes Jun 22, 2026

View reviewed changes

abtreece dismissed stale reviews from noahssarcastic and FooBarWidget via 751e334 September 24, 2026 02:29

abtreece added 9 commits September 29, 2026 20:43

feat: add GCS buckets for APT/YUM archive repositories

617dcf3

feat: add DNS zones for APT/YUM archive subdomains

15bfac2

feat: add Caddy config and version query for archive repos

6db841a

fix: disable force_destroy on archive buckets

ace1c57

Archive buckets hold frozen EOL packages that CI cannot regenerate, so a bucket replacement or destroy must fail rather than silently delete their contents.

docs: require Caddy restart after archive migration

d1c1eb5

docs: describe archive as EOL Ruby or EOL distro, not frozen

de41bb6

abtreece force-pushed the feat/eol-archive-repos branch from 54364e8 to de41bb6 Compare September 30, 2026 01:44

abtreece requested a balanced review from Copilot September 30, 2026 01:50

Copilot started reviewing on behalf of abtreece September 30, 2026 01:51 View session

Copilot AI reviewed Sep 30, 2026

View reviewed changes

abtreece requested review from FooBarWidget and noahssarcastic September 30, 2026 03:08

Navigation Menu

feat: add archive infrastructure for EOL distribution packages - #47

feat: add archive infrastructure for EOL distribution packages#47
abtreece wants to merge 9 commits into
fullstaq-ruby:mainfrom
abtreece:feat/eol-archive-repos

abtreece commented Apr 25, 2026 •

edited

Loading

Uh oh!

noahssarcastic left a comment

Uh oh!

Uh oh!

FooBarWidget left a comment

Uh oh!

Uh oh!

abtreece commented May 23, 2026

Uh oh!

FooBarWidget left a comment

Uh oh!

abtreece commented May 30, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

abtreece commented Sep 30, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants


		- Administered by role: Infra Maintainers

		The Server Edition's APT and YUM archive repositories hold every package whose Ruby version or distribution has reached end-of-life. The live repos hold only packages where both are still supported, so the two sets never overlap. The archive only grows: packages move into it as Ruby versions and distributions reach end-of-life.


		Both buckets are publicly readable. Unlike the live APT/YUM repo buckets, the archive buckets deliberately have no CI write access — packages enter the archive only through a deliberate migration, and that is enforced in IAM rather than by convention. Migration into these buckets happens out-of-band via scripts in the [server-edition repository](https://github.com/fullstaq-ruby/server-edition).

		Users access these archives via `apt-archive.fullstaqruby.org` and `yum-archive.fullstaqruby.org`, which redirect to the bucket contents. Each archive subdomain has its own Azure DNS zone, delegated via NS records in the `fullstaqruby.org` apex zone, with A/AAAA records pointing at the backend server.

Conversation

abtreece commented Apr 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Context

Design notes

Deployment sequence

Test plan

Uh oh!

noahssarcastic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

FooBarWidget left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

abtreece commented May 23, 2026

Uh oh!

FooBarWidget left a comment

Choose a reason for hiding this comment

Uh oh!

abtreece commented May 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

abtreece commented Sep 30, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

abtreece commented Apr 25, 2026 •

edited

Loading