Consolidate libforge and the satellite services into the repository

[Peeja]

[From Claude:]

Stacked on #6 (claude/forge-consolidation); review and merge that first. This PR is the consolidation itself: the code that libforge held becomes three in-repo modules, three satellite services join, the five existing services are rewired onto the in-repo modules, and CI, the container build and the root Makefile stop assuming a fixed set of four services. Nothing outside this repository changes: libforge, guppy, delegator, piri-signing-service, indexing-service and go-ipni-tools keep their repositories and pipelines until the team decides to retire them.

Eight commits, each meant to be reviewed on its own — the commit messages carry the detail; this description is the map.

Commits

#	Commit	What it adds
1	Add protocol: the wire contract as an in-repo module	protocol/ (module github.com/fil-forge/forge/protocol): commands/**, blobindex, receipt, retrieval, bytemap, digestutil from libforge main 81372e7 plus PR fil-forge/libforge#52 (f4b13f7, rebased). Its own Makefile with the codegen gate (make codegen-build gen-check).
2	Add internal: repo-private helpers shared by the services	internal/ (module github.com/fil-forge/forge/internal): jobqueue, identity, piece (libforge); sigv4, s3perm, client/hilt (from live hilt, which is newer than libforge#52's copies); zapucan (libforge#52); client/delegator (delegator's client/); pdpsigner/{types,signer,inprocess,client} (piri-signing-service's pkg/*); ipni/advertisement (a copy of go-ipni-tools' pkg/advertisement, see below). The Go toolchain enforces the internal/ rule across modules under github.com/fil-forge/forge/, so this module is repo-private by construction.
3	Add attestation: the did:mailto attestation extension as its own module	attestation/: attestation, didmailto, attest (was libforge/commands/ucan/attest). Depends on ucantone only.
4	Add delegator, piri-signing-service and indexing-service as services	From their current mains (3d3525c, 8472ad9, a119ea8), module paths rewritten, per-repo .github/, Dockerfile*, deploy/, .storoku.json dropped (source only; each storoku pipeline stays in its repo). indexing-service's tracked 24 MB Lambda bootstrap binary is dropped too. The library halves of delegator and piri-signing-service live in internal/ (commit 2).
5	Rewire the services onto the in-repo modules	piri, hilt, sprue, ingot, smelt (and the stress tester) import protocol/internal/attestation instead of libforge; hilt drops the copies that moved (pkg/sigv4, pkg/s3perm, pkg/lib/zapucan, the S3 RPC client, the auth/bucket error sentinels) and keeps its server side, AdminClient and upload client. go.mods lose the external libforge/hilt/smelt/indexing-service requires and gain replace directives to the in-repo modules. Prose that still said "libforge" is updated.
6	smelt: follow the shared modules and the root build context	pkg/workspace rebuilds every service when any of protocol, internal, attestation is in the workspace (was: libforge); pkg/stack builds images with the repository root as context; tests/compat's otherThan maps service names explicitly and panics on an unknown one instead of vacuously passing.
7	Build and test every module in CI; images from the repository root	ci.yml: a replaces job (.github/scripts/check-replaces.sh), a static 12-module unit matrix with the codegen gate on protocol/attestation, and the stack job building all 7 service images with context: .. docker/Dockerfile becomes a root-context build (SERVICE, MAIN_PKG, BIN, BUILD_TAGS). publish-ghcr.yml publishes 7 services. Root Makefile (SERVICES, LIBRARIES, gen-check, check-replaces), .dockerignore, go.work (12 modules), README.md.
8	smelt: probe the satellites' health with a GET, not wget --spider	What the first stack run on this branch found: the shared debian image ships GNU wget, whose --spider probes with HEAD (busybox's, in the satellites' alpine images, probes with GET), and echo answers HEAD on a GET-only health route with 405 — so the signer (and the delegator, identically built) came up "unhealthy" and compose up aborted. The three satellite probes become wget -q -O /dev/null …, a GET under both wgets with the same pass/fail semantics; reproduced against the signer's echo version before pushing.

How the pieces relate

protocol ──► ucantone            (wire contract; the only module downstream code needs)
internal ──► protocol            (repo-private helpers)
attestation ──► ucantone
piri, hilt, sprue, ingot, delegator, piri-signing-service, indexing-service ──► protocol, internal, attestation (as needed)
hilt ──► smelt/pkg/stack         (its integration tests)
ingot ──► indexing-service       (the indexer's query client)
smelt ──► protocol, internal

• Development uses go.work; CI and the container build use GOWORK=off with replace directives, so each module is hermetic on its own go.mod. A missing replace resolves github.com/fil-forge/forge/... through the proxy to the repository root and fails confusingly ("module found but does not contain package"), which is what check-replaces.sh guards against.
• The S3 wire contract (protocol/commands/s3{,/request,/bucket}: Operation, OperationFor, ClassifyRequest, the error sentinels) is one module, so hilt, ingot and internal/client/hilt share identical types — the design of libforge#52, realised in-repo with live hilt's newer sigv4/s3perm/client code. ErrBucketAlreadyOwned (live hilt) is ported into protocol/commands/s3/bucket.
• ucantone is pinned to v0.0.0-20260904190501-7fca40e13941, the head of Add ucanlib (from libforge) and fixed test identities to testutil ucantone#57 (the fixtures and ucanlib that the rest of the tree needs). The pseudo-version resolves through the module proxy today; bump it once #57 merges.

Deliberately not in this PR

• release.yml is untouched. Per-service tags for the three new services (and whether the library modules get tags at all) need a decision first.
• Deploy pipelines of delegator, piri-signing-service and indexing-service stay in their repositories and keep deploying from there; pointing them at images built here is a follow-up.
• The client (guppy's pkg/client/locator/tokenstore and ingot's forgeclient fork) — Phase 4, in Add forgeclient: ingot's Forge client as a shared module #8.
• go-ipni-tools still depends on libforge: pkg/queue imports libforge/jobqueue and its tests use libforge/testutil. Forge therefore carries its own copy of pkg/advertisement under internal/ipni/advertisement. Once go-ipni-tools moves pkg/queue onto its own job queue (or internal/jobqueue's successor) and its tests onto ucantone/testutil, the copy can go back to an import.
• swarf stays an external module and still pins libforge, so libforge remains an indirect dependency of hilt and ingot (via swarf/pkg/client) and of indexing-service (via go-ipni-tools/pkg/queue). No in-repo package imports it.
• No apidiff gate and no go work sync check yet.

Verification (this branch, GOWORK=off unless noted)

• All 12 modules: go build ./..., go vet ./..., go mod tidy leaves no diff (piri with -tags skiff, as CI builds it).
• Tagged suites compile: smelt -tags itest ./tests/s3, -tags e2e ./tests/e2e, -tags compat ./tests/compat; hilt -tags itest ./itest.
• make -C protocol codegen-build gen-check, make -C attestation codegen-build gen-check: generated codecs are current.
• .github/scripts/check-replaces.sh: every in-repo require has a matching replace.
• Workspace mode (go.work): every module builds.
• Unit tests pass for every module except the packages that need a Docker daemon (smelt pkg/stack image builds, indexing-service pkg/aws and pkg/redis testcontainers, piri's minio store), which this sandbox has no daemon for; CI runs them.
• CI on this branch so far: the replace check and all 12 unit jobs are green, and the stack job built all seven images from the root context. Its e2e step failed on the health probe described under commit 8; the stack job re-runs with that fix.

Reviewing

Commit by commit. Commits 1–4 are mostly file moves: git show --stat -M shows the moves, and each can be checked against its source (diff -r against libforge at 81372e7+PR 52, or the satellite's main at the SHA above; the differences are module paths, dropped deploy material and gofmt). Commit 5 is the one with hand edits, concentrated in hilt; commit 7 is the CI/Docker/Makefile change; commit 8 is four files.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CGAGAib517Ae1kg8SCdcEt