diff --git a/.changeset/cli-install-sql-digest.md b/.changeset/cli-install-sql-digest.md new file mode 100644 index 000000000..197c7f958 --- /dev/null +++ b/.changeset/cli-install-sql-digest.md @@ -0,0 +1,34 @@ +--- +'stash': patch +--- + +Verify the EQL install SQL against its release digest before running it. + +`stash eql install` reads the EQL v3 bundle from the resolved +`@cipherstash/eql` in your `node_modules` and executes it against your +database. That read was a bare `readFileSync` — nothing checked that the bytes +on disk were the bundle the resolved release actually ships. A corrupt, +partially-updated, or tampered package installed silently: the database ended +up carrying SQL the version it reports does not define, and the CLI printed +"EQL extensions installed." + +The CLI now hashes the bundle and compares it to `installSqlSha256` from the +release manifest that ships alongside it, and **refuses** on a mismatch. The +error names the expected digest, the actual digest, the resolved file path and +the EQL version, so the remedy is visible rather than inferred. Verification +happens before any database connection is opened, so a refusal means nothing +was attempted — not that something was rolled back. + +The check covers all three paths that read the bundle: `stash eql install`, +the SQL embedded by `stash eql migration --drizzle` / `--supabase`, and the +expected-surface baseline `stash eql verify` compares your database against. +`@cipherstash/stack-prisma` has verified against this same digest since its v3 +migrations landed; this brings the CLI in line. + +No healthy install is affected — the SQL and its manifest are produced by the +same build of `@cipherstash/eql`, so a mismatch only ever means a broken +dependency tree. + +`skills/stash-cli` documents the new pre-flight alongside the existing +post-install surface check, so an agent reading it does not report a digest +refusal as a failed install. diff --git a/.changeset/eql-3-0-5-migration.md b/.changeset/eql-3-0-5-migration.md new file mode 100644 index 000000000..49e195b93 --- /dev/null +++ b/.changeset/eql-3-0-5-migration.md @@ -0,0 +1,96 @@ +--- +'@cipherstash/stack-prisma': minor +--- + +Move the bundled EQL v3 migrations to **eql-3.0.5**, which renames the SQL +function `eql_v3.ste_vec_contains` to `eql_v3.jsonb_document_contains`. + +**The blast radius is narrower than a renamed public function suggests.** The +`@>` / `<@` operators on `public.eql_v3_json_search` behave exactly as before, +and so do the two function-form entry points that exist for platforms without +operator support — `eql_v3.jsonb_contains(jsonb, jsonb)` and +`eql_v3.jsonb_contained_by(jsonb, jsonb)` are byte-identical to 3.0.4. Those +are what a PostgREST caller invokes, so PostgREST callers on the documented +surface are **not** affected. The renamed function is the typed implementation +those operators dispatch into. + +**And the old name still works.** eql-3.0.5 ships `eql_v3.ste_vec_contains` as +a deprecated delegating alias for both overloads, so hand-written SQL naming it +— an application query, a view, an RLS policy, or a per-function +`GRANT EXECUTE ON FUNCTION eql_v3.ste_vec_contains(…)` — keeps resolving. The +typed overload stays inlinable, so a function-form query through the alias +still matches the same functional GIN index. Migrate to +`jsonb_document_contains` when convenient; nothing forces it at upgrade time. + +**Separately — and true of every EQL upgrade, not just this one:** the install +bundle opens with `DROP SCHEMA IF EXISTS eql_v3 CASCADE`, so applying it drops +every object in `eql_v3` / `eql_v3_internal` and everything that depended on +them. **Encrypted data and column types are not affected** — the storage +domains are `public.eql_v3_*`, deliberately outside both dropped schemas, and +their CHECK functions are re-created rather than dropped. What does not survive +is everything else pointing into the schema, which is two actions, neither of +them to do with the rename: + +1. **Re-run your grant script.** Every grant on every `eql_v3` / + `eql_v3_internal` object is gone. The schema-wide form EQL documents — + `GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA eql_v3 TO app_role` — picks up + both the new name and the alias on its own. +2. **Recreate your functional indexes, then `ANALYZE`.** Indexes over + `eql_v3.eq_term(…)` / `ord_term` / `match_term` / `to_ste_vec_query(…)` + depend on the dropped schema and go with it. Nothing errors afterwards: + encrypted predicates keep working and silently fall back to sequential + scans. A migration runner will not redo an already-applied migration, so + this has to be a *new* one. The `stash-indexing` skill documents the + mechanism ("These indexes do not survive an EQL reinstall or upgrade") and + the `EXPLAIN` check that confirms recovery; capturing and restoring them + automatically is tracked in + [cipherstash/stack#918](https://github.com/cipherstash/stack/issues/918). + +Any RLS policy, view, or constraint that calls an `eql_v3` function is dropped +by the same CASCADE and needs recreating too. **The rename itself needs no +action — the alias makes it non-breaking.** + +Two artefacts carry the new bundle: + +- A new upgrade edge, `20260814T0000_upgrade_eql_v3_3_0_5`, carrying the + invariant `cipherstash:upgrade-eql-v3-bundle-3.0.5-v1`. Databases already + running an earlier bundle re-install through this edge on the next + `prisma-next migration plan` followed by `prisma-next migrate`, exactly as + they did for 3.0.2 and 3.0.4. **`migrate` alone is not enough** — the seed + phase that copies a new migration package into your repo runs only from + `migration plan`, so without it the 3.0.5 directory never reaches disk and + `migrate` is a silent no-op that leaves the database on the older bundle. +- The baseline install migration `20260601T0100_install_eql_v3_bundle`, whose + baked bundle moves to 3.0.5 and which gains a fourth no-SQL carrier op for + the new invariant. Fresh databases therefore land on 3.0.5 from the single + all-additive genesis edge, keeping `db init` (additive-only policy) working. + +**Action required.** The baseline's bytes — and so its `migrationHash` — have +changed. If your project already has a `migrations/cipherstash/` directory +generated against `@cipherstash/stack-prisma@1.0.0` or `@1.1.0`, delete that +directory and re-run `prisma-next migration plan` (or `migrate`); the 1.1.0 +Prisma Next 0.17 upgrade re-anchored the same artefacts, so a space vendored +against either release is stale here. The seed phase regenerates +it byte-identical to the shipped artefacts. Your database keeps its markers, so +already-applied invariants are not re-run — the only new work is the 3.0.5 +upgrade edge. + +If you skip the delete, nothing warns you: a vendored baseline is stale but +internally intact, so it passes every integrity check. On an existing database +the upgrade still applies correctly; on a **fresh** one, `db init` refuses with +`Operation cipherstash.upgrade-eql-v3-bundle-3.0.5 has class "data" which is +not allowed by policy.` — an error that names neither the directory nor the +remedy. See "Upgrading from 1.0.0" in the package README. + +**Why the baseline was re-emitted rather than left frozen.** These artefacts are +content-addressed and normally append-only: an EQL bump ships as a new upgrade +directory and published directories are never rewritten. That rule cannot be +followed here without a second `from: null` genesis edge, because no upgrade +edge can ever be walked by `db init` — every upgrade edge is a self-edge, and +the integrity checker requires a self-edge to carry a `data`-class op, which +`db init`'s additive-only policy refuses. A fresh database must therefore +collect every head-ref invariant from the genesis edge it walks. The +append-only alternative would duplicate the full ~2.6 MB bundle into a new +genesis edge on every EQL release, permanently; re-emitting was taken instead +while 1.0.0 was two weeks old with negligible adoption, and is a decision to be +re-argued on adoption numbers rather than repeated by default. diff --git a/.changeset/eql-exact-runtime-pin.md b/.changeset/eql-exact-runtime-pin.md new file mode 100644 index 000000000..00b4eef9f --- /dev/null +++ b/.changeset/eql-exact-runtime-pin.md @@ -0,0 +1,41 @@ +--- +'@cipherstash/stack-prisma': patch +'stash': patch +--- + +Pin the packed `@cipherstash/eql` dependency to an exact version, closing a +route by which an installed EQL bundle could drift ahead of the code built +against it. + +Both packages declared `"@cipherstash/eql": "workspace:^"` under +`dependencies`. In this workspace that resolves in-tree either way, so nothing +in development or CI could see a difference — but the two specifiers do not +pack the same. pnpm rewrites the protocol when it builds the tarball a customer +actually installs: + + "workspace:^" packs as "^3.0.5" + "workspace:*" packs as "3.0.5" + +The caret is the problem. `@cipherstash/eql` is still published from +`cipherstash/encrypt-query-language` until the publisher repoint, so a 3.0.x can +reach npm without passing through this repository at all — and `^3.0.5` accepts +it. A customer installing `stash` or `@cipherstash/stack-prisma` would then get +SQL that STORES and queries encrypted payloads at one version, while +`@cipherstash/stack`'s v3 domain types (which EMIT those payloads) and +`stack-prisma`'s baked migrations stayed frozen at the version this repo built +and tested against. The two halves of EQL are released in lockstep precisely +because that skew does not fail at install or in CI — it fails in a database. + +`workspace:*` is the only form that closes it. A literal `"3.0.5"` would be an +exact pin too, but it is a registry pin: `pnpm run lint:eql-pins` rejects it, +because resolving EQL from a registry rather than from this repo is the same +drift one layer up. + +No API, behaviour or SQL changes. What changes is the dependency range in the +published tarballs, and only in the narrowing direction — the version resolved +today is the version that was already being resolved. Nothing needs to be done +on upgrade. + +`@cipherstash/stack` declares the same dependency under `devDependencies` and +is deliberately left alone: pnpm rewrites that range too, but no consumer of the +package ever resolves it. diff --git a/.changeset/prisma-skill-upgrade-replan.md b/.changeset/prisma-skill-upgrade-replan.md new file mode 100644 index 000000000..19d93f919 --- /dev/null +++ b/.changeset/prisma-skill-upgrade-replan.md @@ -0,0 +1,16 @@ +--- +'stash': patch +--- + +`skills/stash-prisma` now documents the re-plan step that follows an +`@cipherstash/stack-prisma` upgrade: `rm -rf migrations/cipherstash && npx +prisma-next migration plan`, why only `migration plan` vendors new migration +packages, and the exact `db init` refusal a stale vendored directory produces on +a fresh database (`Operation cipherstash.upgrade-eql-v3-bundle-3.0.5 has class +"data" which is not allowed by policy.`). + +The package README already carried this; the skill did not — and the skill is +what ships inside the `stash` tarball and gets copied into a user's +`.claude/skills/`, so an agent driving the upgrade hit the refusal with no route +out of it. `packages/stack-prisma/test/v3/stale-vendored-space.test.ts` now pins +both files to the planner's real message so they cannot drift apart again. diff --git a/.changeset/protect-ffi-eql-bindings-in-tree.md b/.changeset/protect-ffi-eql-bindings-in-tree.md new file mode 100644 index 000000000..894015531 --- /dev/null +++ b/.changeset/protect-ffi-eql-bindings-in-tree.md @@ -0,0 +1,35 @@ +--- +'@cipherstash/protect-ffi': patch +--- + +Compile `eql-bindings` from this repository rather than from crates.io. + +The native binding pinned `eql-bindings = "=3.0.2"` from the registry. It now +resolves by path from `packages/eql/crates/eql-bindings`, which ships at 3.0.5 +alongside the `@cipherstash/eql` SQL bundle. + +**No behaviour change.** `eql-bindings` is the Rust half of EQL — it EMITS the +encrypted payloads that the SQL half STORES and queries — and its Rust source is +byte-identical across 3.0.2, 3.0.4 and 3.0.5 (`src/`, `bindings/` and `schema/` +compared directly). What 3.0.3 through 3.0.5 changed was SQL, carried on the +shared lockstep version number. So the payloads this binding produces are the +same bytes before and after; what moves is the version stamped on the crate +compiled into `index.node`, from 3.0.2 to 3.0.5. + +**Why it is worth a release anyway.** A registry pin let the two halves of EQL +drift apart silently. Nothing asserted they agreed: a mismatched pair compiles, +passes every suite, and fails in a database — because the failure is a payload +the installed SQL cannot read, which no unit test holds both sides of. Resolving +from the tree makes the skew unrepresentable: the emitter and the SQL are now +the same commit, and `pnpm run lint:eql-pins` fails any change that reintroduces +a registry pin on either. + +The flip was taken while it was a no-op deliberately. Waiting for the first +release where the two halves genuinely diverge would have turned a provenance +change into a behaviour change that had to be argued under credentialed test. + +Verified without credentials: `cargo build -p protect-ffi` clean, the crate test +suite green (310 passed) with `cargo fmt --check` clean, and a +`wasm32-unknown-unknown` build clean — the last of those being the target where a +cross-workspace path dependency would break first, since the EQL workspace never +otherwise builds for wasm32. diff --git a/.changeset/protect-ffi-jsonwebtoken-cve.md.deferred b/.changeset/protect-ffi-jsonwebtoken-cve.md similarity index 100% rename from .changeset/protect-ffi-jsonwebtoken-cve.md.deferred rename to .changeset/protect-ffi-jsonwebtoken-cve.md diff --git a/.changeset/skills-eql-source-and-index-cascade.md b/.changeset/skills-eql-source-and-index-cascade.md new file mode 100644 index 000000000..6b7c75930 --- /dev/null +++ b/.changeset/skills-eql-source-and-index-cascade.md @@ -0,0 +1,37 @@ +--- +'stash': patch +--- + +Correct two things the bundled agent skills were telling customers wrongly +about EQL. + +**`skills/stash-postgres` pointed at the wrong repository.** EQL's source now +lives in `cipherstash/stack` under `packages/eql/`, and that is where operator +gaps and domain-level bugs are filed; only *publishing* still happens from +`cipherstash/encrypt-query-language`, which the skill continues to say. The +skill also cited "the EQL skill" as a source of truth that "ships from +`encrypt-query-language` alongside the bundle" — no such skill ships from +either repository, so the reference is gone and the remaining three sources +(the generated types, the install SQL, and `SELECT eql_v3.version()`) are +renumbered. + +**And it claimed the CLI pins an exact `@cipherstash/eql` version, "so a +database is only ever on one bundle."** Neither half holds: the CLI depends on +the workspace package rather than a pinned literal, and a database is on +whatever bundle was last applied to it — the Prisma Next adapter installs and +upgrades the bundle through its own migrations without involving the CLI at +all. Replaced with the guarantee that does hold: one `stash` release carries +one resolved bundle, and the database is the authority on which bundle it has. + +**`skills/stash-prisma` hands out the functional-index recipe without saying an +EQL upgrade destroys it.** Installing a bundle begins with `DROP SCHEMA IF +EXISTS eql_v3 CASCADE`, which cascade-drops every index over an `eql_v3.*` +extractor — the PSL expression indexes Prisma Next 0.17 introduced and any +`rawSql` index DDL alike; queries keep working and silently sequential-scan. +Because an applied migration is never replayed, recovery is a NEW one: a PSL +expression index has to change its `name:` (the physical name carries a content +hash of the expression, so re-declaring the same one plans no work), and a +`rawSql` recovery op needs a new `id`. Said where the recipe is given, pointing +at `stash-indexing` for the mechanism and at +[cipherstash/stack#918](https://github.com/cipherstash/stack/issues/918) for +capturing and restoring them automatically. diff --git a/.changeset/stack-prisma-upgrade-from-1-0-0.md b/.changeset/stack-prisma-upgrade-from-1-0-0.md new file mode 100644 index 000000000..311ae91a5 --- /dev/null +++ b/.changeset/stack-prisma-upgrade-from-1-0-0.md @@ -0,0 +1,16 @@ +--- +'@cipherstash/stack-prisma': patch +--- + +Document the 1.0.0 → 3.0.5 upgrade in the package README: why +`migrations/cipherstash/` must be deleted and regenerated, what each Prisma Next +command does if it is not, and the exact `db init` refusal +(`Operation cipherstash.upgrade-eql-v3-bundle-3.0.5 has class "data" which is +not allowed by policy.`) that a stale vendored directory produces on a fresh +database. + +The behaviour worth knowing regardless of version: only `prisma-next migration +plan` copies new migration packages into your repo. Running `migrate` or +`db init` after upgrading this package without planning first silently leaves +the database on the older EQL bundle — a stale vendored directory is internally +intact, so it passes every integrity check and nothing reports a problem. diff --git a/.changeset/supabase-skill-eql-305.md b/.changeset/supabase-skill-eql-305.md new file mode 100644 index 000000000..8f65699f7 --- /dev/null +++ b/.changeset/supabase-skill-eql-305.md @@ -0,0 +1,11 @@ +--- +'stash': patch +--- + +Update the bundled agent skills for eql-3.0.5. `skills/stash-supabase` +re-states the PostgREST query-domain limitations against 3.0.5 (unchanged in +substance — the typed `eql_v3.query_*` operand requirement still stands), and +`skills/stash-postgres` drops one of the two places it claimed the CLI pins +`@cipherstash/eql` to an exact version — a claim that stopped being true when +EQL moved in-tree. The second copy goes in the same release, with the rest of +that skill's EQL source and issue pointers. diff --git a/.github/actionlint.yaml b/.github/actionlint.yaml index 21fa08b11..633dcb406 100644 --- a/.github/actionlint.yaml +++ b/.github/actionlint.yaml @@ -10,3 +10,9 @@ self-hosted-runner: labels: - blacksmith-4vcpu-ubuntu-2404 + # Every job in the imported EQL suite. Sixteen of them, and the size is + # deliberate upstream: the sharded SQLx matrix compiles the full Rust + # dependency tree, so the runner is chosen for the compile rather than for + # the tests. Kept as its own label rather than renamed to the 4vcpu one — + # rehoming those jobs is a cost decision, not a lint fix. + - blacksmith-16vcpu-ubuntu-2204 diff --git a/.github/actions/build-ffi-binding/action.yml b/.github/actions/build-ffi-binding/action.yml index ea5bc72e8..bee2c7228 100644 --- a/.github/actions/build-ffi-binding/action.yml +++ b/.github/actions/build-ffi-binding/action.yml @@ -54,13 +54,41 @@ runs: # package.json, and mise.toml pins the toolchain that compiles it. # Edit either and the Rust hash is unchanged, so the cache hit skips # the build step below and the job proceeds on a stale index.node. + # + # The last two are NOT in this package. `crates/protect-ffi/Cargo.toml` + # carries `eql-bindings = { path = "../../../eql/crates/eql-bindings" }` + # — an in-tree path dependency that compiles into `index.node` and into + # the wasm build below. A path dep has no registry checksum, so a + # source-only edit there moves NOTHING this key would otherwise hash: + # not `crates/**`, not either manifest, and not `Cargo.lock`, which + # records the crate by name and version and only moves when the version + # does. Every glob hashes identically, the restore hits, the build below + # is skipped, and every credentialed job runs a stale binding. (This is + # observed, not theoretical: the crate went 3.0.4 -> 4.0.0 -> 3.0.5 + # across three commits and neither key here changed by a byte.) + # + # `crates/**` rather than the one crate, deliberately. `eql-bindings` + # has no path deps of its own TODAY, so the narrower glob would be exact + # — and would silently stop being exact the moment it gains one on a + # sibling in that workspace, which is the same hole one level down. The + # cost of the wider glob is a spurious miss when an unrelated EQL crate + # changes; the cost of the narrower one is a stale binary. Guarded by + # `scripts/__tests__/ffi-binding-action.test.mjs`, which discovers the + # path deps from the manifest rather than trusting this comment. + # + # The EQL workspace root manifest comes with it: cargo reads it for the + # crate's workspace context, so a `.workspace = true` added to + # eql-bindings (three of its four siblings already inherit `[lints]` + # that way) would make it a compile input with no other trace. key: >- ffi-native-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('packages/protect-ffi/crates/**', 'packages/protect-ffi/Cargo.toml', 'packages/protect-ffi/Cargo.lock', 'packages/protect-ffi/package.json', - 'packages/protect-ffi/mise.toml') }} + 'packages/protect-ffi/mise.toml', + 'packages/eql/crates/**', + 'packages/eql/Cargo.toml') }} # Always, hit or miss: `lib/` is `tsc` over `src/`, which is seconds, and # it tracks a different input set than the cached binary. @@ -136,11 +164,20 @@ runs: # under src/ are the eql-v3-types declarations, which that tsconfig # never reads. ffi-binding-action.test.mjs walks the tsconfig's imports # and fails if this stops being the complete input set. + # + # `packages/eql/**` is here for the reason spelled out on the native key + # above, and it is NOT redundant with it: wasm-pack compiles the same + # `eql-bindings` path dependency for wasm32 into + # `protect_ffi_inline.js`, which is the bundle + # `@cipherstash/stack/wasm-inline` imports. Two keys, two archives, the + # same missing input. key: >- ffi-wasm-${{ runner.os }}-${{ hashFiles('packages/protect-ffi/crates/**', 'packages/protect-ffi/Cargo.toml', 'packages/protect-ffi/Cargo.lock', + 'packages/eql/crates/**', + 'packages/eql/Cargo.toml', 'packages/protect-ffi/dist/wasm/*.d.ts', 'packages/protect-ffi/src/errors.ts', 'packages/protect-ffi/package.json', diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 57a53d23c..a4bdb9370 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -62,7 +62,7 @@ updates: update-types: - version-update:semver-major - # ── Cargo (packages/protect-ffi — the only Rust in the repo) ──── + # ── Cargo (packages/protect-ffi — one of two Rust workspaces) ─── # Absorbing protect-ffi brought a 494-crate Cargo.lock in-tree. osv-scanner # already sees it — `--recursive ./` walks the tree and extracts every # lockfile it recognises — so known advisories were visible from day one, but @@ -149,6 +149,69 @@ updates: update-types: - version-update:semver-major + # ── Cargo (packages/eql — the second Rust workspace) ─────────── + # The EQL subtree brought a second Cargo workspace: five members + # (crates/eql-{domains,codegen,tests-macros,bindings} plus tests/sqlx) and + # its own Cargo.lock. It needs its own entry because Dependabot's cargo + # `directory:` is a single workspace root, not a glob — the protect-ffi entry + # above cannot reach it. The lockfile-to-ecosystem check in + # e2e/tests/supply-chain.e2e.test.ts asserts coverage per ECOSYSTEM, so it + # was already satisfied by that entry and would NOT have caught this gap; + # the symptom would simply have been that no PR ever arrived. + - package-ecosystem: cargo + # The workspace root, where Cargo.toml and Cargo.lock live. Not `/`, and + # not a crate directory — same trap as the protect-ffi entry documents. + directory: /packages/eql + # Monthly, matching protect-ffi and for the same reason: validating a + # Cargo.lock bump here means the sharded SQLx matrix in test-eql.yml, which + # compiles the full dependency tree and runs against live Postgres with + # CipherStash credentials. Security fixes are unaffected — those are driven + # by alerts, not by `schedule`. + # + # No `day:`, for the reason recorded above: it applies to `interval: weekly` + # and would be configuration that does nothing. + schedule: + interval: monthly + cooldown: + default-days: 7 + open-pull-requests-limit: 3 + labels: + - dependencies + - supply-chain + commit-message: + prefix: "chore" + include: scope + groups: + cargo-minor-patch: + patterns: + - "*" + update-types: + - minor + - patch + ignore: + # `cipherstash-client = "=0.42.0"` in tests/sqlx/Cargo.toml — the SAME + # exact pin, at the SAME version, as packages/protect-ffi. That is not a + # coincidence and it is the reason the subtree was imported: the two now + # share one release train, and a Dependabot PR that moved one workspace + # and not the other would reintroduce precisely the skew the absorption + # removed. Bump both, manually, in step with the npm catalog. + # + # Same caveat as above: `ignore` suppresses security PRs too. osv-scanner + # is the compensating control and already reaches this lockfile + # (`--recursive ./`). + - dependency-name: "cipherstash-client" + - dependency-name: "cts-common" + - dependency-name: "stack-auth" + - dependency-name: "stack-profile" + # Published from this workspace by release-plz. A Dependabot PR proposing + # a registry version for a crate we release here would fight the lockstep + # version hook (scripts/sync-lockstep-versions.mjs). + - dependency-name: "eql-bindings" + # Major bumps are reviewed and applied manually, not by Dependabot. + - dependency-name: "*" + update-types: + - version-update:semver-major + # ── GitHub Actions ───────────────────────────────────────────── - package-ecosystem: github-actions directory: / diff --git a/.github/workflows/bench-eql.yml b/.github/workflows/bench-eql.yml new file mode 100644 index 000000000..e6818f45f --- /dev/null +++ b/.github/workflows/bench-eql.yml @@ -0,0 +1,174 @@ +name: "Bench EQL" + +# Ported from cipherstash/encrypt-query-language with the subtree. See +# `.github/workflows/test-eql.yml` for the full account of what the move costs; +# this file needed the same four path rewrites plus two changes of its own, +# noted at the steps that carry them. +# +# Runs the slow benchmark / regression / scale SQLx tests gated behind the +# `bench` cargo feature. Not on pull requests — those use the fast `test-eql` +# workflow. +# Triggers: +# - push to main (catches regressions before release) +# - nightly schedule (additional smoke) +# - manual workflow_dispatch (PR triage) +on: + push: + branches: + - main + # Repo-root relative, so every entry gained the subtree prefix. Unprefixed, + # `src/**` and `tests/**` match nothing under this repository's root and the + # bench would simply stop running on pushes — silently, since a workflow + # that never triggers reports nothing at all. + # + # IDENTICAL to `test-eql.yml`'s list, modulo each file naming itself, and + # held that way by `scripts/__tests__/eql-workflow-filters.test.mjs`. The + # bench is a strict subset of that suite's inputs, so a bespoke narrower + # list here is possible and was what the port carried — and it had drifted + # three ways, each of them a change that alters what the bench measures + # while never starting it: + # + # * `src/**/*.sql` missed `src/v3/version.template`, which `tasks/build.sh` + # names in its own `#MISE sources` and `sed`s into `version.sql`. + # * `tests/sqlx/**/*` missed `tests/docker-compose.yml` — the file + # `postgres:up` actually runs, one level above the glob. + # * `mise.toml` was absent entirely, and it defines every `mise run` task + # below plus the toolchain that compiles them. + # + # Two lists that answer the same question are worth more kept equal than + # kept minimal: the cost of over-triggering is one nightly-budget bench run, + # and the cost of under-triggering is a regression that lands and is + # attributed to whatever merges next. + # + # The last four entries are the one place that reasoning is doing visible + # work. The bench runs `postgres:up` and `test:bench` and reads no + # documentation at all — they are inputs to `test-eql.yml`'s `docs-static` + # job, not to anything here, and the derivation in + # `scripts/__tests__/eql-workflow-filters.test.mjs` correctly derives none + # of them for this file. They are listed because the three copies are held + # equal, which is the whole point of the paragraph above: the cost is a + # bench run on a docs push, and the alternative is a bespoke narrower list + # of exactly the kind that had already drifted three ways. + paths: + - ".github/workflows/bench-eql.yml" + - ".github/actions/require-cs-secrets/**" + - "packages/eql/src/**" + - "packages/eql/tests/**" + - "packages/eql/tasks/**" + - "packages/eql/crates/**" + - "packages/eql/packages/**" + - "packages/eql/Cargo.toml" + - "packages/eql/Cargo.lock" + - "packages/eql/mise.toml" + - "packages/eql/docs/**" + - "packages/eql/docker/**" + - "packages/eql/README.md" + - "packages/eql/SUPABASE.md" + + schedule: + # 02:00 UTC daily + - cron: "0 2 * * *" + + workflow_dispatch: + +env: + FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true" + MISE_VERBOSE: "1" + +permissions: + contents: read + +defaults: + run: + shell: bash {0} + # Every `run:` here is written against the EQL root. + working-directory: packages/eql + +jobs: + bench: + name: "Bench EQL (Postgres 17)" + runs-on: blacksmith-16vcpu-ubuntu-2204 + timeout-minutes: 60 + + env: + POSTGRES_VERSION: "17" + + steps: + # SHA-pinned, where upstream used floating major tags. The rest of the + # imported suite already pins by SHA; a mutable tag on a job that holds + # live CipherStash credentials means the code running there can change + # without a commit here. Same three pins as test-eql.yml — keep them in + # step. (checkout moves v4 -> v6 with this, matching that file.) + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + persist-credentials: false + + # Fast pre-flight: fail in seconds if a secret was rotated or cleared, + # before the rust-cache restore and a bench run budgeted at 60 minutes. + # This is a scheduled job, so nobody is watching it start — the difference + # between failing here and failing in `test:bench` is a legible nightly + # failure versus "Auth strategy error: Not authenticated" an hour in. + - uses: ./.github/actions/require-cs-secrets + with: + workspace-crn: ${{ vars.CS_WORKSPACE_CRN }} + client-id: ${{ vars.CS_CLIENT_ID }} + client-key: ${{ secrets.CS_CLIENT_KEY }} + client-access-key: ${{ secrets.CS_CLIENT_ACCESS_KEY }} + + - uses: jdx/mise-action@1648a7812b9aeae629881980618f079932869151 # v4 + with: + version: 2026.4.0 + install: true + cache: true + # mise reads config from cwd and its PARENTS, so an action running at + # the repo root never sees packages/eql/mise.toml. + working_directory: packages/eql + + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 + with: + workspaces: packages/eql + shared-key: sqlx-tests + # `build-archive` in test-eql.yml is the sole SAVER of this shared + # key — it compiles the full heavy dependency tree, so its `target/` + # is the one worth keeping. Every other consumer restores only, or a + # fast-finishing light job wins the save race and overwrites the entry + # with a deps-less `target/`. This job is the worst possible winner of + # that race: it is bench-gated and nightly, so a stale entry it saved + # would go unnoticed until someone wondered why the PR matrix got + # slow. Asserted by scripts/__tests__/eql-suite-ci.test.mjs. + save-if: false + + - name: Setup database + run: | + mise run postgres:up postgres-${POSTGRES_VERSION} --extra-args "--detach --wait" + + - name: Run bench tests + # CS_* scoped to THIS step only (least privilege): test:bench -> test:sqlx:prep + # -> fixture:generate:all encrypts via cipherstash-client and needs BOTH a + # ZeroKMS auth credential (CS_CLIENT_ACCESS_KEY + CS_WORKSPACE_CRN) AND a client + # key (CS_CLIENT_ID + CS_CLIENT_KEY); without them it fails "Auth strategy error: + # Not authenticated". Kept off job scope so checkout/mise/rust-cache actions + # never see them. + # + # CS_WORKSPACE_CRN and CS_CLIENT_ID read from `vars.`, not `secrets.` — + # this repo keeps the two non-sensitive ones there. Reading them from + # `secrets.` yields the empty string rather than an error. + env: + CS_CLIENT_ACCESS_KEY: ${{ secrets.CS_CLIENT_ACCESS_KEY }} + CS_WORKSPACE_CRN: ${{ vars.CS_WORKSPACE_CRN }} + CS_CLIENT_ID: ${{ vars.CS_CLIENT_ID }} + CS_CLIENT_KEY: ${{ secrets.CS_CLIENT_KEY }} + # `set -euo pipefail` because the workflow-wide `shell: bash {0}` drops + # GitHub's implicit `-eo pipefail`: without it a failing `rustup + # component add` is discarded and only `test:bench` decides the step, + # which then fails several minutes later for a reason that reads as a + # bench regression. Enforced by + # `scripts/__tests__/workflow-run-fail-fast.test.mjs`. + # + # Assignment on its own rather than `export x=$(…)`: errexit ignores + # the substitution's status when the line starts with a command word. + run: | + set -euo pipefail + active_rust_toolchain=$(rustup show active-toolchain | cut -d' ' -f1) + rustup component add --toolchain "${active_rust_toolchain}" rustfmt clippy + mise run --output prefix test:bench --postgres ${POSTGRES_VERSION} diff --git a/.github/workflows/fta-v3.yml b/.github/workflows/fta-v3.yml index fc2dff955..827612d73 100644 --- a/.github/workflows/fta-v3.yml +++ b/.github/workflows/fta-v3.yml @@ -82,10 +82,10 @@ jobs: # reflow trips this step without a real complexity increase, re-measure # (`npx fta src --format table`) before assuming the code got worse. - name: Analyze stack (eql/v3) complexity - run: pnpm --filter @cipherstash/stack run analyze:complexity + run: pnpm exec turbo run analyze:complexity --filter @cipherstash/stack - name: Analyze stack-drizzle complexity - run: pnpm --filter @cipherstash/stack-drizzle run analyze:complexity + run: pnpm exec turbo run analyze:complexity --filter @cipherstash/stack-drizzle - name: Analyze stack-supabase complexity - run: pnpm --filter @cipherstash/stack-supabase run analyze:complexity + run: pnpm exec turbo run analyze:complexity --filter @cipherstash/stack-supabase diff --git a/.github/workflows/integration-drizzle.yml b/.github/workflows/integration-drizzle.yml index 107b2b7bd..d931ebc89 100644 --- a/.github/workflows/integration-drizzle.yml +++ b/.github/workflows/integration-drizzle.yml @@ -81,11 +81,22 @@ on: # emits `dist/wasm/protect_ffi_inline.js` — the bundle the # `integration/wasm/**` suites selected below actually load. Both files # are in the WASM cache key of `.github/actions/build-ffi-binding`, so an - # edit to either misses that cache and REBUILDS the bundle; without these - # two entries it gets rebuilt and the only suite that loads it never runs. + # edit to either misses that cache and REBUILDS the bundle. Without these + # two entries the workflow does not trigger at all, so neither the rebuild + # nor the only suite that loads it ever runs. # Pinned by scripts/__tests__/wasm-build-inputs-paths-filter.test.mjs. - 'packages/protect-ffi/tsconfig.wasm-errors.json' - 'packages/protect-ffi/scripts/inline-wasm.mjs' + # Out of that package, and compiled into the same two artifacts: the + # cdylib crate carries + # `eql-bindings = { path = "../../../eql/crates/eql-bindings" }`, so the + # EQL wire types are linked into `index.node` and into + # `protect_ffi_inline.js`. Both cache keys in + # `.github/actions/build-ffi-binding` hash them, so an edit here misses + # that cache and rebuilds the bundle the `integration/wasm/**` suites + # load. Same argument as the two entries above, same guard. + - 'packages/eql/crates/**' + - 'packages/eql/Cargo.toml' pull_request: branches: ['**'] # Repeated verbatim: GitHub Actions does not support YAML anchors/aliases. @@ -155,11 +166,22 @@ on: # emits `dist/wasm/protect_ffi_inline.js` — the bundle the # `integration/wasm/**` suites selected below actually load. Both files # are in the WASM cache key of `.github/actions/build-ffi-binding`, so an - # edit to either misses that cache and REBUILDS the bundle; without these - # two entries it gets rebuilt and the only suite that loads it never runs. + # edit to either misses that cache and REBUILDS the bundle. Without these + # two entries the workflow does not trigger at all, so neither the rebuild + # nor the only suite that loads it ever runs. # Pinned by scripts/__tests__/wasm-build-inputs-paths-filter.test.mjs. - 'packages/protect-ffi/tsconfig.wasm-errors.json' - 'packages/protect-ffi/scripts/inline-wasm.mjs' + # Out of that package, and compiled into the same two artifacts: the + # cdylib crate carries + # `eql-bindings = { path = "../../../eql/crates/eql-bindings" }`, so the + # EQL wire types are linked into `index.node` and into + # `protect_ffi_inline.js`. Both cache keys in + # `.github/actions/build-ffi-binding` hash them, so an edit here misses + # that cache and rebuilds the bundle the `integration/wasm/**` suites + # load. Same argument as the two entries above, same guard. + - 'packages/eql/crates/**' + - 'packages/eql/Cargo.toml' jobs: integration: diff --git a/.github/workflows/integration-prisma-next.yml b/.github/workflows/integration-prisma-next.yml index 4af283142..9f94acda2 100644 --- a/.github/workflows/integration-prisma-next.yml +++ b/.github/workflows/integration-prisma-next.yml @@ -49,6 +49,18 @@ on: - 'packages/protect-ffi/Cargo.lock' - 'packages/protect-ffi/package.json' - 'packages/protect-ffi/mise.toml' + # Out of that package, and a compile input all the same: the cdylib + # crate carries + # `eql-bindings = { path = "../../../eql/crates/eql-bindings" }`, so the + # EQL wire types are linked into the `index.node` this job encrypts + # through. The native cache key in `.github/actions/build-ffi-binding` + # hashes both, so an edit here misses that cache and the binding gets + # rebuilt — correctly. Without these two entries the miss never happens, + # because the workflow does not trigger at all: a payload-encoding change + # under packages/eql/crates merges with this suite green by absence. + # Pinned by scripts/__tests__/wasm-build-inputs-paths-filter.test.mjs. + - 'packages/eql/crates/**' + - 'packages/eql/Cargo.toml' pull_request: branches: ['**'] # Repeated verbatim: GitHub Actions does not support YAML anchors/aliases. @@ -84,6 +96,18 @@ on: - 'packages/protect-ffi/Cargo.lock' - 'packages/protect-ffi/package.json' - 'packages/protect-ffi/mise.toml' + # Out of that package, and a compile input all the same: the cdylib + # crate carries + # `eql-bindings = { path = "../../../eql/crates/eql-bindings" }`, so the + # EQL wire types are linked into the `index.node` this job encrypts + # through. The native cache key in `.github/actions/build-ffi-binding` + # hashes both, so an edit here misses that cache and the binding gets + # rebuilt — correctly. Without these two entries the miss never happens, + # because the workflow does not trigger at all: a payload-encoding change + # under packages/eql/crates merges with this suite green by absence. + # Pinned by scripts/__tests__/wasm-build-inputs-paths-filter.test.mjs. + - 'packages/eql/crates/**' + - 'packages/eql/Cargo.toml' jobs: integration: diff --git a/.github/workflows/integration-protect-ffi.yml b/.github/workflows/integration-protect-ffi.yml index fcd48f90b..f95d4c743 100644 --- a/.github/workflows/integration-protect-ffi.yml +++ b/.github/workflows/integration-protect-ffi.yml @@ -61,6 +61,16 @@ on: # toolchain pins, and the PG* connection env. - 'packages/protect-ffi/package.json' - 'packages/protect-ffi/mise.toml' + # Out of this package, and a real compile input: the cdylib crate carries + # `eql-bindings = { path = "../../../eql/crates/eql-bindings" }`, so the + # EQL wire types are linked into `index.node` AND into the wasm32 build + # this suite's `wasm-round-trip` test loads. Both cache keys in + # `.github/actions/build-ffi-binding` hash these, so an edit misses the + # cache and rebuilds the binding — without these entries it would be + # rebuilt and no suite that loads it would start. Pinned by + # scripts/__tests__/wasm-build-inputs-paths-filter.test.mjs. + - 'packages/eql/crates/**' + - 'packages/eql/Cargo.toml' - '.github/workflows/integration-protect-ffi.yml' - '.github/actions/build-ffi-binding/**' - '.github/actions/require-cs-secrets/**' @@ -79,6 +89,16 @@ on: - 'packages/protect-ffi/tsconfig.wasm-errors.json' - 'packages/protect-ffi/package.json' - 'packages/protect-ffi/mise.toml' + # Out of this package, and a real compile input: the cdylib crate carries + # `eql-bindings = { path = "../../../eql/crates/eql-bindings" }`, so the + # EQL wire types are linked into `index.node` AND into the wasm32 build + # this suite's `wasm-round-trip` test loads. Both cache keys in + # `.github/actions/build-ffi-binding` hash these, so an edit misses the + # cache and rebuilds the binding — without these entries it would be + # rebuilt and no suite that loads it would start. Pinned by + # scripts/__tests__/wasm-build-inputs-paths-filter.test.mjs. + - 'packages/eql/crates/**' + - 'packages/eql/Cargo.toml' - '.github/workflows/integration-protect-ffi.yml' - '.github/actions/build-ffi-binding/**' - '.github/actions/require-cs-secrets/**' diff --git a/.github/workflows/integration-supabase.yml b/.github/workflows/integration-supabase.yml index 9084dcbf8..61f1f4ad6 100644 --- a/.github/workflows/integration-supabase.yml +++ b/.github/workflows/integration-supabase.yml @@ -67,6 +67,18 @@ on: - 'packages/protect-ffi/Cargo.lock' - 'packages/protect-ffi/package.json' - 'packages/protect-ffi/mise.toml' + # Out of that package, and a compile input all the same: the cdylib + # crate carries + # `eql-bindings = { path = "../../../eql/crates/eql-bindings" }`, so the + # EQL wire types are linked into the `index.node` this job encrypts + # through. The native cache key in `.github/actions/build-ffi-binding` + # hashes both, so an edit here misses that cache and the binding gets + # rebuilt — correctly. Without these two entries the miss never happens, + # because the workflow does not trigger at all: a payload-encoding change + # under packages/eql/crates merges with this suite green by absence. + # Pinned by scripts/__tests__/wasm-build-inputs-paths-filter.test.mjs. + - 'packages/eql/crates/**' + - 'packages/eql/Cargo.toml' pull_request: branches: ['**'] paths: @@ -125,6 +137,18 @@ on: - 'packages/protect-ffi/Cargo.lock' - 'packages/protect-ffi/package.json' - 'packages/protect-ffi/mise.toml' + # Out of that package, and a compile input all the same: the cdylib + # crate carries + # `eql-bindings = { path = "../../../eql/crates/eql-bindings" }`, so the + # EQL wire types are linked into the `index.node` this job encrypts + # through. The native cache key in `.github/actions/build-ffi-binding` + # hashes both, so an edit here misses that cache and the binding gets + # rebuilt — correctly. Without these two entries the miss never happens, + # because the workflow does not trigger at all: a payload-encoding change + # under packages/eql/crates merges with this suite green by absence. + # Pinned by scripts/__tests__/wasm-build-inputs-paths-filter.test.mjs. + - 'packages/eql/crates/**' + - 'packages/eql/Cargo.toml' jobs: integration: diff --git a/.github/workflows/macro-expand-eql.yml b/.github/workflows/macro-expand-eql.yml new file mode 100644 index 000000000..fe213b526 --- /dev/null +++ b/.github/workflows/macro-expand-eql.yml @@ -0,0 +1,127 @@ +name: "Macro expand EQL" + +# Ported from cipherstash/encrypt-query-language with the subtree. See +# `.github/workflows/test-eql.yml` for what the move costs (three path rewrites +# here) and `scripts/__tests__/eql-suite-ci.test.mjs` for why a workflow left in +# `packages/eql/.github/` runs nowhere. +# +# Regenerates the matrix `cargo expand` snapshots (one per reachable +# `scalar_matrix!` arm: integer = [eq, ord], text = [eq, ord, search], boolean = +# [storage]) and fails if any has drifted from its committed copy. This is a +# body-level fidelity backstop for the matrix macros — the name-inventory +# snapshot (test-eql.yml `matrix-coverage` job) catches add/remove of whole +# arms; this catches changes *inside* the generated bodies. +# +# Non-blocking by design: it is NOT a required PR check. `cargo expand` needs a +# nightly toolchain, so it is isolated off the PR path. +# - nightly schedule (the backstop that flags a forgotten local regen) +# - manual workflow_dispatch +# +# GAP (intended): there is no `pull_request` trigger, so a change that only +# touches macro *bodies* (no arm add/remove) can merge without ever running +# here and will first surface as a red nightly run afterwards. Accept this — the +# expand lane needs nightly and stays off the PR critical path by design. +# +# The pinned nightly date lives in ONE place: the `cargo +nightly-...` invocation +# in the `test:matrix:expand` mise task. The install step below DERIVES the date +# from mise.toml (grep), so there is nothing to keep in lockstep — bump it once in +# mise.toml. The snapshot then only moves when the macro moves, not when nightly +# reformats its expansion. +on: + schedule: + # 03:00 UTC daily + - cron: "0 3 * * *" + + workflow_dispatch: + +env: + FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true" + MISE_VERBOSE: "1" + +defaults: + run: + shell: bash {0} + # Every `run:` here is written against the EQL root: `mise.toml` for the + # nightly grep, and `tests/sqlx/snapshots` for the drift diff. + working-directory: packages/eql + +permissions: + contents: read + +jobs: + macro-expand: + name: "Macro expand drift (nightly)" + runs-on: blacksmith-16vcpu-ubuntu-2204 + timeout-minutes: 30 + + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + persist-credentials: false + + - uses: jdx/mise-action@1648a7812b9aeae629881980618f079932869151 # v4 + with: + version: 2026.4.0 + install: true + cache: true + working_directory: packages/eql + + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 + with: + # The workspace ROOT, not the `tests/sqlx` member this job expands. + # `workspaces:` tells rust-cache where to find the `Cargo.lock` it + # hashes into the key and the `target/` it archives, and both are at + # `packages/eql` — `tests/sqlx` is a `members` entry with neither. So + # the ported value hashed no lockfile and saved an empty directory, + # while reporting success; the only symptom was a slower nightly. + # Every sibling step in test-eql.yml and bench-eql.yml names the root. + workspaces: packages/eql + shared-key: sqlx-tests + # `build-archive` in test-eql.yml is the sole SAVER of this shared + # key — it compiles the full heavy dependency tree, so its `target/` + # is the one worth keeping. Every other consumer restores only, or a + # fast-finishing light job wins the save race and overwrites the entry + # with a deps-less `target/`. Asserted by + # scripts/__tests__/eql-suite-ci.test.mjs. + save-if: false + + # Derive the pinned nightly date from mise.toml (single source of truth — + # the `cargo +nightly-...` invocation in the `test:matrix:expand` task) so + # there is nothing to bump in lockstep here. cargo-expand is likewise pinned + # once in mise.toml's [tools] (`cargo:cargo-expand`) and installed by the + # mise-action step above, so its version is single-sourced too — no + # hardcoded version lives in this workflow. It drives the rustfmt pass, so + # an unpinned version could drift the snapshot even with a frozen macro + + # nightly. The snapshot then only moves when the macro moves. + - name: Install pinned nightly toolchain + # `set -eu`, deliberately WITHOUT `pipefail`: the workflow-wide + # `shell: bash {0}` drops GitHub's implicit `-eo pipefail`, so errexit + # has to be asked for (see + # `scripts/__tests__/workflow-run-fail-fast.test.mjs`) — but `head -1` + # closes the pipe as soon as it has its line, so under `pipefail` a + # grep that still had output to write reports 141 and fails the step on + # a successful match. Same SIGPIPE hazard as + # `scripts/__tests__/workflow-grep-q-pipelines.test.mjs`. The `test -n` + # below already covers the case this pipeline can genuinely fail in. + run: | + set -eu + NIGHTLY=$(grep -oE 'nightly-[0-9]{4}-[0-9]{2}-[0-9]{2}' mise.toml | head -1) + test -n "$NIGHTLY" || { echo "could not find pinned nightly in mise.toml"; exit 1; } + rustup toolchain install "$NIGHTLY" --profile minimal --component rustfmt + + - name: Regenerate and verify the matrix expansion snapshots + # `set -euo pipefail` is load-bearing here, more than anywhere else in + # this file. Without it the workflow-wide `shell: bash {0}` discards the + # regeneration's exit code and the `git diff` below decides the step — + # so when `test:matrix:expand` BREAKS, the snapshots are untouched, the + # diff is clean, and this job reports success having verified that a + # build which never ran produced no drift. Enforced by + # `scripts/__tests__/workflow-run-fail-fast.test.mjs`. + run: | + set -euo pipefail + mise run test:matrix:expand + git diff --exit-code -- \ + tests/sqlx/snapshots/integer_expanded.rs \ + tests/sqlx/snapshots/text_expanded.rs \ + tests/sqlx/snapshots/boolean_expanded.rs \ + || { echo "Expansion snapshot stale — run 'mise run test:matrix:expand' (needs the pinned nightly) and commit."; exit 1; } diff --git a/.github/workflows/prisma-example-readme-e2e.yml b/.github/workflows/prisma-example-readme-e2e.yml index 132edf09c..09114477b 100644 --- a/.github/workflows/prisma-example-readme-e2e.yml +++ b/.github/workflows/prisma-example-readme-e2e.yml @@ -32,6 +32,18 @@ on: - 'packages/protect-ffi/Cargo.lock' - 'packages/protect-ffi/package.json' - 'packages/protect-ffi/mise.toml' + # Out of that package, and a compile input all the same: the cdylib + # crate carries + # `eql-bindings = { path = "../../../eql/crates/eql-bindings" }`, so the + # EQL wire types are linked into the `index.node` this job encrypts + # through. The native cache key in `.github/actions/build-ffi-binding` + # hashes both, so an edit here misses that cache and the binding gets + # rebuilt — correctly. Without these two entries the miss never happens, + # because the workflow does not trigger at all: a payload-encoding change + # under packages/eql/crates merges with this walkthrough green by absence. + # Pinned by scripts/__tests__/wasm-build-inputs-paths-filter.test.mjs. + - 'packages/eql/crates/**' + - 'packages/eql/Cargo.toml' pull_request: branches: - '**' @@ -52,6 +64,18 @@ on: - 'packages/protect-ffi/Cargo.lock' - 'packages/protect-ffi/package.json' - 'packages/protect-ffi/mise.toml' + # Out of that package, and a compile input all the same: the cdylib + # crate carries + # `eql-bindings = { path = "../../../eql/crates/eql-bindings" }`, so the + # EQL wire types are linked into the `index.node` this job encrypts + # through. The native cache key in `.github/actions/build-ffi-binding` + # hashes both, so an edit here misses that cache and the binding gets + # rebuilt — correctly. Without these two entries the miss never happens, + # because the workflow does not trigger at all: a payload-encoding change + # under packages/eql/crates merges with this walkthrough green by absence. + # Pinned by scripts/__tests__/wasm-build-inputs-paths-filter.test.mjs. + - 'packages/eql/crates/**' + - 'packages/eql/Cargo.toml' jobs: walkthrough: diff --git a/.github/workflows/prisma-next-e2e.yml b/.github/workflows/prisma-next-e2e.yml index ae61fda56..d0e8b3cbe 100644 --- a/.github/workflows/prisma-next-e2e.yml +++ b/.github/workflows/prisma-next-e2e.yml @@ -31,6 +31,18 @@ on: - 'packages/protect-ffi/Cargo.lock' - 'packages/protect-ffi/package.json' - 'packages/protect-ffi/mise.toml' + # Out of that package, and a compile input all the same: the cdylib + # crate carries + # `eql-bindings = { path = "../../../eql/crates/eql-bindings" }`, so the + # EQL wire types are linked into the `index.node` this job encrypts + # through. The native cache key in `.github/actions/build-ffi-binding` + # hashes both, so an edit here misses that cache and the binding gets + # rebuilt — correctly. Without these two entries the miss never happens, + # because the workflow does not trigger at all: a payload-encoding change + # under packages/eql/crates merges with this suite green by absence. + # Pinned by scripts/__tests__/wasm-build-inputs-paths-filter.test.mjs. + - 'packages/eql/crates/**' + - 'packages/eql/Cargo.toml' pull_request: branches: - '**' @@ -52,6 +64,18 @@ on: - 'packages/protect-ffi/Cargo.lock' - 'packages/protect-ffi/package.json' - 'packages/protect-ffi/mise.toml' + # Out of that package, and a compile input all the same: the cdylib + # crate carries + # `eql-bindings = { path = "../../../eql/crates/eql-bindings" }`, so the + # EQL wire types are linked into the `index.node` this job encrypts + # through. The native cache key in `.github/actions/build-ffi-binding` + # hashes both, so an edit here misses that cache and the binding gets + # rebuilt — correctly. Without these two entries the miss never happens, + # because the workflow does not trigger at all: a payload-encoding change + # under packages/eql/crates merges with this suite green by absence. + # Pinned by scripts/__tests__/wasm-build-inputs-paths-filter.test.mjs. + - 'packages/eql/crates/**' + - 'packages/eql/Cargo.toml' jobs: e2e: @@ -149,7 +173,7 @@ jobs: run: pnpm exec turbo run build --filter @cipherstash/stack-prisma - name: Emit example contract - run: pnpm --filter @cipherstash/prisma-example emit + run: pnpm exec turbo run emit --filter @cipherstash/prisma-example - name: Start E2E Postgres container working-directory: examples/prisma diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 15708561f..6b0c0abec 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -278,11 +278,81 @@ jobs: - name: Install dependencies run: pnpm install --frozen-lockfile + # REQUIRED BY `version:` BELOW, and nothing in this file says so without + # this comment — which is why there is also a test. `pnpm run version` is + # `changeset version && node scripts/sync-lockstep-versions.mjs`, and that + # script ends in `execFileSync('mise', ['run', + # 'release:prepare_bindings_assets', …])`, which reaches + # `packages/eql/tasks/build.sh` and two `cargo run -p eql-codegen` calls. + # mise is NOT preinstalled on GitHub's ubuntu images, so without this step + # the hook dies with ENOENT — AFTER `changeset version` has already + # rewritten every manifest and changelog, in a job holding + # `contents: write`. Asserted by + # scripts/__tests__/workflow-mise-setup.test.mjs. + # + # It only fires on the branch where `.changeset/` is non-empty (the + # Version Packages branch), which is why a publish rehearsal never + # exercised it. + # + # `working_directory: packages/eql` for the reason test-eql.yml records at + # length: mise reads config from the current directory and its PARENTS, so + # an action running at the repo root never sees `packages/eql/mise.toml`. + # It would install nothing and leave the config untrusted, and the first + # `mise run` fails with "Config files … are not trusted" — which reads as + # a toolchain problem rather than a path one. That file is also where the + # Rust toolchain comes from (`[tools] rust`), so this step is the cargo + # setup as well; there is deliberately no second one. + # + # `cache: false` IS NOT THE DEFAULT — jdx/mise-action caches by default, + # and scripts/lint-no-workflow-caching.mjs forbids a GitHub Actions cache + # restore anywhere an artifact gets published. A poisoned entry here would + # execute in the job that holds the npm publishing credential. + # + # `add_shims_to_path: false` IS LOAD-BEARING, and it is the input a + # copy-paste from test-eql.yml would not carry. `packages/eql/mise.toml` + # pins `node = "22"` under `[tools]`, and mise's shim directory is + # PREPENDED to PATH for every later step — so with the default `true`, + # mise's own Node would shadow the one `actions/setup-node` installed, and + # `changeset publish` would shell out to that Node's bundled npm 10.x + # instead of the `npm@^11.5.1` installed above. OIDC trusted publishing + # requires >= 11.5.1 and fails with E404 below it, which is the exact + # failure the two comments above this step exist to prevent. `mise run` + # resolves its own toolchain internally, so nothing here needs the shims. + # + # `env: false` for a smaller version of the same argument: the default + # exports that file's `[env]` block — `DATABASE_URL`, `POSTGRES_PASSWORD` + # and friends, all pointed at a Postgres this job does not have — into + # GITHUB_ENV for every subsequent step. + # + # SHA-pinned, matching .github/actions/build-ffi-binding/action.yml: + # mise-action executes third-party code in the job that publishes, so a + # mutable `@v4` would let that code change with no commit here. The pin is + # the same v4 commit test-eql.yml uses. Dependabot moves the pin and the + # trailing comment together (.github/dependabot.yml covers github-actions). + - name: Install mise (the lockstep version hook shells out to it) + uses: jdx/mise-action@1648a7812b9aeae629881980618f079932869151 # v4 + with: + version: 2026.4.0 + install: true + cache: false + working_directory: packages/eql + add_shims_to_path: false + env: false + - name: Publish to npm id: changesets uses: changesets/action@v1.9.0 with: publish: pnpm run release + # LOAD-BEARING, and it fails OPEN if removed. Without `version:` the + # action runs its own built-in `changeset version` and never invokes + # the root `version` script — so `scripts/sync-lockstep-versions.mjs` + # would not run, npm would bump while + # packages/eql/crates/eql-bindings/Cargo.toml and the bundled SQL + # assets kept the old version, and the first symptom would be a + # published crate disagreeing with the SQL bundle it ships. + # Asserted by scripts/__tests__/release-version-hook.test.mjs. + version: pnpm run version commitMode: 'github-api' env: # No NPM_TOKEN — publishing authenticates via npm OIDC trusted diff --git a/.github/workflows/test-eql.yml b/.github/workflows/test-eql.yml new file mode 100644 index 000000000..4f9d65ba4 --- /dev/null +++ b/.github/workflows/test-eql.yml @@ -0,0 +1,912 @@ +name: "Test EQL" + +# The EQL suite, ported from cipherstash/encrypt-query-language when the subtree +# landed at `packages/eql`. It arrived under `packages/eql/.github/workflows/` — +# a directory GitHub never reads — so between the import and this file it ran +# NOWHERE. That is the same failure the protect-ffi absorption hit, and +# `scripts/__tests__/eql-suite-ci.test.mjs` is what stops it recurring. +# +# Four things changed in the port, all of them because the tree moved one level +# down. Each is silent when wrong: +# +# * `defaults.run.working-directory` — every `run:` here is written against +# the EQL root (`mise run …`, `release/*.sql`, `tests/sqlx/snapshots`). +# Without it they resolve against the monorepo root and fail with paths that +# look like typos. +# * `working_directory:` on each mise-action step — mise reads config from the +# current directory and its PARENTS, so an action running at the repo root +# never sees `packages/eql/mise.toml`. It would install nothing and leave the +# config untrusted, and the first `mise run` fails with "Config files … are +# not trusted", which reads as a toolchain problem rather than a path one. +# `defaults.run` does not reach a `uses:` step, so this is separate. +# * `workspaces:` on each rust-cache step — the Cargo workspace is nested now. +# * artifact `path:`/`path:` on upload/download — those are workspace-root +# relative and `defaults.run` does not touch them either. +# +# The credentials also changed shape. Upstream read all four from `secrets.`; +# this repo keeps the two non-sensitive ones in `vars.` (see +# `.github/actions/require-cs-secrets`), so `CS_WORKSPACE_CRN` and +# `CS_CLIENT_ID` are `vars.` here. Reading them from `secrets.` would not error +# — it yields the empty string, and the suite would fail at client construction +# in every shard instead of saying what was missing. +# +# NB: NO path filter on `pull_request`. A workflow skipped by a path filter +# leaves its required checks stuck Pending and blocks merge. Relevance is +# computed by the `changes` job and applied per-job via `if:` instead. `push` +# carries one because nothing waits on a push run, so a skipped one costs +# nothing — and the two lists are kept identical by +# `scripts/__tests__/eql-workflow-filters.test.mjs`. +# +# WHY `push` AND `schedule` EXIST. The port arrived with `merge_group` as the +# ONLY route to the full PG 14-17 matrix, and with no `push:` at all, +# justified by "under a required merge queue, push-to-main validation is +# redundant". Every clause of that premise is false in this repository, checked +# against the live API: there is no merge queue on `main` +# (`mergeQueue(branch:"main")` -> null), `main` is not protected +# (`branches/main/protection` -> 404), and the active ruleset carries no +# `required_status_checks` rule, so `ci-required` is required by nothing. +# `merge_group` is an event this repo never emits — so PG 14, 15 and 16 were +# tested NEVER, and nothing EQL-related ran after a merge. Both failures are +# invisible, because an event that is never delivered produces no runs to +# notice the absence of. Pinned by +# `scripts/__tests__/eql-matrix-triggers.test.mjs`, which refuses to count +# `merge_group` or `workflow_dispatch` as reachable. +# +# WHY THIS SHAPE, and not "full matrix on every push". The expensive axis is +# the fan-out, not the trigger: PG 14-17 x 2 shards is roughly double the +# runner-minutes of the PR lane. So the cost is bounded by the FILTER rather +# than by running a smaller matrix — a `push` run only starts when the merge +# actually touched EQL, which is a small fraction of merges to `main`, and when +# it does start the extra PG versions are the entire reason to run it at all. +# Attributing a PG14 break to the merge that caused it is worth much more than +# finding it detached at 04:00 the next morning. `schedule` then backstops the +# case the filter cannot see (a dependency or toolchain moving underneath a +# tree nobody touched), and both use the same full matrix. +# +# `pull_request` keeps the fast lane — PG17 x 4 shards, every PR, unfiltered, +# with relevance applied per job — because latency is what matters there and +# PG17 catches all but version-specific breakage. +on: + pull_request: {} + push: + branches: [main] + # Identical to the `relevant:` filter in the `changes` job below. GitHub + # Actions has no YAML anchors, so the list is written twice; the guard named + # above fails a one-sided edit. + # + # The last four entries are the documentation surface, and they are here + # because a job in THIS file reads them: `docs-static` runs + # `mise run test:docs_v3_grep`, which scans `README.md`, `SUPABASE.md`, + # `docker/README.md` and every tracked `docs/` markdown for the eql_v2 + # surface removed in 3.0.0. All four were missing from all three copies of + # this list, so a push to main touching only docs started no run of this + # workflow at all — `pull_request` was unaffected, because it applies no + # `paths:` filter and `docs-static` is deliberately not relevance-gated. + # Derived rather than remembered now: the second half of + # `scripts/__tests__/eql-workflow-filters.test.mjs` walks each `mise run` + # out to the paths the task names and fails on any this list does not select. + paths: + - ".github/workflows/test-eql.yml" + - ".github/actions/require-cs-secrets/**" + - "packages/eql/src/**" + - "packages/eql/tests/**" + - "packages/eql/tasks/**" + - "packages/eql/crates/**" + - "packages/eql/packages/**" + - "packages/eql/Cargo.toml" + - "packages/eql/Cargo.lock" + - "packages/eql/mise.toml" + - "packages/eql/docs/**" + - "packages/eql/docker/**" + - "packages/eql/README.md" + - "packages/eql/SUPABASE.md" + schedule: + - cron: "0 4 * * *" # 04:00 UTC daily; full matrix, off the merge path + merge_group: {} # inert today (no queue); kept so enabling one works + workflow_dispatch: {} # manual runs use the PR shape (PG17 x 4 shards) + +env: + FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true" + MISE_VERBOSE: "1" + # CI compile-time tuning (CI-only; local dev keeps full debuginfo + incremental). + # Clean CI builds never reuse incremental state, so it only bloats target/ and + # the rust-cache up/download. line-tables-only keeps readable panic backtraces + # for failing tests at a fraction of full-debuginfo compile cost. nextest's + # `test` profile inherits these from `dev`. + CARGO_INCREMENTAL: "0" + CARGO_PROFILE_DEV_DEBUG: "line-tables-only" + +defaults: + run: + shell: bash {0} + # Every `run:` in this workflow is written against the EQL root. This is the + # one-line form of what would otherwise be ~30 per-step `working-directory:` + # lines, each of which could be forgotten independently. + working-directory: packages/eql + +permissions: + contents: read + +# PRs cancel superseded runs; the merge queue must NOT cancel — a cancelled +# merge_group run never reports a final status and ejects the PR from the queue. +concurrency: + group: test-eql-${{ github.event_name }}-${{ github.ref }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + +jobs: + # Runs on EVERY event and MUST always succeed (never skipped, never failed) — + # downstream heavy jobs `needs: [changes]`, and a skipped/failed `changes` + # would either skip the merge-queue matrix or deadlock `ci-required`. + changes: + name: "Detect relevant changes" + runs-on: blacksmith-16vcpu-ubuntu-2204 + outputs: + relevant: ${{ steps.r.outputs.relevant }} + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + persist-credentials: false + + # Diff ONLY on pull_request, where a base ref is well-defined. On + # push/schedule/merge_group/workflow_dispatch there is no base ref to + # diff against and the filter errors or empties — so those events take the + # hardcoded default in the step below instead. `push` does not need it: + # its `on:`-level `paths:` has already decided whether the run happens. + - id: f + if: github.event_name == 'pull_request' + uses: dorny/paths-filter@d1c1ffe0248fe513906c8e24db8ea791d46f8590 # v3 + with: + # Every path except the workflow itself gained the `packages/eql/` + # prefix with the subtree. dorny/paths-filter matches against + # repo-root-relative paths, so the unprefixed globs would match + # `packages/stack/src/**` and friends instead — firing the whole + # matrix on changes that cannot affect EQL, and (worse) NOT firing on + # EQL's own `src/`, since nothing at the repo root is called that. + # + # Identical to the `push` `paths:` list at the top of this file — the + # same question asked of two mechanisms GitHub gives no way to share. + # `scripts/__tests__/eql-workflow-filters.test.mjs` compares them, and + # also fails any entry here that matches no tracked file (a dead + # `packages/eql/sql/**` sat in this list from the port: there is no + # such directory, so it narrowed the filter by exactly nothing while + # reading as coverage). + # + # The four documentation entries are inputs to `docs-static` and + # `doc-anchors`, neither of which is relevance-gated, so this copy + # does not strictly need them. It carries them because the three + # copies are held EQUAL — see the note above the `push` list, and the + # reasoning on `bench-eql.yml`'s own filter for why equal beats + # minimal. + filters: | + relevant: + - ".github/workflows/test-eql.yml" + - ".github/actions/require-cs-secrets/**" + - "packages/eql/src/**" + - "packages/eql/tests/**" + - "packages/eql/tasks/**" + - "packages/eql/crates/**" + - "packages/eql/packages/**" + - "packages/eql/Cargo.toml" + - "packages/eql/Cargo.lock" + - "packages/eql/mise.toml" + - "packages/eql/docs/**" + - "packages/eql/docker/**" + - "packages/eql/README.md" + - "packages/eql/SUPABASE.md" + + # Explicit default (not `|| 'true'`, which trips GitHub's inconsistent + # treatment of the string 'false'). push/schedule/merge_group/dispatch + # never run the filter above — it needs a base ref that only a pull + # request has — so they take the default, and `push` is already narrowed + # by its own `on:`-level `paths:`. + # + # The event arrives through `env:`, not through a `${{ }}` interpolated + # into the body. That is what lets + # `scripts/__tests__/eql-matrix-triggers.test.mjs` EXECUTE this script + # once per event and read back what it wrote, rather than reimplementing + # the branch and agreeing with a rewritten one by luck. + - id: r + env: + EVENT_NAME: ${{ github.event_name }} + FILTER_RELEVANT: ${{ steps.f.outputs.relevant }} + run: | + set -euo pipefail + if [ "$EVENT_NAME" = "pull_request" ]; then + echo "relevant=$FILTER_RELEVANT" >> "$GITHUB_OUTPUT" + else + echo "relevant=true" >> "$GITHUB_OUTPUT" + fi + + # Pure bash; no checkout/toolchain. Derives the PG-version + shard fan-out + # from the event: PR and manual dispatch -> PG17 x 4 shards (latency lane); + # everything else -> PG 14-17 x 2 shards (coverage lane). + setup: + name: "Compute matrix" + runs-on: blacksmith-16vcpu-ubuntu-2204 + # Opts out of the workflow-level `working-directory: packages/eql`. This job + # deliberately does not check out — it is pure bash deriving a matrix — so + # `packages/eql` does not exist on the runner and every `run:` step fails + # before its first line, with `chdir: No such file or directory`. Guarded by + # `scripts/__tests__/workflow-working-directory.test.mjs`. + defaults: + run: + working-directory: . + outputs: + pg-versions: ${{ steps.cfg.outputs.pg }} + shard-total: ${{ steps.cfg.outputs.shard_total }} + shards: ${{ steps.cfg.outputs.shards }} + steps: + # Written as an exclusion, and that direction is the point: the FULL + # matrix is the default, so a trigger added later lands in the coverage + # lane rather than in the fast one. The version of this that enumerated + # `merge_group` as the only full-matrix event is why PG 14-16 went + # untested for the life of the port — an allowlist fails shut on the case + # nobody enumerated, and here "shut" meant three Postgres versions. + # + # The event arrives through `env:` rather than an interpolated `${{ }}` + # so `scripts/__tests__/eql-matrix-triggers.test.mjs` can run this script + # against each event and read the real fan-out back out of + # `$GITHUB_OUTPUT`. + - id: cfg + env: + EVENT_NAME: ${{ github.event_name }} + run: | + set -euo pipefail + case "$EVENT_NAME" in + pull_request|workflow_dispatch) + # Latency lane: PG17 only, split four ways. + echo 'pg=[17]' >> "$GITHUB_OUTPUT" + echo 'shard_total=4' >> "$GITHUB_OUTPUT" + echo 'shards=[1,2,3,4]' >> "$GITHUB_OUTPUT" + ;; + *) + # Coverage lane: every version tests/docker-compose.yml ships a + # container for. Kept in step by the guard named above. + echo 'pg=[14,15,16,17]' >> "$GITHUB_OUTPUT" + echo 'shard_total=2' >> "$GITHUB_OUTPUT" + echo 'shards=[1,2]' >> "$GITHUB_OUTPUT" + ;; + esac + + # Compile the test binaries ONCE. Runs on every non-PR event, and on PRs only + # when relevant files changed (docs-only PRs never pay the ~4-min compile). + build-archive: + name: "Build test archive" + needs: [changes] + # This repo is PUBLIC and accepts fork PRs (the approval policy only gates + # first-time contributors). build-archive is the sole holder of the CS_* + # credentials below, so it must never run on a fork PR. The trailing clause + # is the only thing that skips it: a fork pull request. Every other event — + # push, schedule, merge_group, dispatch, and a same-repo branch PR — runs; + # the downstream test/validate shards `needs:` it, so they skip on fork PRs + # too. + # + # Both clauses are written as exclusions rather than as a list of admitted + # events. An allowlist fails SHUT on the event nobody enumerated, and the + # `push`/`schedule` triggers added above would have skipped every job in + # this file while the run still reported success. Pinned by + # `scripts/__tests__/eql-matrix-triggers.test.mjs`; the fork clause itself + # is compared verbatim across all eight copies in the repo by + # `scripts/__tests__/workflow-dispatch-job-conditions.test.mjs`. + if: >- + (github.event_name != 'pull_request' + || needs.changes.outputs.relevant == 'true') + && (github.event_name != 'pull_request' + || github.event.pull_request.head.repo.full_name == github.repository) + runs-on: blacksmith-16vcpu-ubuntu-2204 + env: + # test:sqlx:archive depends on test:sqlx:prep, which copies the built EQL + # into migrations/, applies it to a live Postgres, and regenerates the + # per-type fixtures — both are include_str!'d into the test binaries at + # COMPILE time, so they must exist before `cargo nextest archive`. Fixture + # generation needs a live PG with EQL installed (the postgres:up step + # below) plus CS_* creds. The job-level `if:` above keeps those creds off + # fork-PR runs. + POSTGRES_VERSION: "17" + CS_CLIENT_ACCESS_KEY: ${{ secrets.CS_CLIENT_ACCESS_KEY }} + CS_WORKSPACE_CRN: ${{ vars.CS_WORKSPACE_CRN }} + CS_CLIENT_ID: ${{ vars.CS_CLIENT_ID }} + CS_CLIENT_KEY: ${{ secrets.CS_CLIENT_KEY }} + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + persist-credentials: false + + # Fast pre-flight: fail in seconds if a secret was rotated or cleared, + # before the rust-cache restore and the ~4-minute nextest archive build. + # Ordering is asserted by scripts/__tests__/ffi-binding-step-order.test.mjs. + - uses: ./.github/actions/require-cs-secrets + with: + workspace-crn: ${{ vars.CS_WORKSPACE_CRN }} + client-id: ${{ vars.CS_CLIENT_ID }} + client-key: ${{ secrets.CS_CLIENT_KEY }} + client-access-key: ${{ secrets.CS_CLIENT_ACCESS_KEY }} + + - uses: jdx/mise-action@1648a7812b9aeae629881980618f079932869151 # v4 + with: + version: 2026.4.0 + install: true + cache: true + working_directory: packages/eql + + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 + with: + workspaces: packages/eql + shared-key: sqlx-tests + # The sole saver of the shared cache: this job compiles the full heavy + # dep tree, so it must own the `sqlx-tests` key. All other jobs set + # `save-if: false` so a fast-finishing light job can't win the save race + # and overwrite the key with a deps-less target/. + + - name: Setup database (Postgres 17) + run: | + mise run postgres:up postgres-${POSTGRES_VERSION} --extra-args "--detach --wait" + + - name: Build EQL + archive test binaries + run: | + mise run test:sqlx:archive + + # Ship the built release artifacts: build_validation_tests read + # cipherstash-encrypt{,-uninstall}.sql from ../../release at RUN time + # (std::fs, not embedded), and release/ is gitignored so the shard checkout + # has none of them. `mise run build` (via prep) produced them in build-archive. + # + # `path:` is workspace-root relative — `defaults.run.working-directory` + # does not reach a `uses:` step — hence the prefix. upload-artifact v4 + # roots the archive at the least common ancestor of everything it matched, + # which for these two is `packages/eql`, so the artifact still contains + # `nextest.tar.zst` and `release/*.sql` at its top level and the shard's + # `download-artifact` puts them back where the archive expects them. + - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 + with: + name: nextest-archive + path: | + packages/eql/nextest.tar.zst + packages/eql/release/*.sql + retention-days: 1 + if-no-files-found: error + + # Sharded sqlx suite. No longer needs [schema, codegen] (gate removed) — + # shards start right after build-archive. + test: + name: "Shard PG${{ matrix.postgres-version }} ${{ matrix.shard }}/${{ needs.setup.outputs.shard-total }}" + needs: [changes, setup, build-archive] + # Exclusion, not an allowlist — see the note on `build-archive`. + if: >- + github.event_name != 'pull_request' + || needs.changes.outputs.relevant == 'true' + runs-on: blacksmith-16vcpu-ubuntu-2204 + strategy: + fail-fast: false + matrix: + postgres-version: ${{ fromJSON(needs.setup.outputs.pg-versions) }} + shard: ${{ fromJSON(needs.setup.outputs.shards) }} + env: + POSTGRES_VERSION: ${{ matrix.postgres-version }} + SHARD: ${{ matrix.shard }} + SHARD_TOTAL: ${{ needs.setup.outputs.shard-total }} + # No CS_* here: the shard runs the prebuilt archive (fixtures + migration + # embedded by build-archive), so it does not regenerate fixtures and needs + # no credentials. It only needs the live Postgres (below) for sqlx::test's + # per-test scratch databases and the release/*.sql from the artifact. + steps: + # Checkout path MUST be identical to build-archive so the archive's + # workspace remap lines up (design: archive<->commit coupling). + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + persist-credentials: false + + - uses: jdx/mise-action@1648a7812b9aeae629881980618f079932869151 # v4 + with: + version: 2026.4.0 + install: true + cache: true + working_directory: packages/eql + + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 + with: + workspaces: packages/eql + shared-key: sqlx-tests + save-if: false + + # Extracts to the workspace root by default, which would drop + # `nextest.tar.zst` and `release/` two levels above where the archive's + # workspace remap and the build_validation_tests expect them. + - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 + with: + name: nextest-archive + path: packages/eql + + - name: Setup database (Postgres ${{ matrix.postgres-version }}) + run: | + mise run postgres:up postgres-${POSTGRES_VERSION} --extra-args "--detach --wait" + + - name: Run shard ${{ matrix.shard }}/${{ needs.setup.outputs.shard-total }} + run: | + mise run test:sqlx:partition + + # docs:validate + Clean-DB v3 install smoke. Both are version-relevant, so + # they follow the event's PG set (PG17 on PR; 14-17 in the queue). Moved out + # of the old per-version test job so they run ONCE per version, not per shard. + validate: + name: "Validate (Postgres ${{ matrix.postgres-version }})" + needs: [changes, setup] + # Exclusion, not an allowlist — see the note on `build-archive`. + if: >- + github.event_name != 'pull_request' + || needs.changes.outputs.relevant == 'true' + runs-on: blacksmith-16vcpu-ubuntu-2204 + strategy: + fail-fast: false + matrix: + postgres-version: ${{ fromJSON(needs.setup.outputs.pg-versions) }} + env: + POSTGRES_VERSION: ${{ matrix.postgres-version }} + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + persist-credentials: false + + - uses: jdx/mise-action@1648a7812b9aeae629881980618f079932869151 # v4 + with: + version: 2026.4.0 + install: true + cache: true + working_directory: packages/eql + + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 + with: + workspaces: packages/eql + shared-key: sqlx-tests + save-if: false + + - name: Setup database (Postgres ${{ matrix.postgres-version }}) + run: | + mise run postgres:up postgres-${POSTGRES_VERSION} --extra-args "--detach --wait" + + # Source-only doc checks (coverage + required-tags) moved to the dedicated + # `docs-static` job so they run exactly once, not per-Postgres. This step + # keeps only the DB-backed SQL-syntax validation, which genuinely needs the + # per-version Postgres. + - name: Validate documented SQL syntax (Postgres ${{ matrix.postgres-version }}) + run: | + mise run docs:validate:documented-sql + + - name: Clean-DB v3 install smoke (Postgres ${{ matrix.postgres-version }}) + # `set -euo pipefail` for the reason spelled out on `matrix-coverage`'s + # inventory step and enforced by + # `scripts/__tests__/workflow-run-fail-fast.test.mjs`: the workflow-wide + # `shell: bash {0}` drops GitHub's implicit `-eo pipefail`, so without + # it a failing BUILD on the first line is discarded and the smoke test + # on the second decides the step — i.e. the install smoke would pass by + # installing the previous build. + run: | + set -euo pipefail + mise run clean && mise run build + mise run test:clean_install_v3 + + schema: + name: "JSON Schema validation" + needs: [changes] + # Exclusion, not an allowlist — see the note on `build-archive`. + if: >- + github.event_name != 'pull_request' + || needs.changes.outputs.relevant == 'true' + runs-on: blacksmith-16vcpu-ubuntu-2204 + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + persist-credentials: false + - uses: jdx/mise-action@1648a7812b9aeae629881980618f079932869151 # v4 + with: + version: 2026.4.0 + install: true + cache: true + working_directory: packages/eql + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 + with: + workspaces: packages/eql + shared-key: sqlx-tests + save-if: false + - name: Validate v2.2 / v2.3 payload schemas + run: | + mise run test:schema + + rust-crates: + name: "Rust workspace crates" + needs: [changes] + # Exclusion, not an allowlist — see the note on `build-archive`. + if: >- + github.event_name != 'pull_request' + || needs.changes.outputs.relevant == 'true' + runs-on: blacksmith-16vcpu-ubuntu-2204 + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + persist-credentials: false + - uses: jdx/mise-action@1648a7812b9aeae629881980618f079932869151 # v4 + with: + version: 2026.4.0 + install: true + cache: true + working_directory: packages/eql + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 + with: + workspaces: packages/eql + shared-key: sqlx-tests + save-if: false + # `mise run test:crates` runs `cargo fmt --check` at the workspace root, + # which covers tests/sqlx (a workspace member). This subsumes the old + # standalone `test:lint` step that the removed per-version test job ran. + - name: Compile, lint and test the Rust workspace crates + run: | + set -euo pipefail + # Assignment on its own, not `export x=$(…)`: errexit ignores the + # substitution's status when the line starts with a command word, so + # the `export` form would swallow a failing `rustup show` even under + # `set -e`. + active_rust_toolchain=$(rustup show active-toolchain | cut -d' ' -f1) + rustup component add --toolchain "${active_rust_toolchain}" rustfmt clippy + mise run test:crates + + # Freshness gate for the eql-types codegen output: regenerate the + # TypeScript bindings and JSON Schemas and fail if the checked-in + # copies differ. Reuses the toolchain from the step above. + - name: Verify eql-types bindings and schemas are fresh + run: | + mise run types:check + + # Publish gate for eql-bindings (the one crate we ship to crates.io via + # release-plz). `--dry-run` packages + compiles the crate exactly as + # crates.io would, catching publish-blockers — missing `license`/metadata, + # a real path dependency without a version — on the PR rather than at + # release time. No token needed. `--allow-dirty` tolerates any files the + # preceding regenerate-and-diff steps leave in the working tree. + - name: Verify eql-bindings packages cleanly for crates.io + run: | + cargo publish -p eql-bindings --dry-run --allow-dirty + + codegen: + name: "Encrypted-domain codegen" + needs: [changes] + # Exclusion, not an allowlist — see the note on `build-archive`. + if: >- + github.event_name != 'pull_request' + || needs.changes.outputs.relevant == 'true' + runs-on: blacksmith-16vcpu-ubuntu-2204 + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + persist-credentials: false + - uses: jdx/mise-action@1648a7812b9aeae629881980618f079932869151 # v4 + with: + version: 2026.4.0 + install: true + cache: true + working_directory: packages/eql + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 + with: + workspaces: packages/eql + shared-key: sqlx-tests + save-if: false + - name: Verify generator parity (golden) + run: | + mise run codegen:parity + - name: Verify schema split parity (SQL owned_schemas vs Rust consts) + run: | + mise run test:schemas:parity + + self-contained-v3: + name: "eql_v3 self-containment" + needs: [changes] + # Exclusion, not an allowlist — see the note on `build-archive`. + if: >- + github.event_name != 'pull_request' + || needs.changes.outputs.relevant == 'true' + runs-on: blacksmith-16vcpu-ubuntu-2204 + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + persist-credentials: false + - uses: jdx/mise-action@1648a7812b9aeae629881980618f079932869151 # v4 + with: + version: 2026.4.0 + install: true + cache: true + working_directory: packages/eql + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 + with: + workspaces: packages/eql + shared-key: sqlx-tests + save-if: false + - name: Build EQL + run: mise run clean && mise run --force build + - name: Assert eql_v3 is self-contained + run: mise run test:self_contained_v3 + - name: Assert the installer contains every ordered file + run: mise run test:installer_complete + - name: Symbol-order cross-check (v3) + run: mise run test:symbol_order_v3 + - name: Build-ordering helper unit tests + run: mise run test:build_ordering_helpers + + matrix-coverage: + name: "Matrix coverage inventory" + needs: [changes] + # Exclusion, not an allowlist — see the note on `build-archive`. + if: >- + github.event_name != 'pull_request' + || needs.changes.outputs.relevant == 'true' + runs-on: blacksmith-16vcpu-ubuntu-2204 + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + persist-credentials: false + - uses: jdx/mise-action@1648a7812b9aeae629881980618f079932869151 # v4 + with: + version: 2026.4.0 + install: true + cache: true + working_directory: packages/eql + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 + with: + workspaces: packages/eql + shared-key: sqlx-tests + save-if: false + - name: Verify the matrix test-name inventory + # The workflow-wide `shell: bash {0}` default drops GitHub's implicit + # `-eo pipefail`, so without fail-fast this multi-command step only + # reports the LAST command's exit code and a failing inventory task is + # silently swallowed. Restore fail-fast for this step. + run: | + set -euo pipefail + mise run test:matrix:inventory + mise run test:matrix:inventory:jsonb_entry + mise run test:matrix:inventory:ope + mise run test:v3-jsonb:inventory + git add -N tests/sqlx/snapshots + git diff --exit-code -- tests/sqlx/snapshots \ + || { echo "Coverage inventory stale — run the relevant inventory task and commit."; exit 1; } + - name: Verify catalog-surface coverage + run: mise run test:matrix:catalog-coverage + + splinter: + name: "Supabase splinter" + needs: [changes] + # Exclusion, not an allowlist — see the note on `build-archive`. + if: >- + github.event_name != 'pull_request' + || needs.changes.outputs.relevant == 'true' + runs-on: blacksmith-16vcpu-ubuntu-2204 + env: + POSTGRES_VERSION: "17" + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + persist-credentials: false + - uses: jdx/mise-action@1648a7812b9aeae629881980618f079932869151 # v4 + with: + version: 2026.4.0 + install: true + cache: true + working_directory: packages/eql + # This job looks Rust-free and is not. `test:splinter` is a bash script + # that pipes SQL through psql, but the build step below runs `mise run + # build`, and `tasks/build.sh` shells out to `cargo run -q -p eql-codegen` + # twice — so without this restore the job cold-compiles that crate and its + # dependency tree on every run, beside nine sibling jobs that restore a + # warm `target/` for the same workspace. The only symptom is a slower run. + # + # `save-if: false` for the reason `build-archive` records: it is the sole + # designated saver of this shared key. Both halves are asserted by + # scripts/__tests__/eql-suite-ci.test.mjs, which finds the cargo by + # closing over the mise task graph rather than grepping this file — the + # string `cargo` appears nowhere in it. + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 + with: + workspaces: packages/eql + shared-key: sqlx-tests + save-if: false + - name: Setup database + run: | + mise run postgres:up postgres-${POSTGRES_VERSION} --extra-args "--detach --wait" + - name: Build and install EQL + run: | + set -euo pipefail + mise run clean && mise run --output prefix --force build + # Redirected, not piped. `psql -v ON_ERROR_STOP=1` exits at the first + # SQL error, which under `pipefail` leaves a `cat` writing into a + # closed pipe and reports the step as 141 — the SIGPIPE hazard + # `scripts/__tests__/workflow-grep-q-pipelines.test.mjs` documents. + # The redirect is the same bytes with no writer to signal, so the + # step reports psql's own status and says which statement failed. + docker exec -i postgres-${POSTGRES_VERSION} \ + psql -v ON_ERROR_STOP=1 \ + postgresql://cipherstash:password@localhost/cipherstash -f- \ + < release/cipherstash-encrypt.sql + - name: Run splinter + run: | + mise run --output prefix test:splinter --postgres ${POSTGRES_VERSION} + + # Source-only SQL documentation validation (coverage + required Doxygen tags). + # Deliberately NOT relevance-gated: it runs on EVERY pull_request — including + # docs-only PRs that skip the heavy jobs — so documentation is always + # validated. DB-free and creds-free (the psql-backed syntax check stays in the + # per-version `validate` job). + docs-static: + name: "SQL doc validation" + runs-on: blacksmith-16vcpu-ubuntu-2204 + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + persist-credentials: false + - uses: jdx/mise-action@1648a7812b9aeae629881980618f079932869151 # v4 + with: + version: 2026.4.0 + install: true + cache: true + working_directory: packages/eql + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 + with: + workspaces: packages/eql + shared-key: sqlx-tests + save-if: false + - name: Validate SQL doc coverage + required tags + run: | + mise run docs:validate:source + + - name: Assert user-facing docs are free of the removed eql_v2 surface + run: | + mise run test:docs_v3_grep + + - name: Assert public files contain no private issue identifiers + run: | + mise run test:public_identifiers + + # Every `known_failure` marker must name a real, OPEN issue, and its constant + # must actually be referenced by a test. The Rust half of the contract is + # self-expiring (the marker FAILS once the wrapped assertion starts passing); + # this is the half that stops a suppression outliving a closed issue. + # + # Credential-free and DB-free — it only reads the registry and asks GitHub for + # issue state, so it runs on every PR rather than hiding behind the e2e job. + known-failures: + name: "known-failure markers" + runs-on: blacksmith-16vcpu-ubuntu-2204 + permissions: + contents: read + issues: read + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + persist-credentials: false + - uses: jdx/mise-action@1648a7812b9aeae629881980618f079932869151 # v4 + with: + version: 2026.4.0 + install: true + cache: true + working_directory: packages/eql + # The gate detects drift by cross-checking a loose declaration count against + # a strict parse, so an over-accepting parser defeats it silently. Runs + # first, and needs no token: a broken parser makes the gate below + # meaningless, not merely wrong. + - name: The gate's strict ISSUE_ parser is sound + run: | + mise run test:known-failures:parser + + - name: Every suppressed test names an open issue + env: + # `gh` needs a token to read issue state; the default job token is + # sufficient (issues: read, same repo). + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + mise run test:known-failures + + # Markdown anchor links. DB-free, credential-free and fast, and deliberately + # NOT relevance-gated: its inputs are the docs themselves, so gating it on the + # `relevant` filter (src/**, crates/**) would skip it on exactly the docs-only + # PRs it exists to check. + doc-anchors: + name: "doc anchor links" + runs-on: blacksmith-16vcpu-ubuntu-2204 + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + persist-credentials: false + - uses: jdx/mise-action@1648a7812b9aeae629881980618f079932869151 # v4 + with: + version: 2026.4.0 + install: true + cache: true + working_directory: packages/eql + - name: Every intra-document anchor link resolves + run: | + mise run test:doc-anchors + + # The e2e (fresh-encryption) property suite. Encrypts random values through + # ZeroKMS at run time, so it needs CS_* creds and is PG-version-independent — + # one PG17 run, never the matrix. Compiles the `proptest-e2e`-gated binaries + # (which the default-feature sharded archive excludes) and runs only the + # e2e oracle. Like build-archive, it holds CS_* and so carries the same + # fork-PR guard to keep the secrets off fork runs. + e2e: + name: "e2e property suite (fresh encryption)" + needs: [changes, setup] + # Exclusion, not an allowlist — see the note on `build-archive`, which + # carries the same two clauses for the same two reasons. + if: >- + (github.event_name != 'pull_request' + || needs.changes.outputs.relevant == 'true') + && (github.event_name != 'pull_request' + || github.event.pull_request.head.repo.full_name == github.repository) + runs-on: blacksmith-16vcpu-ubuntu-2204 + env: + POSTGRES_VERSION: "17" + CS_CLIENT_ACCESS_KEY: ${{ secrets.CS_CLIENT_ACCESS_KEY }} + CS_WORKSPACE_CRN: ${{ vars.CS_WORKSPACE_CRN }} + CS_CLIENT_ID: ${{ vars.CS_CLIENT_ID }} + CS_CLIENT_KEY: ${{ secrets.CS_CLIENT_KEY }} + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + persist-credentials: false + # Fast pre-flight: fail in seconds if a secret was rotated or cleared, + # before the rust-cache restore and the proptest-e2e compile. + - uses: ./.github/actions/require-cs-secrets + with: + workspace-crn: ${{ vars.CS_WORKSPACE_CRN }} + client-id: ${{ vars.CS_CLIENT_ID }} + client-key: ${{ secrets.CS_CLIENT_KEY }} + client-access-key: ${{ secrets.CS_CLIENT_ACCESS_KEY }} + - uses: jdx/mise-action@1648a7812b9aeae629881980618f079932869151 # v4 + with: + version: 2026.4.0 + install: true + cache: true + working_directory: packages/eql + - uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 + with: + workspaces: packages/eql + shared-key: sqlx-tests + save-if: false + - name: Setup database (Postgres 17) + run: | + mise run postgres:up postgres-${POSTGRES_VERSION} --extra-args "--detach --wait" + - name: Run e2e property suite + run: | + mise run test:sqlx:e2e + + # The ONE required status check. Stable name on every event, so branch + # protection never references an event-dependent leaf name (which would + # deadlock). Passes iff every needed job is success or skipped. Treating + # skipped as pass is intentional: heavy jobs are legitimately skipped on + # docs-only PRs, and a genuine failure is still caught because the FAILING + # source job is itself in `needs` and reports failure. + ci-required: + name: "ci-required" + needs: [changes, setup, build-archive, test, validate, schema, rust-crates, + codegen, self-contained-v3, matrix-coverage, splinter, docs-static, + known-failures, doc-anchors, e2e] + if: always() + runs-on: blacksmith-16vcpu-ubuntu-2204 + # Same opt-out as `setup`, and this is the worse of the two to get wrong: + # this job is the required check. It reads `needs.*.result` and checks out + # nothing, so under the workflow-level `working-directory` it fails for a + # reason unrelated to any job it is reporting on — a red required check that + # says nothing about the suite. + defaults: + run: + working-directory: . + steps: + - name: Assert all required jobs passed or were skipped + run: | + set -euo pipefail + results='${{ join(needs.*.result, ' ') }}' + echo "needed results: $results" + for r in $results; do + case "$r" in + success|skipped) ;; + *) echo "gate fail: a needed job reported '$r'"; exit 1 ;; + esac + done + echo "ci-required: all needed jobs passed or were skipped" diff --git a/.github/workflows/tests-rust.yml b/.github/workflows/tests-rust.yml index f5a949543..facb00def 100644 --- a/.github/workflows/tests-rust.yml +++ b/.github/workflows/tests-rust.yml @@ -21,6 +21,24 @@ on: - 'packages/protect-ffi/Cargo.lock' - 'packages/protect-ffi/mise.toml' - 'packages/protect-ffi/package.json' + # Out of that package, and compiled by every check below all the same: + # `crates/protect-ffi/Cargo.toml` carries + # `eql-bindings = { path = "../../../eql/crates/eql-bindings" }`, so + # `cargo test`, `cargo fmt --check` and both clippy passes build that + # tree. A change there can break this job in a PR touching no file under + # packages/protect-ffi at all. The workspace root manifest comes with it: + # cargo reads it for the crate's workspace context, so a + # `.workspace = true` added to eql-bindings becomes a compile input + # with no other trace. Same two entries as the native cache key in + # `.github/actions/build-ffi-binding`, which is where the list is derived + # from — pinned by + # scripts/__tests__/wasm-build-inputs-paths-filter.test.mjs. + # + # This is the only copy: `push:` below is `branches: [main]` with no + # `paths:`, so main runs a superset of what PRs run (recorded as a + # deliberate asymmetry in workflow-paths-filter-parity.test.mjs). + - 'packages/eql/crates/**' + - 'packages/eql/Cargo.toml' - '.github/workflows/tests-rust.yml' push: branches: [main] diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 04c8bf71a..1e782fe9d 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -128,14 +128,14 @@ jobs: uses: ./.github/actions/build-ffi-binding - name: Type tests (stack) - run: pnpm --filter @cipherstash/stack run test:types + run: pnpm exec turbo run test:types --filter @cipherstash/stack # The v3 domain catalog lives in the test kit, and its # `satisfies Record` is what forces a new SDK # domain to be covered. That check only fires under `tsc`, so without this # step a domain added to stack would slip through untested. - name: Type tests (test-kit — enforces v3 domain coverage) - run: pnpm --filter @cipherstash/test-kit run test:types + run: pnpm exec turbo run test:types --filter @cipherstash/test-kit # The adapter packages carry their own `.test-d.ts` type-contract guards # (the M1 client-surface / #622 erasure guards, the Supabase key-set @@ -143,19 +143,29 @@ jobs: # were never wired into CI, so type-level regressions in them went # undetected — run them explicitly. - name: Type tests (stack-drizzle) - run: pnpm --filter @cipherstash/stack-drizzle run test:types + run: pnpm exec turbo run test:types --filter @cipherstash/stack-drizzle - name: Type tests (stack-supabase) - run: pnpm --filter @cipherstash/stack-supabase run test:types + run: pnpm exec turbo run test:types --filter @cipherstash/stack-supabase # prisma-next's operator-capability gating is proven by `.test-d.ts` # `@ts-expect-error` assertions (an unsupported operator on a column # must be a compile error). Those only fire when tsc processes them, # so the package `typecheck` must run — and be required — here (#684). - # Its tsconfig resolves stack subpaths to SOURCE, so no build step is - # needed first. + # Its tsconfig resolves stack subpaths to SOURCE — but NOT + # `@cipherstash/eql`, which it reaches through the package `exports` map + # at `./dist/sql.d.ts`. That used to be a registry tarball with `dist/` + # already in it, so a bare `pnpm --filter` resolved it whatever CI had + # built. The EQL subtree import made it a workspace package whose `dist/` + # is a build output, and this step became the first in the job to need + # something built — failing `TS2307` on three files. + # + # Through turbo, so `typecheck`'s `dependsOn: ["^build"]` builds + # `@cipherstash/eql` first. This is the trap + # `workflow-turbo-build-deps.test.mjs` was written for; the step was on + # its grandfathered KNOWN_BARE list, and the import is what collected. - name: Typecheck (prisma-next — enforces v3 operator-capability gating) - run: pnpm --filter @cipherstash/stack-prisma run typecheck + run: pnpm exec turbo run typecheck --filter @cipherstash/stack-prisma # `packages/bench` is a live importer of `@cipherstash/stack` and # `@cipherstash/stack-drizzle`, but it has no `test` script (its suites @@ -171,7 +181,7 @@ jobs: # nothing here caught the three `auth.AutoStrategy` resolution errors that # sat in `main` until #771. Gate it so they cannot come back silently. - name: Typecheck (wizard) - run: pnpm --filter @cipherstash/wizard run typecheck + run: pnpm exec turbo run typecheck --filter @cipherstash/wizard # `examples/*` are standalone apps outside the `./packages/*` filter that # root `build`/`test` use, so nothing in CI compiled them. `examples/basic` @@ -246,6 +256,42 @@ jobs: - name: Lint — no references to deleted package directories run: pnpm run lint:package-paths + # `eql-bindings` emits EQL payloads; `@cipherstash/eql` carries the SQL + # that stores them. Both live here now and release at one lockstep + # version. A registry pin on either lets them drift apart — it compiles, + # it passes every suite, and it fails in a customer's database. + - name: Lint — EQL resolves in-tree, not from a registry + run: pnpm run lint:eql-pins + + # THE SAME GATE release.yml RUNS, one merge earlier. + # + # `changeset publish` publishes every public workspace package whose + # version is absent from npm — with no dependency ordering, and a failed + # publish returns a result rather than throwing, so it does not stop the + # siblings that depend on it. A merge can therefore ARM a release that + # cannot succeed, and this repository has done it. `@cipherstash/eql` is + # published from cipherstash/encrypt-query-language, so a version bumped + # in this workspace cannot be published from here — while `packages/cli` + # and `packages/stack-prisma` carry it in RUNTIME dependencies, so pnpm + # packs whatever version this tree holds into both tarballs. Bump it past + # what the registry carries and both ship a range no published version + # satisfies, in tarballs that publish fine. That is what the hand-applied + # 3.0.5 bump did; upstream has since released 3.0.5 and it cleared on its + # own. The CONDITION is what this step is for — the verdict is whatever it + # prints on the day. + # + # The gate in release.yml blocks the publish, which is the load-bearing + # half. This step is the half that says so BEFORE the merge rather than + # after it, which is the difference between a red pull request and a red + # main. Same script, same verdict, no second copy to drift. + # + # It talks to the registry, in a job that already installs the whole + # workspace from it — so this adds a class of dependency the job does not + # already have, only a few more requests. A lookup that fails for any + # reason other than a 404 throws rather than being read as "published". + - name: Lint — this tree can be published, and installed + run: pnpm run release:gate + - name: Test — lint script self-tests run: pnpm run test:scripts diff --git a/AGENTS.md b/AGENTS.md index cb61d57ca..4f5bc9482 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -83,6 +83,7 @@ If these variables are missing, tests that require live encryption will fail or - `packages/utils`: Shared config (`utils/config`) and logger (`utils/logger`) - `packages/bench`: Performance / index-engagement benchmarks (private, not published) - `packages/protect-ffi`: Native FFI bindings to the CipherStash Client SDK (`@cipherstash/protect-ffi`) — the Rust core that `packages/stack` encrypts and decrypts through, absorbed from `cipherstash/protectjs-ffi`. Contains a **nested Cargo workspace** (`crates/`) and six per-platform binary packages under `platforms/*`, each published as `@cipherstash/protect-ffi-` and linked here via `workspace:*`. See the "Working on protect-ffi" notes below before touching it — its default `test` and `build` are deliberately Rust-free. +- `packages/eql`: The Encrypt Query Language subtree — the SQL bundle that stores and queries encrypted payloads — absorbed from `cipherstash/encrypt-query-language`. **The directory is the subtree root, not the package.** It was imported at a *verbatim prefix* so its repo-root-relative paths (mise tasks, `Doxyfile`, `sync-generated.mjs`) keep resolving, which puts the npm package `@cipherstash/eql` two levels down at `packages/eql/packages/eql` — the same shape as `packages/protect-ffi/platforms/*`, and enrolled the same way, by an explicit `packages/eql/packages/*` glob in `pnpm-workspace.yaml`. The subtree root deliberately carries no `package.json`. Also contains a **nested Cargo workspace** at `packages/eql/crates/` (`eql-bindings`, published in lockstep with the npm package, plus `eql-domains` / `eql-codegen` / `eql-tests-macros`, which are not), a SQLx test crate at `packages/eql/tests/sqlx`, an ~900-line `mise.toml` task surface, its own `AGENTS.md`, and `docs/`. See the "Working on EQL" notes below before touching it. - `e2e/*`: Cross-package end-to-end tests (package managers, supply chain, Prisma example README) - `examples/*`: Working apps (basic, prisma, supabase-worker) - `docs/plans/*`: Internal design plans. User-facing documentation lives at https://cipherstash.com/docs (not in this repo). @@ -90,11 +91,12 @@ If these variables are missing, tests that require live encryption will fail or ## Working on protect-ffi -`packages/protect-ffi` is the only Rust in this repo, and its scripts are split -so that stays true for everyone else. +`packages/protect-ffi` carries one of this repo's two Cargo workspaces (the +other is `packages/eql/crates`), and its scripts are split so a Rust toolchain +stays optional for everyone else. - **The default `test` and `build` never invoke cargo.** Root `pnpm test` runs - `turbo test --filter './packages/*'`, which reaches this package — so a cargo + `turbo test --filter './packages/**'`, which reaches this package — so a cargo process on that path is a Rust toolchain on every contributor's machine. `test` is the JS chain; `build` is `tsc`. - **CI does build the binding, in the jobs that need it.** That is the limit of @@ -113,9 +115,25 @@ so that stays true for everyone else. every job that receives a `CS_*` credential must build the binding, and the `require-cs-secrets` pre-flight must come first. Both scan the workflow directory rather than a list, so a new job is covered the day it lands. -- **Rust checks live behind `test:cargo`** (`cargo test` + `cargo fmt --check`) - and `mise run lint:rust` (clippy, host and wasm32). `build:native` carries - `cargo build --release`. +- **Rust checks live behind `test:cargo`** (`cargo test --locked` + `cargo fmt + --check`) and `mise run lint:rust` (clippy, host and wasm32). `build:native` + carries `cargo build --release`. + **`--locked` is on the CHECK and deliberately not on the builds.** Nothing in + this repo passed it at all until the #915 follow-up, and the bill came due + through `sync-lockstep-versions.mjs`: it rewrites `eql-bindings`'s crate + version on every lockstep bump, `packages/protect-ffi` depends on that crate + by path, so its `Cargo.lock` records the version — and nothing updated it. + After the 3.0.5 bump `cargo metadata --locked` exited 101 while every cargo + command in CI regenerated the lock in memory, built against the regenerated + one and threw it away with the runner. Nothing went red. `build:native` is a + documented local command, and a contributor who has just edited `Cargo.toml` + regenerates the lock on their next build, legitimately — `--locked` there is + a failure at the end of a compile. The check answers the same question on the + same commit without standing in front of a build. + `src/lintWiring.test.ts` holds this: every cargo script reachable from + `test:cargo` must carry `--locked` or be exempted with a reason, and the one + exemption (`cargo fmt`, an external subcommand that resolves nothing and + forwards the flag to rustfmt) expires if it ever stops applying. - **`src/lintWiring.test.ts` enforces the split**: no `test:*` script may be unreachable from both entry points, nothing cargo may be reachable from `test`, and every cargo check must be reachable from `test:cargo`. A check @@ -134,17 +152,40 @@ so that stays true for everyone else. Rust. Everything else under `dist/` stays ignored. The re-inclusion chain spans the root `.gitignore`, the package's own, and a `.gitignore` wasm-pack generates — see the comments in each. -- **Publishing has moved here.** npm trusted publishing for all seven packages - is repointed at this repo, bound to `release.yml`, so write changesets for - them normally. Nothing has actually published from here yet — the first FFI - release is still ahead, and until it lands treat the path as configured rather - than proven. The remaining steps and what is still unverified live in +- **Publishing has moved here, and the path is proven.** npm trusted publishing + for all seven packages is repointed at this repo, bound to `release.yml`, so + write changesets for them normally. `@cipherstash/protect-ffi@0.32.0` was + published from here: its SLSA provenance names + `https://github.com/cipherstash/stack` and `.github/workflows/release.yml`, + which is the only evidence that settles it — + + ``` + curl -s https://registry.npmjs.org/-/npm/v1/attestations/@cipherstash%2fprotect-ffi@0.32.0 + ``` + + This paragraph said the opposite until 2026-08-20. It was written while + 0.31.0 was newest and true then; 0.32.0 landed and nothing brought the + sentence with it, so "treat the path as configured rather than proven" + outlived the release that proved it. Check the registry before repeating a + claim about what has or has not shipped. The remaining steps live in `docs/plans/2026-08-04-protect-ffi-monorepo-absorption.md`, Phase 4. -- **A `.md.deferred` changeset is now inert, not a CI failure.** The parking - convention and the `lint-no-ffi-changeset` guard that enforced it are both - gone. If you find such a file, it was written on a branch cut before the - cutover: `git mv` it back to `.md`, or the change it describes ships with no - changelog entry. Nothing detects one for you. +- **A `.md.deferred` changeset is now a CI failure, having briefly been inert.** + The parking convention and the `lint-no-ffi-changeset` guard that enforced it + are both gone — `e77bfcec` retired the guard and renamed the two files parked + at the time in one commit, and they released in + `@cipherstash/protect-ffi@0.32.0`. A file still carrying that suffix is + invisible to `@changesets/read`, so whatever it describes would ship with an + empty changelog entry and nothing in `changeset version` or `changeset + publish` would say so. + **Before renaming one back, check whether it has already released** — a + long-lived branch cut before the cutover still carries both files, and + reactivating them there republishes a shipped entry and re-bumps the package + for a change two versions old. Delete in that case; `git mv` back to `.md` + only if it is genuinely unreleased. + `scripts/__tests__/no-parked-changesets.test.mjs` fails on a parked file + either way, and also fails if the retired guard is reinstated alongside it — + the two rules contradict each other, and a half-retired convention is what + produces a parked file in the first place. - **The pipeline that publishes them.** `release.yml` asks `scripts/release-gate.mjs` which committed versions are missing from npm; if any FFI one is, `_build-ffi-artifacts.yml` compiles the six platforms with an @@ -218,6 +259,266 @@ EQL versions installed** in the database. what stops it going quiet again, and it deliberately scans only the repo-root workflow directory. +## Working on EQL + +`packages/eql` is a subtree import, not a package directory, and almost +everything surprising about it follows from that. Its own `AGENTS.md` +(`packages/eql/AGENTS.md`) covers EQL-internal work — SQL authoring, the codegen +pipeline, documentation standards. The notes below cover the *seams* with this +monorepo, which is where the silent failures are. + +- **The package is at `packages/eql/packages/eql`, two levels down.** The + subtree root has no `package.json` by design, so a tool that globs one level + under `packages/` selects the root — a directory with no manifest and no + scripts — and not the package. That is why root `pnpm test` is + `turbo test --filter './packages/**'` and not `'./packages/*'`: under the + one-level filter the task graph contained `@cipherstash/eql#build` (pulled in + transitively by its consumers) and **no `#test` at all**, so its Vitest suite + ran nowhere while CI stayed green. `build` can stay one-level because + consumers pull it through `^build`. Anything else that walks `packages/*` needs + the same treatment — `scripts/lint-typecheck-scope.mjs` already carries the + two nested roots explicitly. +- **Anything invoking a mise task must run with `working_directory: + packages/eql`.** `packages/eql/mise.toml` is ~900 lines and its + `[task_config].includes` pulls in `tasks/`, `tasks/postgres.toml` and + `tasks/fixtures.toml`; task bodies address `tasks/…`, `release/…` and + `tests/sqlx/…` relative to the subtree root. mise reads config from the + current directory and its parents, so invoking from the repo root finds no EQL + config and fails with a *trust* error that reads like a broken toolchain + rather than a wrong directory. `[env]` also pins `EQL_ROOT = {{config_root}}`, + because two task scripts use `git rev-parse --show-toplevel`, which after the + import returns the **monorepo** root — and one of them, + `tasks/test/doc-anchors.sh`, fails silently when that is wrong. +- **`packages/eql/.github/` is a dead deposit.** GitHub reads workflows from the + repo root and nowhere else, so the eleven files there run on nothing. Seven + are workflows — four publish something (`release.yml`, `release-plz.yml`, + `release-postgres-eql-image.yml`, `rebuild-docs.yml`) and three exist only to + serve them (two `workflow_call` reusables and `lint-release.yml`) — alongside + a `workflows/README.md`, the release-notes config, and two + repository-settings files. `scripts/__tests__/eql-suite-ci.test.mjs` holds + them as a **shrinking allowlist** (`UNPORTED_DEPOSIT`), asserted by equality so + it fails in both directions: porting a workflow means deleting it from the + deposit *and* from the list in the same commit, and dropping a new file in + there without listing it fails too. The same test asserts that the three SQLx + suite tasks are invoked by name from a root workflow, that every + `dorny/paths-filter` path is scoped to `packages/eql/`, and that every mise + task shelling out to cargo is either reachable from a root workflow or + exempted with a written reason. It is the guard against the failure this + absorption keeps rediscovering: a check that arrives as a file and executes on + no event reads exactly like a check that passes. + That scan now follows a task's `tasks/*.sh` delegations **transitively** + (cycle-guarded, and it throws rather than truncating past + `MAX_SCRIPT_DEPTH`), and it reads the `run:` bodies of composite actions a + workflow reaches through `uses: ./…`. Both used to stop at one hop, and both + failure directions were live: a cargo helper reached only at depth 2 dropped + out of `CARGO_TASKS` entirely — no orphan reported, no exemption demanded, + and the job running it stopped counting as a Rust job for the cache check — + while a mise task invoked from a composite action read as run by nobody, + whose natural repair is an exemption claiming CI does not run it. +- **The EQL path filters are three copies of one list, and the list is derived + now, not remembered.** `test-eql.yml` writes it twice (an `on: push: paths:` + filter deciding whether the workflow starts at all, and a + `dorny/paths-filter` `relevant:` block gating the heavy jobs inside a pull + request); `bench-eql.yml` writes a third copy. GitHub has no YAML anchors, so + nothing but `scripts/__tests__/eql-workflow-filters.test.mjs` keeps them + together. That file now also walks each `mise run` in both workflows out to + the task it names — through `[tasks."…"]` tables, file tasks under `tasks/`, + `depends`, and nested `mise run` calls — and fails if the `push` filter does + not select a path those task bodies name. That is what found + `packages/eql/docs/**`, `packages/eql/docker/**`, `packages/eql/README.md` + and `packages/eql/SUPABASE.md` missing from all three copies while + `docs-static`'s `mise run test:docs_v3_grep` scanned every one of them: a + push to main touching only documentation started no EQL workflow at all. + **`pull_request` was never affected** — it applies no `paths:` filter, and + `docs-static` and `doc-anchors` are deliberately not relevance-gated. The + derivation reads paths that are WRITTEN DOWN; it cannot see `postgres:up` + picking up `tests/docker-compose.yml` from its working directory, or the glob + pathspec in `tasks/test/doc-anchors.sh` (`git ls-files '*.md'`, i.e. every + tracked markdown file in the subtree). Those are still read by hand. +- **One version, five artefacts.** `@cipherstash/eql` (npm), the `eql-bindings` + crate, the SQL bundle, the docs and the `postgres-eql` image all ship at a + single version V. The npm package's `version` is the source of truth + (`changeset version` owns it) and `scripts/sync-lockstep-versions.mjs` + propagates it to the crate and, via `mise run + release:prepare_bindings_assets`, to the stamped SQL and release manifests. It + runs from the root `version` script — Changesets only invokes the *root* one, + which is why the script lives at the repo root and derives the subtree path + itself. **`mise run build --version X` does not treat `--version` as a + cache-key input**: it is absent from `tasks/build.sh`'s `#MISE sources`, so on + unchanged SQL and Rust it is a cache hit that re-serves whatever version the + previous build stamped. `tasks/release/prepare-bindings-assets.sh` passes + `--force` for exactly that reason and then greps the stamp back out of the + SQL before writing a manifest over it — read its comment before touching that + path. Without both, the bundle ships stamped one version under a manifest, + crate and npm package claiming another, and every digest still verifies. + `scripts/__tests__/eql-sql-asset-freshness.test.mjs` is what holds the + result. It compares the npm package's `version` against every + `release-manifest.json` under the subtree, against + `src/generated/release-manifest.ts`, against the `eql-bindings` crate + manifest, and — the one the cache hit actually breaks — against the `COMMENT + ON SCHEMA eql_v3 IS '…'` stamp inside the install bundle itself. All of those + are needed: the digests are recomputed over whatever bytes were served, so a + stale bundle verifies perfectly; only the stamp records which build produced + it. The predicate lives in `scripts/sync-lockstep-versions.mjs` + (`eqlLockstepSkew`) because the release hook needs the same answer, and a + PR-time guard that could disagree with the release-time decision is two + guards. **It is deliberately not keyed to `FROZEN_PUBLISHERS`.** The gate's + `FROZEN_ARTEFACT_DIGESTS` check compares the tree against *npm* and is + deleted at the Phase-5 cutover; this compares the tree against *itself*, + which is a property of a lockstep release rather than of who publishes it, so + it survives. +- **The version hook no longer rewrites the SQL assets on a release that does + not bump EQL.** `scripts/sync-lockstep-versions.mjs` runs on *every* release, + and its step 4 (`mise run release:prepare_bindings_assets`) re-hashed freshly + built SQL and overwrote all four release manifests plus both copies of the + bundle — unconditionally, including when the version had not moved. + `packages/eql/mise.toml` pins `rust = { version = "latest" }`, so the + toolchain compiling `eql-codegen` is not the same one month to month, and + nothing anywhere proves that regenerating from in-tree source reproduces + npm's published bytes. Put those together and a release that never touched + EQL can pick up a new digest under an unchanged version, at which point the + next `release-gate.mjs` run fires `frozen-bytes-skew`, the `gate` job exits + non-zero and `release` is skipped — the whole release blocked by an artefact + nobody was releasing, *after* `changeset version` has already rewritten every + manifest and CHANGELOG in the tree. + Step 4 now runs only when `eqlLockstepSkew` finds a disagreement, and + re-checks afterwards that the copy actually landed in both directories. A + real bump always disagrees (`package.json` moves first), so the skip is not + reachable by bumping. What it declines is the no-op — and for in-tree SQL + *source* changed without a changeset, declining is the correct answer rather + than a missed one: regenerating there republishes different bytes under a + released number, which is precisely what the gate refuses. The changeset is + what brings the rebuild back. + **Pinning `rust` in `packages/eql/mise.toml` was considered and rejected**: + it is a one-line divergence in the ~900-line file upstream edits most, it + changes what the whole EQL CI surface compiles against, and with the skip in + place the release-stopper is closed without it. If wanted, the pin belongs + upstream and arrives by subtree pull. +- **`eql-bindings` resolves by path from `packages/protect-ffi`, never from + crates.io**, and `scripts/lint-no-eql-registry-pins.mjs` (`pnpm run + lint:eql-pins`) is what keeps it that way. The two halves of EQL are the Rust + that EMITS a payload and the SQL that STORES and queries one; a registry pin + lets them drift apart silently — it compiles, it passes CI, and it fails in a + database. The linter reads every `Cargo.toml` and `package.json` plus + `pnpm-workspace.yaml` (pnpm resolves `overrides` and `catalogs` from there, + and a top-level npm-format `overrides` block in a `package.json` is silently + ignored, so it is the one place a workspace-wide pin can be written and take + effect). It exits **2**, not 0, when its own configuration has gone stale — + a source it could not read, a declaration it expected and no longer sees, or + an exemption excusing nothing. There is one exemption today + (`packages/protect-ffi/integration-tests`, which installs with `npm ci` and + cannot take a `workspace:` specifier); adding another means writing the reason + down. + It also reads the Cargo redirect tables — `[patch.*]` (including + `[patch."https://…"]` and the dotted `[patch.crates-io.eql-bindings]` form) + and `[replace]` — plus **`.cargo/config.toml`**, because cargo honours a + `[patch]` written there and a walk keyed on manifest filenames never opens + it. That is the `pnpm-workspace.yaml` failure one ecosystem along: the + quietest place to re-point a dependency is the file the linter was not + reading. Still not read: `[source.*] replace-with`, which redirects the whole + registry rather than naming a crate, so there is no `eql-bindings` + declaration to classify — closing it is a different check (does this tree + redirect crates.io at all), not an extension of this one. + **`packages/eql`'s own cargo tasks still pass no `--locked`.** `mise run + test:crates`, `codegen:parity` and the SQLx archive/partition tasks all shell + to a plain `cargo …`. `cargo tree --locked` exits 0 there today, so the flag + would pass if added — and this is the workspace whose `Cargo.lock` the + lockstep bump actually moves. Left open only because it means editing the + subtree's `mise.toml`. +- **Two SQLx test constants are keyed to this repo's CI workspace.** SteVec + selectors are MACs over (column context, JSONPath) under a *workspace keyset*, + so `SELECTOR` in `tests/sqlx/src/fixtures/v3_doc_integer.rs` and `SEL_HELLO_OP` + in `tests/sqlx/src/fixtures/v3_ste_vec.rs` changed value — with no change to + Rust, SQL or fixture logic — the moment CI moved to this repo's + `CS_WORKSPACE_CRN` (`9467cc5d`, `da141339`). Each has a drift guard that + prints the candidate selectors and their discriminators rather than inferring + a replacement, because guessing wrong re-pins to the wrong leaf silently — + which had already happened once, `SEL_HELLO_OP` naming `$.number` while + claiming `$.hello` and surviving because an equality-only assertion cannot + separate them. **If you run the suite against your own CipherStash workspace + the guards will fire: do not commit your local value.** Rotating + `CS_WORKSPACE_CRN` re-pins both. `tests/sqlx/src/selectors.rs` holds five more + workspace-keyed constants with no consumers and no guard — delete or guard + them before using any of them. +- **Publishing has not moved yet.** npm trusted publishing for `@cipherstash/eql` + still names `cipherstash/encrypt-query-language` — the SLSA provenance on + `@cipherstash/eql@3.0.5` records that repository and + `.github/workflows/release.yml` — and the package's own `repository` / `bugs` + fields still point there, as do the `eql-bindings` crate's. Repointing all of + it is the Phase-5 cutover, together with the nine parked workflows above. + This was protect-ffi's situation until its own cutover, with one difference: + the guard is not a changeset lint but `scripts/release-gate.mjs`, and it is + the stronger of the two. Its `FROZEN_PUBLISHERS` map lists every package that lives here but is + published from another repository, and the gate **exits non-zero** — failing + the `gate` job, which skips `release` entirely — if such a package's committed + version is missing from npm, if any published package carries a runtime + `workspace:` range that only that package could satisfy, or — the third + check, added after a review caught it by hand — if a frozen package's + in-tree artefact is **not the bytes published under the version the tree + claims**. Any one stops the Version Packages PR and the publish alike. A + changeset-side guard sees none of them — which is why the hand-applied 3.0.5 + bump needed this one. + + **That third check is the one worth understanding before you touch + `packages/eql`.** For a package this repo publishes, in-tree bytes differing + from npm is an unreleased change — every pull request. For a frozen one it is + a contradiction: the version cannot be released from here, so the tree is not + proposing those bytes, it is *asserting they are already on npm under that + number*. Nothing local can notice when that stops being true, because + `sql/release-manifest.json` is regenerated with the SQL and goes on agreeing + with it; only the registry disagrees. This branch shipped exactly that — a + `3.0.5` subtree whose install bundle hashed `7ad9c9f8…` against npm's + `accde0030…`, because upstream restored the deprecated `ste_vec_contains` + aliases in the real release — and `stash eql install` would have executed that + SQL against a customer database, which would then have carried functions the + version it reports does not define. (The CLI now refuses a bundle whose bytes + do not hash to its own release manifest — see + `packages/cli/src/installer/bundle-digest.ts`. That catches a corrupt or + tampered `node_modules`, **not** this: a frozen-package skew regenerates the + manifest alongside the SQL, so the two agree locally and only the registry + disagrees. The release gate is still the only thing that notices.) The check + `npm pack`s the frozen package + and compares the two release manifests; `FROZEN_ARTEFACT_DIGESTS` says which + artefact, keyed identically to `FROZEN_PUBLISHERS` and deleted with it at the + cutover. + **That `npm pack` + `tar` extraction was itself executed by no test until the + #915 follow-up** — every `frozenBytesSkew` unit test injected both digests, + and the end-to-end process test shimmed `npm` with a script answering `npm + view` only, so `npm pack` got a synthetic `E404` and the comparison was + skipped on every run. Driving it for real found two defects in it: `--silent` + suppressed the very error text the function classified on, and npm answers a + missing *version* of an existing package with `ETARGET`, not `E404` — so + `@cipherstash/eql@`, the exact case the function documents, became an + uncaught throw with an empty reason, raised while building the blocker array, + which meant `reportBlockers` never printed the actionable `frozen-publisher` + remedy it had already computed. + **Whether the gate is blocking anything right now is a question for the + registry, not for this file: run `node scripts/release-gate.mjs` and read what + it says.** `tests.yml` runs the same script at PR time so the answer arrives a + merge earlier. `@cipherstash/eql` is the map's ONLY entry: the seven + protect-ffi packages were listed there too, and the cutover that repointed + their publisher at this repo did not take them out — which left the gate + armed against the first release that cutover had just enabled. Delete the + `@cipherstash/eql` entry in the Phase-5 cutover — + `scripts/__tests__/frozen-publisher-docs.test.mjs` fails until this paragraph + goes with it, and `release-gate.test.mjs` now asserts the map carries no FFI + name, so that particular mistake cannot be made twice. + **`SECURITY.md`'s "Note on publishing" is the third document that guard + holds**, added after it was found still naming the seven + `@cipherstash/protect-ffi*` packages as published from `protectjs-ffi` — a + sentence the FFI cutover made false and nothing checked, in the one file that + tells a reporter which pipeline built the artefact they are reporting on. The + guard now fails on a doc that freezes a package the map does not, so the same + drift cannot outlive the next cutover either. + Note too that 3.0.5 did *not* come from `changeset version` — + eleven unrelated changesets were pending, so the bump was entered by hand in + `packages/eql/packages/eql/CHANGELOG.md` and the parked + `rename-ste-vec-contains.md.deferred` deleted with it, precisely so the + cutover cannot apply the same bump twice. Read that CHANGELOG entry before + assuming a version's provenance. +- **Changesets for `@cipherstash/eql` go in the repo-root `.changeset/`.** + `packages/eql/.changeset/` is an empty leftover of the subtree. + ## Agent Skills — these ship to customers `skills/*/SKILL.md` are **published artifacts, not internal notes.** Treat a wrong diff --git a/SECURITY.md b/SECURITY.md index 7cde6b1bf..08d349c34 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -19,6 +19,37 @@ This repository is the CipherStash Stack monorepo for JavaScript/TypeScript. It | `@cipherstash/wizard` | AI-powered encryption setup | | `@cipherstash/protect-ffi` | Native FFI bindings to the CipherStash Client SDK — the Rust core `@cipherstash/stack` encrypts and decrypts through | | `@cipherstash/protect-ffi-darwin-arm64`
`@cipherstash/protect-ffi-darwin-x64`
`@cipherstash/protect-ffi-linux-arm64-gnu`
`@cipherstash/protect-ffi-linux-x64-gnu`
`@cipherstash/protect-ffi-linux-x64-musl`
`@cipherstash/protect-ffi-win32-x64-msvc` | Prebuilt per-platform binaries for `@cipherstash/protect-ffi`. Installed as optional dependencies; one is selected at load time for the host platform | +| `@cipherstash/eql` | Encrypt Query Language — the PostgreSQL SQL bundle (`eql_v3` schema: domains, operators, index-term extractors) that stores and queries encrypted payloads, plus its generated TypeScript types. Applied by `stash eql install` and by the Prisma Next adapter's migrations. Released in lockstep with the `eql-bindings` Rust crate, which emits the payloads this SQL reads | + +This repository also carries the source of the **`eql-bindings`** Rust crate +(`packages/eql/crates/eql-bindings`), published to crates.io and released in +lockstep with `@cipherstash/eql`. It is in scope for security reports on the +same terms as the npm packages above. + +> **Note on publishing.** `@cipherstash/eql` and the `eql-bindings` crate are +> developed here but are still *published* from +> `cipherstash/encrypt-query-language` — the npm package's `repository` / +> `bugs` fields and the crate's `repository` / `homepage` all still name it, as +> does npm trusted publishing, and repointing every one of them is Phase 5 of +> `docs/plans/2026-08-13-eql-monorepo-absorption.md`. Everything else in the +> table above — including all seven `@cipherstash/protect-ffi*` packages, whose +> own cutover has completed — is published from this repository by +> `.github/workflows/release.yml`. Source, issues, and security reports for all +> of them belong here regardless. +> +> The provenance attestation on a release names the repository that built it, +> and is the only thing that settles the paragraph above: +> `curl -s https://registry.npmjs.org/-/npm/v1/attestations/@cipherstash%2fprotect-ffi@0.32.0` +> returns `cipherstash/stack`; the same call against `@cipherstash%2feql@3.0.5` +> returns `cipherstash/encrypt-query-language`. Check there before repeating +> either claim: this note went on naming `cipherstash/protectjs-ffi` as the +> protect-ffi publisher through `0.32.0` — the release that proved the cutover +> and disproved the sentence. +> +> `scripts/__tests__/frozen-publisher-docs.test.mjs` now holds this paragraph to +> `FROZEN_PUBLISHERS` in `scripts/release-gate.mjs`. It fails if the note names +> a package the map does not freeze, and fails again on the Phase-5 cutover that +> empties the map — so the next half of this note to go stale does so loudly. **Security fixes are released for the latest release line of each package.** Security reports are welcome for any version, but fixes land in the latest release — if you are running an older major version, plan to upgrade to receive them. @@ -68,7 +99,7 @@ We will never take legal action against good-faith security researchers who foll The following are **in scope**: - The `cipherstash/stack` GitHub repository -- All npm packages published from this repository (listed under Supported Packages above) +- All npm packages listed under Supported Packages above — scope follows the source, not the release pipeline (see the note on publishing) - CipherStash Stack cryptographic implementations, configuration layers, and CLI tooling - Key-handling, authenticated encryption behaviour, JSON/JSONB field-level encryption flows - Documentation or code examples that could lead to insecure usage diff --git a/biome.json b/biome.json index 63fb55273..4e94394bc 100644 --- a/biome.json +++ b/biome.json @@ -1,5 +1,5 @@ { - "$schema": "https://biomejs.dev/schemas/2.5.2/schema.json", + "$schema": "https://biomejs.dev/schemas/2.5.3/schema.json", "files": { "includes": [ "**", @@ -10,6 +10,12 @@ "!packages/protect-ffi/lib", "!packages/protect-ffi/target", "!packages/protect-ffi/src/eql-v3-types", + "!packages/eql/crates/eql-bindings/bindings", + "!packages/eql/crates/eql-bindings/schema", + "!packages/eql/packages/eql/src/generated", + "!packages/eql/release", + "!packages/eql/target", + "!packages/eql/docs/api", "!**/*.grit", "!**/*.generated.ts", "!**/contract.json", diff --git a/docs/plans/2026-08-04-protect-ffi-monorepo-absorption.md b/docs/plans/2026-08-04-protect-ffi-monorepo-absorption.md index 3aea12a27..5b1645a85 100644 --- a/docs/plans/2026-08-04-protect-ffi-monorepo-absorption.md +++ b/docs/plans/2026-08-04-protect-ffi-monorepo-absorption.md @@ -1873,11 +1873,15 @@ The only irreversible steps. Both halves in one PR — the guard exists to stop that changeset landing early. The `.md.deferred` extension is what makes parking safe: `@changesets/read` and the guard both select on `.endsWith('.md')`, so the file is inert to `changeset version`/`publish` until renamed. Check for more than one parked file — any protect-ffi change landing during the window parks its changeset the same way. - Done in the PR that ticks this box. Two parked files were renamed, not one: `protect-ffi-lazy-load.md` (minor) and `protect-ffi-repository-url.md` (patch). + **Done correctly, in one commit.** `e77bfcec` retired the guard, its self-test, its fixtures, the `lint:ffi-changeset` script and the `tests.yml` step, *and* renamed both parked files — `protect-ffi-lazy-load` (the **minor** this item names) and `protect-ffi-repository-url`. Both released in `@cipherstash/protect-ffi@0.32.0`. + + **The residual hazard is on the other side, and it is not what this item anticipated.** A long-lived branch cut before `e77bfcec` still carries both files under `.md.deferred`, and they survive a merge with the cutover — the branch has a path the base deleted. There, "activate the parked changeset" is exactly wrong: it republishes a changelog entry that has already shipped and re-bumps the package for a change two versions old. The EQL branch hit this and had to *delete* both rather than rename them. Guarded now by `scripts/__tests__/no-parked-changesets.test.mjs`, which fails on the suffix regardless of which resolution is right and points the reader at the released CHANGELOG to decide. Six fixtures under `scripts/__tests__/fixtures/lint-no-ffi-changeset/` were deleted with the self-test. An earlier revision of this line claimed there were none, on the strength of a check against `scripts/fixtures/` — the wrong directory. The self-test used a tmpdir for two cases only (CRLF, and the parked-file case); the rest loaded these on-disk fixtures through ``resolve(…, `../fixtures/lint-no-ffi-changeset/${name}`)``. Deleting the test without them leaves six files no referent points at, and `pnpm run test:scripts` stays green either way — which is exactly why it needed checking rather than inferring. **A third changeset is parked on the branch of #905** (the `jsonwebtoken` CVE bump); once this lands, nothing reads or warns about that extension, so #905 must rename its own file before merging or the fix ships with an empty changelog. + + **And it happened.** #905 merged as `f7abfec4` without renaming its own file, so `.changeset/protect-ffi-jsonwebtoken-cve.md.deferred` is sitting on `main` right now — a `@cipherstash/protect-ffi` **patch** for the `jsonwebtoken` CVE bump (`ec19129a`, CVE-2026-25537) that `@changesets/read` cannot see. The committed version is `0.32.0` and npm's newest is `0.32.0`, so the change it describes is genuinely unreleased and the correct resolution there is `git mv` back to `.md`, not deletion. That fix belongs on `main`: this branch was cut before #905 landed and does not carry the file. `scripts/__tests__/no-parked-changesets.test.mjs` fails on it the moment the two meet. - [ ] Let the Version Packages job create the release PR. Verify it bumps all seven FFI packages to `0.32.0` and rewrites the wrapper's six `optionalDependencies`. **Do not expect a Stack patch.** This criterion originally read "patch-bumps the six Stack packages", which follows from the pinning analysis and is the right prediction for an FFI bump in isolation. It is not what will happen: `.changeset/prisma-next-0-17.md` carries `'@cipherstash/stack-prisma': major`, which propagates through the Stack fixed group, so `changeset status` reports all six Stack packages at **major** — and does so on `origin/main` too, with no FFI changeset in play. The Stack major is unrelated to this phase and must not be read as evidence the FFI bump misbehaved. diff --git a/docs/plans/2026-08-13-eql-monorepo-absorption.md b/docs/plans/2026-08-13-eql-monorepo-absorption.md new file mode 100644 index 000000000..19e30c5d1 --- /dev/null +++ b/docs/plans/2026-08-13-eql-monorepo-absorption.md @@ -0,0 +1,391 @@ +# EQL monorepo absorption — implementation plan + +**Goal:** Build, version and publish the EQL subsystem — the `@cipherstash/eql` npm package, the `eql-bindings` crate, the SQL install bundle, the API docs, the `postgres-eql` Docker image and the dbdev extension — from this repository instead of `cipherstash/encrypt-query-language`, and eliminate the EQL version skew between the Rust that emits payloads and the SQL that stores them. + +**Architecture:** A verbatim-prefix `git subtree` import at `packages/eql/`, so every repo-root-relative path inside EQL keeps resolving. The npm package therefore lands at `packages/eql/packages/eql` and needs its own pnpm-workspace glob — the `packages/protect-ffi/platforms/*` precedent. Publishing moves surface by surface, each inert behind a guard until its own cutover, as `scripts/lint-no-ffi-changeset.mjs` does for the FFI packages today. + +**Tech Stack:** pnpm 10.33.2 workspaces + catalogs, Turborepo 2.10.4, Changesets 2.31, release-plz + crates.io OIDC, Rust/Cargo (a second nested workspace), SQLx + cargo-nextest, mise (a second nested `mise.toml`), Doxygen + Python 3.13, Docker/GHCR, Vitest 3.2.7, Biome 2.5.3. + +--- + +## Global constraints + +Inherited from the protect-ffi absorption, all still binding: + +- **Root `pnpm test` must never invoke cargo.** `turbo test --filter './packages/*'` reaches every package; a cargo process on that path is a Rust toolchain on every contributor's machine. EQL's `codegen:parity` drift gate and the whole SQLx suite are cargo, so they need the same `test:cargo` split protect-ffi has, enforced the same way. +- **Publish workflows must never restore the GitHub Actions cache** (`scripts/lint-no-workflow-caching.mjs`). EQL's own release workflows already respect this — EQL is where that rule came from — but the reusable `_build-sql.yml` / `_build-docs.yml` need re-checking against this repo's target list once they move. +- **npm and crates.io trusted publishing both validate against the entry-point workflow filename**, and both require the manifest's repository field to match the publishing repository exactly. Seven EQL manifests plus `release-plz.toml` name `cipherstash/encrypt-query-language` today. +- **Do not write `pnpm run