Skip to content

feat(kms): add root/app key epochs, SealedEnv and epoch-scoped authorization - #1533

Open
kvinwang wants to merge 162 commits into
nextfrom
feat/kms-epochs-combined
Open

kvinwang wants to merge 162 commits into
nextfrom
feat/kms-epochs-combined

Conversation

@kvinwang

@kvinwang kvinwang commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

Register and rotate root/app key epochs, retain authorized historical keys, migrate disk headers and SealedEnv, and enforce epoch-scoped authorization while preserving temporary-CA compatibility.

This aggregates the linear split stack:
#1516 → #1519 → #1517 → #1520 → #1521 → #1528 → #1530 → #1532.

Restores all original feature commits after the previous stack rewrite omitted application epoch support. Includes the review fixes from #1534: optional selectors, grouped epoch fields, scoped operation names, consistent authorization methods, removal of legacy KMS setters, consolidated app epoch policy interface, and RPC injection/isolation regression coverage. No KMS contract-address equality check was added.

The final tree is identical to #1534 at 2c017875bf. It was independently compared against the original complete #1533/#1534 tree plus the intended review fixes; no unrelated feature changes remain.

Validation on Linux x86_64 with Rust 1.92.0, Bun 1.4.4, Forge 1.8.4 and Solc 0.8.24: cert-client 7, KMS 75, local auth 9, Solidity 87, VMM SealedEnv 4, simple auth 33, mock auth 15, Bun Ethereum auth 50 and Jest Ethereum auth 22 tests passed. Guest-agent and dstack-util compile checks passed, including the earlier guest-key split independently. Solidity used pinned submodules and forge test --offline --ffi -vv; Rust reused Cargo caches. Injection tests use loopback HTTP, a supplied vsock CID, synthetic payloads and temporary VM storage; no live TEE/chain or kernel AF_VSOCK identity test was performed. Tree equality, stack ancestry, Rust formatting and diff whitespace checks passed.

GitHub currently locks the base of #1519 because it is a stack member. Its commits descend from #1516, but its PR base remains next; both GraphQL and REST base updates were rejected.

Describe recovery from KMS root-key exposure using a fleet-wide
maintenance window: per-epoch roots registered on DstackKms, a single
GetAppKey epoch selector (absent = epoch 0, -1 = latest), LUKS2 keyslot
rotation with an epoch token, in-CVM env re-encryption, full-history KMS
onboarding with a rotation mode, and a clean gateway reset.

Signed-off-by: Kevin Wang <wy721@qq.com>
DstackKms keeps an append-only list of root epochs (CA and k256 public
keys, the URL of the attestation proving their generation, and a keccak256
chain hash over all epochs so far) and a separately activated current
epoch. Registration and activation are distinct steps so every KMS instance
can import an epoch before it goes live. A pending epoch can be popped when
the nodes holding it are lost before it is activated. Epoch 0 must be the
kmsInfo keys when those are set.

The attestation stays off chain: its report data binds the keys, so it
needs no on-chain hash, and storing it would cost about 720 gas per byte.

The KMS sends the root epoch it serves and its chain hash as
rootEpoch/rootEpochHash in the boot info; every auth backend (auth-eth,
auth-eth-bun, auth-simple, auth-mock, dstack-auth) denies a KMS that does
not serve the current epoch with the registered keys. A KMS that predates
root epochs reports none and is cut off once a later epoch is current. Info
responses report the current epoch so a KMS can pick it at startup.

dstack-types gains the shared chain hash computation and test vectors.

Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
Evidence updates are recoverable and mismatched evidence fails verification. Revert the unnecessary chain hash requirement for URL updates; retain it for activation and removal.

Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
kvinwang force-pushed the feat/kms-epochs-combined branch 3 times, most recently from 1a7dd23 to a5457bf Compare November 8, 2026 14:14
Signed-off-by: Kevin Wang <wy721@qq.com>
The KMS stores every root epoch it holds (the newest at the existing paths,
older ones under epochs/<n>/) and at startup serves the epoch the auth
backend reports as current, provided its own keys produce the registered
chain hash. Every boot check carries the served epoch and its chain hash so
the backend can deny a stale or forked KMS; on a denial the KMS re-reads the
current epoch and exits to restart onto it if it moved.

GetAppKey takes an optional root_epoch (unset = current) so guests can
fetch the keys of an older epoch to migrate, and responses carry the epoch
served.

Onboarding hands over the whole keyring and checks it against the chain.
With rotate, the new KMS also generates the next epoch and attests it;
the source must then be at the current epoch and pass the strict KMS
authorization check.

Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
fix(auth-eth): refuse legacy authorization once a later root epoch is current
Prove app-epoch monotonicity, the epoch bound on policy writes,
rejection of inactive epochs and foreign operations, legacy
isAppAllowed equivalence at the current epoch, and epoch isolation
of authorization decisions. Mutation testing confirms each property
catches a targeted bug.

Document the FOUNDRY_DYNAMIC_TEST_LINKING=false workaround needed to
run Halmos 0.3.3 with Forge 1.8.

Signed-off-by: Kevin Wang <wy721@qq.com>
Add positive properties: authorizeApp allows exactly when the epoch
policy admits the app, and each per-epoch setter takes effect.

Signed-off-by: Kevin Wang <wy721@qq.com>
Prove the root registry step invariant (append-only entries, chain
hash recursion, hash-bound activation and pop, kmsInfo following the
current epoch), canonical key encodings, authorization epoch bounds
and root epoch reporting, bootstrap-only authorization before
registration, KMS and app authorization matching epoch policy, epoch
isolation of KMS policy writes, and legacy entry points following the
current epochs. Mutation testing confirms each property catches a
targeted bug.

Signed-off-by: Kevin Wang <wy721@qq.com>
…acts

Follow the interface of #1542: AuthorizationRequest and AuthorizationResult
live in IAppEpochAuth.sol, `isAllowed` is `authorized`, and DstackApp's
authorizeApp takes the request.

Update the properties that the review intentionally changed:
- *AtEpoch setters may prepare the next app epoch, and revert past it;
- the legacy isAppAllowed refuses at a nonzero app epoch, and at epoch 0
  decides at epoch 0 even with the next row prepared;
- epoch-less KMS setters write every row from the current epoch to the
  latest one;
- the legacy isKmsAllowed decides at root epoch 0.

Each changed property was mutation-tested against the matching bug.

Signed-off-by: Kevin Wang <wy721@qq.com>
Model the root epoch registry and KMS onboarding, data disk key
migration with crashes, and the VMM env update window, as implemented,
and check them with TLC.

The models find that a crash during luksHeaderRestore can destroy the
volume key, that an initialization interrupted before the epoch token
is written fails every later boot, that lost boot notifications leave
the env update window open to application code, and that an ordinary
join can hand over a losing pending root. A model of a fix for the two
disk issues passes.

Signed-off-by: Kevin Wang <wy721@qq.com>
…tation

DiskMigration now models the two-keyslot migration of #1539 (luksAddKey,
token bound to its keyslot, luksKillSlot) instead of luksHeaderRestore, and
lets the KMS rotate at any point of a boot. All invariants hold for epochs
0-3; the torn-restore script and the configs for the replaced scheme go.

KmsRotation checks the latest registered epoch's chain hash on handover, as
onboard_service.rs does since #1542, so HandoverAgreesWithRegistry now holds
and joins the main config.

EnvUpdate gains FailClosed for the guest of #1546, which holds with a lossy
host API; the lossy config keeps the earlier guest as a counterexample.

The README records the mutations each model catches.

Signed-off-by: Kevin Wang <wy721@qq.com>
Since #1545, isKmsAllowed and isAppAllowed refuse while a later root epoch
is current. Replace the epoch-0 equivalence check, which setUp's rotated
state no longer reaches, with two checks: both entry points refuse after a
rotation, and before any rotation isKmsAllowed decides as kms_serve at
epoch 0 even with the next epoch registered and prepared.

Removing either rotation guard or reading the pending row each produce a
counterexample.

Signed-off-by: Kevin Wang <wy721@qq.com>
test(auth-eth): add Halmos symbolic tests for app and root epochs
docs(formal): add TLA+ models of the key rotation protocol
Only auth-eth-bun (on-chain) and dstack-auth (file-based) remain as
maintained authorization backends; every protocol change had to be
repeated in each one. Nothing in CI or the docs ran auth-mock. For
development, the KMS's built-in `type = "dev"` auth API allows
everything.

Signed-off-by: Kevin Wang <wy721@qq.com>
chore(kms): remove the auth-mock webhook backend
…B checks

Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
…th-dstack-auth

chore(kms): replace auth-simple with dstack-auth
auth-eth-bun serves the same webhook API from the same contracts, so the
Fastify/ethers server in dstack/kms/auth-eth is a second implementation to
keep in sync. Remove it together with its Jest tests, npm manifests and the
integration scripts that only existed to run it.

dstack/kms/auth-eth keeps everything the contracts need: contracts/,
Foundry tests, script/*.s.sol, lib/ submodules, foundry.toml, Slither
config, .openzeppelin/ and the formal-verification docs. The README now
covers only the contracts, plus a short Anvil + auth-eth-bun recipe that
replaces scripts/setup-local-chain.sh.

kms/dstack-app/compose-dev.yaml builds auth-eth-bun from GIT_REPOSITORY at
GIT_REV instead of the Node server, and deploy-to-vmm.sh passes through
ETH_CHAIN_ID and ETH_FINALITY_CONFIRMATIONS. Docs and tutorials that ran
the Node server now run auth-eth-bun; the contract tutorial uses forge.

Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
chore(kms): remove the Node auth-eth webhook server
chore(kms): rename auth-eth-bun to auth-eth
@kvinwang

kvinwang commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator Author

@Leechael Thanks for the thorough review. Below is how each item in the review (against 08b3f262) was handled.

The follow-up PRs from this review (#1539, #1540, #1542, #1543, #1544) are all merged into feat/kms-epochs-combined. "Done" below means merged.

Blocking

Item Decision Severity Where Status
#1 LUKS format/migration atomicity Accepted. Also fixed a permanent boot failure when a rotation follows an interrupted migration Medium #1539 e087e135, 0deb65d1 Done
#2 Contract upgrade and epoch-0 registration not atomic Rejected None — No change
#3 Handover authorization check Mostly rejected: legacy endpoints refuse once a later root epoch is current (#1545). Added audit logging and tests Low #1539 b9139453, ebbd969b, 7161298c; #1540 Done
#4 Epoch-less legacy setters write only the pending row Accepted. They now apply to every row in current..latest High #1539 57b7af82 Done
#5 Ethereum backends do not read finalized blocks Partially accepted: new ETH_BLOCK_TAG, default finalized, overridden per network in each deployment Low #1543 Done
#6 KMS-local high-water mark Rejected None — No change
#7 App epoch cannot be staged ahead Accepted. Staging allowed up to current+1 Low #1539 41e8328b Done
#8 App epoch side lookup fails open Accepted Low #1539 c45bbc7e Done
#9 Allowed when the KMS omits rootEpoch Partially accepted: rejected when current ≠ 0 Medium #1539 3e7296ca Done
#10 Join does not verify the pending epoch Accepted. Only the latest chain hash is verified Medium #1542 Done
#11 DstackApp.isAppAllowed at non-zero epochs Accepted Low #1539 f17d72f2 Done
#12 popRootEpoch leaves stale policy rows Accepted. Policy rows are now keyed by chainHash Low #1539 2aab4c1f Done
#13 Empty list means "any" in dstack-auth Accepted Low #1539 5676e0ff Done
#14 Upgrade reference does not match production Accepted Low #1539 751c2304, 0cf765ce Done
Required removal: legacy cert fallback Accepted, removed Low #1539 d701ed6b Done

Should-fix

Item Decision Severity Where Status
1 SealedEnv pins are a flat set Rejected, documented Low #1539 ba8d16ce Done
2 Trusting ListImages.sealed_env Rejected Low — No change
3 Historical env search continues after a rejection Rejected, design doc corrected None #1539 7ea3e50c Done
4 No read-back check after resealing Rejected, covered by integration tests instead Low Docs #1539 9861dddb; tests in #841 Docs done, tests pending
5 Self-check cache may be stale Cache kept; comments and docs fixed; tests in #841 Low #1539 f81db3cc; tests in #841 Docs done, tests pending
6 File backend has no per-epoch rows Rejected (dev-only backend) None — No change
7 Legacy getters/events cover epoch 0 only Rejected, natspec added None #1539 ea699452 Done
8 Rotate does not require the admin token Rejected, design doc corrected Low #1539 e0fa7e54 Done
9 GetTempCaCert returns temp_ca_key Rejected None — No change
10 v2 report data v2 dropped, v1 only, test vectors added Low #1539 3a6b8772 Done
11 evidenceUrl is mutable Rejected, comment corrected None #1539 07a68bca Done
12 DstackApp.authorizeApp accepts KMS operations Rejected, already handled by existing code None — No change
13 SDK guidance for derived wallets Accepted, docs only Low #1539 bd387fac Done

Crash consistency and resource exhaustion

Item Decision Severity Where Status
1 Core dumps may contain the keyring Rejected here; CVM-wide disabling handled separately None — No change
2 No cleanup on abnormal exit Rejected None — No change
3 Owner env update may be lost Rejected Low — No change
4 Inconsistent env size limits Rejected (pre-existing) Low — No change
5 Unauthenticated request amplification Partially accepted: bounded 1 s cache for /app/:id Medium #1539 f0d2c63e Done
6 guest-agent historical root cache comment Comment corrected None #1539 32a4d09c Done
7 epochs/<n>/ directory entry not persisted Accepted, create_dir_all removed Low #1539 0170966e Done

Changes that came out of the discussion

Change Where Status
Rename isAllowed to authorized, unify the app-level AuthorizationRequest/Result, separate the error body from /bootAuth #1542 Done
Rename the private contract helper _isKmsAllowed to _kmsPolicy (ABI unchanged) #1544 Done
Legacy isKmsAllowed/isAppAllowed refuse once a later root epoch is current #1545 Done
Sealed env write-back window fails closed: it is always closed before application code runs #1546 Done
vmm-cli strips the 0x prefix when comparing pinned KMS signers #1535 Done

Not yet decided

Section Content Proposal
Informational Marked no action No reply
Test coverage 7 minimum test groups Groups 3–6 covered, group 7 rejected. Still missing: handler-level rejection tests for GetAppKey/SignCert (group 1) and SDK unit tests with app_epoch ≠ 0 (group 2), proposed for this PR. The end-to-end part goes to #841
Follow-up issues One-time handover tickets, emergencyHalt/retireRootEpoch, documenting the residual historical GetAppKey risk, residual-risk list Tracked as follow-ups
Follow-up: squash Squash the stack into 8 logical PRs Undecided
Deployment prerequisite Off-chain backend endpoint requirements Document in the release notes

Still pending: integration tests in #841 (not merged yet) for Should-fix 4, Should-fix 5, and the end-to-end part of Test coverage.

The v0.5.9 and v0.6.0 sources under contracts/test-utils/reference are
kept verbatim as upgrade references, so trailing-whitespace and
end-of-file-fixer must not rewrite them.

Signed-off-by: Kevin Wang <wy721@qq.com>
…ontracts

ci(prek): skip whitespace fixers on reference contracts
Root-key handover now lives on the admin listener, so the public
KMS.GetKmsKey entry is only needed while a 0.5.x KMS, or an onboarding
request without source_token, still onboards from the cluster. Default
core.onboard.public_key_handover to false in kms.toml and in the serde
default, and make the refusal name the option to enable.

The onboarding test runbooks onboard without a source token, so they now
set the option on the source KMS explicitly.

Signed-off-by: Kevin Wang <wy721@qq.com>
…r-off

feat(kms)!: turn off the public KMS key handover by default
kvinwang added a commit that referenced this pull request Nov 11, 2026
Since #1533, KMS key release verifies a TDX caller's OS image. The SDK
simulator replays a captured legacy quote whose image (e61be43d...) was
never published, so every simulated KMS client failed at the image
download.

Derive the KMS fixtures' simulator files with the candidate
dstack-tee-simulator instead: it computes the quote measurements from the
lite tdx_measurement document of the verifier fixture, the same way the
verifier does, and the captured RTMR3 events are replayed on top. The
verifier then takes the lite path and downloads nothing. The simulator
runs in a private user and mount namespace because the TDX ABI paths
under /sys are fixed.

Signed-off-by: Kevin Wang <wy721@qq.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants