Repository navigation
Conversation
Every KMS now holds root keys for epochs 0..current_epoch. The current epoch's roots stay in the existing top-level files, so all paths that only use the current roots are unchanged; older epochs live under epochs/<n>/ and a root-epoch marker records the stored epoch. Startup refuses a configured epoch that does not match the stored keys. GetAppKey gains a sint32 key_epoch selector (absent = epoch 0, -1 = current) and reports the resolved epoch, so old OS images keep their keys while new ones can ask for the latest epoch or an older one to migrate. Older epochs are not time-limited: their roots are already exposed, so withholding them only strands late instances. Key handover sends the full keyring tagged by epoch. Onboarding requires exactly epochs 0..N from the source, or with rotate imports 0..N-1 and generates epoch N locally, attesting the new public keys bound to the epoch. Bootstrap only creates epoch 0. Signed-off-by: Kevin Wang <wy721@qq.com>
…yrings DstackKms gains an append-only registry of each epoch's root public keys (registerKeyEpoch, keyEpochCount, keyEpoch). auth-eth and auth-simple list the registered epochs in their info response and report the latest one in every boot decision. The KMS checks at startup that its keyring matches the registered public keys epoch by epoch, and on every authorization that the latest registered epoch is still its own. A node left behind after a newer epoch is registered stops serving keys, certificates, and handovers. Deployments that never register epochs keep working at epoch 0. Signed-off-by: Kevin Wang <wy721@qq.com>
The guest now asks the KMS for the current key epoch and records the served epoch in the KMS key provider. A KMS that predates epochs ignores the selector and serves epoch 0, which is accepted. When the data disk does not unlock with the current key, the epoch recorded in its LUKS2 header (none means 0) selects which historical disk key to request from the same KMS. luksChangeKey replaces the keyslot 0 passphrase in place, keeping the volume key, and a dstack-kms token records the new epoch. The header validator accepts exactly that token. An unrotated epoch-0 disk is left untouched so older OS images can still open it; disks formatted at a later epoch are tagged immediately. Signed-off-by: Kevin Wang <wy721@qq.com>
…p them in When the encrypted env does not decrypt with the current key, boot tries the older epochs from newest to oldest. Once one decrypts it, the env is re-encrypted to the current env public key inside the CVM, checked to round-trip, and recorded with the SHA-256 of the env it came from. GuestApi.GetReencryptedEnv returns that copy; it reaches only the VMM, which talks to the guest over its own vsock, so its origin is the VM. The new Vmm.ApplyReencryptedEnv RPC (vmm-cli apply-reencrypted-env) replaces the stored env with it only if the stored env still matches the recorded hash, so a concurrent update is never overwritten. Signed-off-by: Kevin Wang <wy721@qq.com>
Signed-off-by: Kevin Wang <wy721@qq.com>
…etKey Epoch 1 is now the original root keys and 0 means the current epoch in every selector, so an absent field asks for the latest keys. Before any rotation that is epoch 1, so older OS images see no change; after one they are cut off by the OS allowlist anyway. A peer that predates root epochs reports 0, which is read as epoch 1. The key_epoch fields are renamed root_epoch to leave room for a per-app epoch later. At boot the guest fetches the app keys of every older root epoch once. Disk and env migration reuse them, and their app root keys go into the keys file. Guest API v1 GetKey takes root_epoch (0 = current) and reports the epoch it derived from; the SDKs expose it as an optional argument (Rust get_key_at_root_epoch, Go GetKeyAtRootEpoch). Signed-off-by: Kevin Wang <wy721@qq.com>
registerRootEpoch now takes the attestation the generating KMS produced for the new keys, whose report data binds the epoch and both public keys, and rootEpoch returns it, so anyone can check that a registered epoch came from an approved KMS in a TEE. Add RegisterRootEpoch to Manage.s.sol and list the registered epochs in GetKmsSettings. Signed-off-by: Kevin Wang <wy721@qq.com>
…red one The source KMS was only checked for running an allowed image, so a legitimate image that had bootstrapped its own keys could hand over an unrelated keyring, and a rotation would build the new epoch on top of it. Onboarding now reads the registered root epochs and requires the handed- over pairs to be exactly epochs 1..=N (a join) or 1..=N-1 (a rotation), key for key, which also proves the source is at the expected epoch. Rotating past epoch 1 therefore needs the earlier epochs registered. Deployments that never registered epochs can still join at epoch 1. Signed-off-by: Kevin Wang <wy721@qq.com>
A rotation builds the new epoch on the source's keyring, so the source must be authorized on chain. Refuse to rotate with the dev auth backend, which allows any KMS and has no root epoch registry, and do not accept a source certificate that lacks its OS image hash, which would judge its image by mrAggregated alone. A plain join keeps the existing checks as a safeguard; there the source authorizing the receiver is what matters. Signed-off-by: Kevin Wang <wy721@qq.com>
The dev backend exists for development and testing, where rotating without a registry is useful; production backends still authorize the source and check its keyring against the registry. Signed-off-by: Kevin Wang <wy721@qq.com>
Every webhook backend now lists the registered root epochs in its info response and reports the latest one in boot decisions, like auth-eth and auth-simple. auth-eth-bun reads the DstackKms registry at the same finalized block as the decision and treats a contract without it as having none; dstack-auth reads rootEpochs from its allowlist, which dstack run preserves when it rewrites the file; auth-mock takes them from MOCK_ROOT_EPOCHS. Without this, a KMS past epoch 1 refuses to serve. Signed-off-by: Kevin Wang <wy721@qq.com>
It now always returns the encrypted env the host should store: the copy boot re-encrypted to the current root epoch, or the boot env itself when it needed no change. ApplyReencryptedEnv replaces nothing in that case. Signed-off-by: Kevin Wang <wy721@qq.com>
…ield Epoch 0 is the original root keys, as before root epochs existed, so a peer that predates them reports exactly what it serves. Selectors are now optional: an unset root_epoch in GetAppKey and GetKey means the current epoch, and any number names that epoch. Old OS images, which send no selector, get the current keys; before a rotation those are epoch 0. The registry indexes epochs from 0 and exposes rootEpochCount; auth backends report the latest epoch as the count minus one. Signed-off-by: Kevin Wang <wy721@qq.com>
DstackKms now keeps a chain hash per root epoch, committing to the public keys of that epoch and every earlier one, and separates registering an epoch from making it current: activateRootEpoch moves the current epoch forward one step at a time, and registering epoch 0 makes it current. Auth backends report just the current epoch and its chain hash. The KMS drops its key_rotation config. It serves the current epoch on chain, provided its keyring holds it and reproduces its chain hash, and waits rather than exits until it does. A move of the current epoch makes it restart, so it re-signs its RPC certificate under the new root CA. Onboarding imports the source's whole keyring and checks it by chain hash; a rotation must start from the current epoch and generates the next one. A whole cluster can therefore be prepared at a new epoch while still serving the old one, then switched by one on-chain activation. GetAppKey can return every older epoch's app keys in one response, so a guest makes one request per boot however many epochs exist. Signed-off-by: Kevin Wang <wy721@qq.com>
Drop GetAppKey's include_history and the historical app roots in the keys file. The guest now asks the KMS for one older epoch only when it needs it: the epoch recorded on the data disk, or older epochs one by one until the env decrypts. The guest agent fetches an older app root when GetKey names its epoch, through its existing KMS client, and keeps it in memory. Old roots are no longer resident in every CVM, and a boot with nothing to migrate makes a single request as before. Signed-off-by: Kevin Wang <wy721@qq.com>
The guest now hands the env it re-encrypted to the current root epoch to the VMM through a new host API call, UpdateEncryptedEnv, made on every boot before any application code runs (empty when nothing changed). The VMM records the env each boot starts with and accepts one update per boot, only while the boot has not reached init scripts or containers, and only if the stored env is still the one the VM booted with. The request size is bounded by Rocket's per-method limit, 1 MiB by default under [host_api.limits]. This drops the guest-side re-encrypted env file, GuestApi.GetEncryptedEnv and its VMM proxy, Vmm.ApplyReencryptedEnv, and the vmm-cli command, so no operator step is needed. Signed-off-by: Kevin Wang <wy721@qq.com>
Share one chain-hash check between serving, onboarding, and the per-request epoch check. Collapse the guest agent's borrowed and historical app roots into one owned type. Re-key the data disk on the validated in-memory header copy and restore it to the disk in one step, so the boot never acts on an unvalidated header and the epoch token always matches slot 0. This drops the luksDump and test-passphrase probes. Signed-off-by: Kevin Wang <wy721@qq.com>
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
KMS root-key rotation leaves encrypted environment variables and application data on an older root epoch. Without an in-CVM migration path, a CVM can no longer boot its environment after the active KMS epoch changes, or it must keep using stale ciphertext indefinitely.
Changes
GetEncryptedEnvandUpdateEncryptedEnvprotocol surfaces and update the Rust, Go, Python, and JavaScript clients.This PR is stacked on
kms-getkey-root-epochand includes the follow-up root-epoch implementation needed by the migration flow.Validation
cargo test --manifest-path dstack/Cargo.toml -p dstack-guest-agent -p dstack-util --locked --quiet: 160 + 118 tests passed.cargo test --manifest-path dstack/Cargo.toml -p dstack-kms -p dstack-vmm --locked --quiet: 67 + 220 tests passed, 3 ignored.git diff --checkpassed.Tests ran locally on Linux x86_64 with Rust 1.92.0. No live TEE, production KMS, or external credentials were used.