kvinwang · Nov 6, 2026 · Nov 6, 2026 · Nov 6, 2026 · Nov 6, 2026 · Nov 6, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -9,26 +9,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+- KMS: retain `GetTempCaCert`, temporary CA files and legacy handover fields so
+  KMS nodes can be upgraded before existing OS images.
+
 - dstack-util, cert-client: guests mint self-issued RA-TLS client
   certificates for KMS calls instead of fetching the temp CA and minting from
   it; the KMS authenticates the quote inside the certificate, not its issuer.
-  The KMS root CA now comes from `GetMeta`, over the same unauthenticated
-  connection `GetTempCaCert` used, so the trust posture is unchanged: relying
-  parties bind the KMS through the `key-provider` launch event (RTMR3/PCR14)
-  against the on-chain `kmsInfo` or a pinned key, and `verify_key_provider_id`
-  also enforces it in-guest when the app pins `key_provider_id`. A guest built
-  from this change needs KMS 0.6.0 or later; older KMS releases pin the temp CA
-  and refuse the certificate at the TLS handshake
+  Guests fetch the root CA with the key response and bind it to the attested
+  RPC certificate, without a metadata preflight. Provider pins and the
+  `key-provider` launch event (RTMR3/PCR14) retain the root identity binding.
+  The new flow requires an attested KMS that accepts self-issued certificates.
 - dstack-types: `KeyProvider::Kms` drops `tmp_ca_key` and `tmp_ca_cert`, so
   `.appkeys.json` no longer carries them. Files written by older images still
   parse, but out-of-tree readers that require those fields fail on new files
 
 ### Fixed
 
-- kms: onboarding from a source older than 0.6.0 (no `source_token`) now
-  verifies the source's attestation on the connection that returns the root
-  keys, and checks it against local policy before using them. It used to
-  verify the source only on the earlier `GetTempCaCert` connection
+- kms: onboarding with or without `source_token` verifies the source's
+  attestation on the connection that returns the root keys and checks local
+  policy before using them. Legacy sources requiring temp-CA-issued clients
+  must be upgraded.
 
 ## [0.6.3] - 2026-11-01
 

diff --git a/docs/deployment.md b/docs/deployment.md
@@ -131,9 +131,13 @@ Production KMS requires:
 | Custom | Your own authorization logic | Implement webhook interface |
 
 All auth servers implement the same webhook interface:
-- `GET /` - Health check
-- `POST /bootAuth/app` - App boot authorization
-- `POST /bootAuth/kms` - KMS boot authorization
+
+- `GET /` - Health check and current root epoch
+- `POST /authorizeApp` - Application key and certificate authorization
+- `POST /authorizeKms` - KMS self-authorization and key handover
+
+The legacy `/bootAuth/app` and `/bootAuth/kms` routes remain available for older
+clients and KMS bootstrap/onboarding checks.
 
 #### Using auth-simple (Config-Based)
 
@@ -470,9 +474,9 @@ Additional KMS instances can onboard from an existing KMS to share the same root
 **How it works:**
 
 1. New KMS starts in onboard mode (empty `auto_bootstrap_domain`)
-2. New KMS generates an RA-TLS client certificate with its TDX quote: self-issued when given a `source_token`, or minted from the temp CA fetched with `GetTempCaCert` for a source older than 0.6.0
-3. New KMS calls `GetKmsKey` over mTLS: `Admin.GetKmsKey` with the token, or the legacy public `KMS.GetKmsKey` without one
-4. Source KMS verifies attestation via `bootAuth/kms` webhook
+2. New KMS generates an RA-TLS client certificate with its attestation: self-issued with an admin token, or signed by the source temporary CA for tokenless legacy onboarding
+3. New KMS calls `GetKmsKey` over mTLS: `Admin.GetKmsKey` with the token, or the explicitly enabled public `KMS.GetKmsKey` without one
+4. Source KMS verifies the attestation and authorizes handover via `/authorizeKms`
 5. If approved, source KMS returns root keys
 6. New KMS verifies the source's attestation on that same connection and checks it against its own policy before using the keys
 7. Both KMS instances now derive identical keys
@@ -513,6 +517,54 @@ curl http://<new-kms>:9203/finish
 >
 > **Key handover.** It is served as `Admin.GetKmsKey` on the `[core.admin]` listener, which needs `[core.admin.tls]` so it can verify the onboarding KMS's RA-TLS certificate. Pass the source admin URL and its token (`source_token`) in the onboarding request. `core.onboard.public_key_handover = true` keeps the legacy public `KMS.GetKmsKey`.
 
+#### Rotating KMS Root Keys
+
+KMS root keys are versioned by epoch; epoch 0 is the original roots. Every KMS
+keeps all epochs so CVMs can migrate from any of them, and serves the epoch
+governance made current. See the [rotation design](./kms-emergency-key-rotation-protocol.md)
+for the full procedure; the KMS steps are:
+
+1. If epochs were never registered, publish the attestation the KMS produced at
+   bootstrap (kept in `bootstrap-info.json` on the bootstrapped node) and
+   register the current root public keys as epoch 0:
+   `DstackKms.registerRootEpoch(caPubkey, k256Pubkey, evidenceUrl)`, via
+   `script/Manage.s.sol:RegisterRootEpoch`, or a `rootEpochs` entry in the
+   auth-simple config. Epoch 0 is current once registered, and must be the
+   `kmsInfo` keys when those are set. A deployment whose `kmsInfo` holds a
+   legacy `quote` and `eventlog` instead of an attestation converts them with
+   `dstack-kms legacy-attestation --quote <hex> --eventlog <hex>` and publishes
+   the result first: from then on `kmsInfo` follows the current epoch's keys and
+   its quote and event log are cleared.
+2. Onboard a fresh KMS with `"rotate": true` (the **Rotate** checkbox in the
+   onboard UI) from a KMS at the current epoch N-1. It imports epochs 0..N-1,
+   generates epoch N, and returns the new public keys with an attestation whose
+   report data is
+   `keccak256("dstack-kms-generated-keys-v2:{N};{ca_pubkey_hex};{k256_pubkey_hex};")`.
+3. Verify the attestation, publish it, and register epoch N with its URL. The
+   cluster keeps serving N-1.
+4. Onboard the other KMS nodes from the rotated one; they import 0..N and keep
+   serving N-1.
+5. Activate epoch N (`ActivateRootEpoch`, or `currentRootEpoch` for
+   auth-simple). Each KMS restarts onto N on its next uncached authorization
+   check; a KMS that does not hold N waits
+   until it is onboarded again.
+   If the nodes holding N are lost before N is activated, drop it with
+   `PopRootEpoch` (or remove the `rootEpochs` entry) and rotate again; nodes
+   that imported the dropped N must be onboarded again.
+
+Each `authorizeKms` or `authorizeApp` result carries the current epoch and its chain hash,
+including when authorization is denied. KMS verifies these against its keyring;
+there is no cached serving epoch. An optional `requestedRootEpoch` selects historical
+keys, and omission selects the returned current epoch. Requests for future
+epochs are denied. No registered epoch returns a rejection with
+`rejectionReason: "Root epoch not registered"` and a zero hash.
+
+RPC certificates still require a listener restart when their root CA changes;
+callers retry after the certificate is refreshed. Register epoch 0 before normal
+key release. Bootstrap keeps using scoped KMS authorization so it can generate
+the first roots before registration. See the
+[authorization protocol](../dstack/kms/README.md#authorization-protocol).
+
 ---
 
 ## Deploying Apps

diff --git a/docs/onboarding.md b/docs/onboarding.md
@@ -312,3 +312,20 @@ dstack logs <kms-vm-id>
 ```
 
 The KMS VM ID is printed by `dstackup install` when it creates or reuses the KMS CVM.
+
+### Upgrading the local KMS to root epochs
+
+Upgrade the bundled `dstack-auth` together with the KMS. Before normal key release,
+set `kmsContractAddr` in the host's `auth-allowlist.json` to the KMS's attested app ID
+and register the verified bootstrap CA/k256 public keys in `rootEpochs` (epoch zero).
+Use the bootstrap evidence to verify these keys; do not trust an unauthenticated
+metadata response. Set `currentRootEpoch` to the active registered epoch (initially
+zero). See [root registration and rotation](deployment.md#rotating-kms-root-keys).
+The local installer does not register these keys automatically; an older installed
+KMS can continue using the legacy boot routes while this migration is prepared.
+
+The Rust backend implements `/authorizeKms` and `/authorizeApp` with the same
+registered-root requirement as the other backends. Missing registration denies key
+release, and invalid epoch configuration returns HTTP 503. Legacy `/bootAuth/*`
+routes remain available for bootstrap. Local compose/device allowlists apply to
+all permitted historical roots; they are not the on-chain per-epoch matrices.
diff --git a/docs/security/public-security-reports.md b/docs/security/public-security-reports.md
@@ -59,7 +59,7 @@ These issues were filed as concrete vulnerability reports, security audit findin
 | [#616](https://github.com/Dstack-TEE/dstack/issues/616) Host-controlled Docker registry mirror enables image substitution attacks | Closed | Not a production vulnerability | Registry mirrors are untrusted transport. Digest-pinned image references and measured compose configuration protect against substitution. No code fix was applied |
 | [#617](https://github.com/Dstack-TEE/dstack/issues/617) Guest agent exposes raw private keys to all local processes | Closed | Not a production vulnerability | dstack treats a CVM as one application trust domain. It does not provide per-container key isolation inside the same measured application. No code fix was applied |
 | [#618](https://github.com/Dstack-TEE/dstack/issues/618) Disk encryption disableable via kernel cmdline, not measured in RTMR | Closed | Not a production vulnerability | The kernel command line is measured into RTMR2, so changing `dstack.storage_encrypted=false` changes attestation evidence. No code fix was applied |
-| [#619](https://github.com/Dstack-TEE/dstack/issues/619) KMS `get_temp_ca_cert` returns temp CA private key without authentication | Closed | Duplicate | The report duplicates the private advisory response for the temp CA bootstrap flow. [#1106](https://github.com/Dstack-TEE/dstack/pull/1106) removes the reason the temp CA existed - the KMS no longer pins it - so callers can be migrated off it; the RPC itself is retained until guests and KMS-to-KMS onboarding are |
+| [#619](https://github.com/Dstack-TEE/dstack/issues/619) KMS `get_temp_ca_cert` returns temp CA private key without authentication | Closed | Duplicate | The report duplicates the private advisory response for the temp CA bootstrap flow. [#1106](https://github.com/Dstack-TEE/dstack/pull/1106) removes the reason the temp CA existed - the KMS no longer pins it - so callers can be migrated off it; the RPC and temporary CA material remain available for existing OS images and legacy onboarding while new guests use self-issued certificates |
 
 ## Related security roadmap and hardening
 

diff --git a/docs/security/security-best-practices.md b/docs/security/security-best-practices.md
@@ -111,7 +111,7 @@ Development settings are intentionally easy to audit, but they are not productio
 - The KMS contract pins a concrete gateway app id. Do not use `gateway_app_id = "any"` for production traffic.
 - TEE quotes are evaluated by deployment policy, including TCB status and expected OS/application measurements.
 
-The KMS TLS listener verifies client certificates by the attestation they carry rather than by an issuer CA, so it needs no `rpc.tls.mutual` section. It still accepts connections without a client certificate, because bootstrap and public metadata endpoints must be reachable before a client has an RA-TLS certificate. Current guests present self-issued certificates, but `GetTempCaCert` remains in use by older guest images and by KMS-to-KMS onboarding without a `source_token`, which mint their client certificates from that CA; it returns temp CA private material, so treat it as bootstrap-sensitive.
+The KMS TLS listener verifies client certificates by the attestation they carry rather than by an issuer CA, so it needs no `rpc.tls.mutual` section. It still accepts connections without a client certificate, because bootstrap and public metadata endpoints must be reachable before a client has an RA-TLS certificate. Guests and onboarding clients use self-issued certificates. The temporary-CA RPC remains available for old OS images and tokenless legacy onboarding; its public key material authenticates no caller.
 
 App key release and KMS key handover still require verified caller attestation from the RA-TLS client certificate. Certificate signing verifies the CSR signature and embedded attestation before signing.
 

diff --git a/docs/security/security-model.md b/docs/security/security-model.md
@@ -425,7 +425,7 @@ The KMS Rocket TLS listener permits connections without a client certificate bec
 
 App key release and KMS key handover require verified caller attestation from the RA-TLS client certificate. Certificate signing verifies the CSR signature and the attestation embedded in the CSR before signing.
 
-The unauthenticated or non-client-certificate surface includes bootstrap and temp-CA bootstrap material retrieval, env-encryption public-key retrieval, metadata, health, and metrics behavior documented for operators. `GetTempCaCert` returns temp CA private material and remains in use by older guest images and by KMS-to-KMS onboarding without a `source_token`, which mint their client certificates from that CA; operators must treat it as bootstrap-sensitive rather than harmless public metadata.
+The unauthenticated or non-client-certificate surface includes bootstrap, env-encryption public-key retrieval, metadata, health, and metrics behavior documented for operators. Clients create self-issued certificates locally. The temporary-CA RPC and its public key material remain available for legacy clients. Guest key retrieval verifies the actual response connection's attestation and checks that the returned CA signed its RPC certificate. Provider pins and launch measurement continue to bind the selected KMS root identity; attestation alone does not authorize an arbitrary CVM as the expected KMS.
 
 KMS-to-KMS root-key handover is served by `Admin.GetKmsKey` on the authenticated
 admin listener, which additionally requires a verified RA-TLS client certificate.

diff --git a/docs/tutorials/kms-cvm-deployment.md b/docs/tutorials/kms-cvm-deployment.md
@@ -73,8 +73,6 @@ Inside the CVM at `/etc/kms/certs/`:
 | `root-ca.key` | Root CA signing key (P256 ECDSA) |
 | `rpc.crt` | TLS certificate for RPC server |
 | `rpc.key` | RPC server private key |
-| `tmp-ca.crt` | Temp CA, served by the legacy `GetTempCaCert` bootstrap RPC |
-| `tmp-ca.key` | Temp CA private key, served alongside it |
 | `root-k256.key` | Ethereum signing key (secp256k1) |
 | `bootstrap-info.json` | Public keys and TDX attestation quote |
 

diff --git a/docs/tutorials/troubleshooting-kms-deployment.md b/docs/tutorials/troubleshooting-kms-deployment.md
@@ -303,7 +303,7 @@ sudo systemctl restart qgsd
 
 ### GetMeta Returns "Connection refused" on Port 9200
 
-**Symptom:** KMS responds to `GetTempCaCert` but GetMeta returns:
+**Symptom:** The KMS health endpoint responds but GetMeta returns:
 ```json
 {"error": "error sending request for url (http://127.0.0.1:9200/): ...Connection refused (os error 111)"}
 ```

diff --git a/dstack/Cargo.lock b/dstack/Cargo.lock
diff --git a/dstack/cert-client/Cargo.toml b/dstack/cert-client/Cargo.toml
@@ -20,10 +20,12 @@ tdx-attest.workspace = true
 dstack-guest-agent-rpc.workspace = true
 http-client = { workspace = true, features = ["prpc"] }
 tokio.workspace = true
+x509-parser = { workspace = true, features = ["verify"] }
 
 [dev-dependencies]
+ra-rpc = { workspace = true, features = ["rocket", "client"] }
+mock-attestation = { path = "../crates/mock-attestation" }
 hex.workspace = true
 rocket.workspace = true
 tempfile.workspace = true
-x509-parser.workspace = true
 tokio = { workspace = true, features = ["macros", "rt", "rt-multi-thread", "sync"] }