Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Changed

- KMS: `core.onboard.public_key_handover` defaults to `false`, so
`KMS.GetKmsKey` is no longer served on the public port. Onboard through the
source's admin listener with `source_token`; a source that still onboards
0.5.x KMS nodes must set the option to `true` until they are upgraded.
- KMS: onboarding refuses a source whose certificate carries no OS image hash,
which drops the fallback for KMS 0.5.x sources. Join only from a 0.6+ source;
upgrade a 0.5.x cluster to 0.6.x first.
Expand Down
2 changes: 1 addition & 1 deletion docs/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -533,7 +533,7 @@ curl http://<new-kms>:9203/finish

> **Admin authentication.** The KMS *admin* RPCs (for example `ClearImageCache`) are served on a dedicated `[core.admin]` listener behind the shared HTTP authenticator, just like the VMM and gateway: set `[core.admin] enabled = true` with an `auth_token` (or the `DSTACK_KMS_ADMIN_TOKEN` / `ADMIN_API_TOKEN` env vars), and clients send `Authorization: Bearer <token>` or `X-Admin-Token`. Enabled with neither `auth_token` nor `htpasswd_file` (and `insecure_no_auth = false`) fails closed — the KMS refuses to start. See [dstack-kms admin authentication](../dstack/kms/README.md#admin-api-authentication).
>
> **Key handover.** It is served as `Admin.GetKmsKey` on the `[core.admin]` listener, which needs `[core.admin.tls]` so it can verify the onboarding KMS's RA-TLS certificate. Pass the source admin URL and its token (`source_token`) in the onboarding request. `core.onboard.public_key_handover = true` keeps the legacy public `KMS.GetKmsKey`.
> **Key handover.** It is served as `Admin.GetKmsKey` on the `[core.admin]` listener, which needs `[core.admin.tls]` so it can verify the onboarding KMS's RA-TLS certificate. Pass the source admin URL and its token (`source_token`) in the onboarding request. The legacy public `KMS.GetKmsKey` is off by default; set `core.onboard.public_key_handover = true` on the source only when an onboarding KMS cannot use the admin listener, such as a 0.5.x KMS or an onboarding request without `source_token`.

#### Rotating KMS Root Keys

Expand Down
3 changes: 2 additions & 1 deletion docs/security/security-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -429,7 +429,8 @@ The unauthenticated or non-client-certificate surface includes bootstrap, env-en

KMS-to-KMS root-key handover is served by `Admin.GetKmsKey` on the authenticated
admin listener, which additionally requires a verified RA-TLS client certificate.
The legacy public `KMS.GetKmsKey` is controlled by `core.onboard.public_key_handover`.
The legacy public `KMS.GetKmsKey` is off by default and is served only when
`core.onboard.public_key_handover = true` is set explicitly.

## Limitations

Expand Down
6 changes: 4 additions & 2 deletions docs/upgrading-to-0.6.md
Original file line number Diff line number Diff line change
Expand Up @@ -232,8 +232,10 @@ Configuration changes:
- Key release on AMD SEV-SNP, AWS NitroTPM, and AWS Nitro Enclaves stays off
until you set `sev_snp_key_release`, `aws_nitro_tpm_key_release`, or
`nitro_enclave_key_release`.
- Keep `core.onboard.public_key_handover = true` while any KMS still onboards
through the public port, and turn it off afterwards.
- `core.onboard.public_key_handover` now defaults to `false`, which removes
`KMS.GetKmsKey` from the public port. While a 0.5.x KMS still onboards from
this cluster, it can only use that public entry, so set the option to `true`
on the source node, and remove it once every node is upgraded.
- Auto-bootstrap refuses to overwrite existing root keys.

If you use `auth-simple`, replace it with `dstack-auth`, which reads the same
Expand Down
5 changes: 3 additions & 2 deletions dstack/kms/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,9 @@ below, which decides whether a CVM may boot and receive keys.

KMS-to-KMS onboarding calls `Admin.GetKmsKey` with a self-issued RA-TLS client
certificate plus the source's admin bearer token; other admin RPCs need no client
certificate. `core.onboard.public_key_handover = true` (the default) keeps
`KMS.GetKmsKey` on the public listener for older clients.
certificate. `core.onboard.public_key_handover` is off by default; set it to
`true` only to also serve `KMS.GetKmsKey` on the public listener for older
clients.

Three settings relax authorization, each over a different scope:

Expand Down
6 changes: 3 additions & 3 deletions dstack/kms/kms.toml
Original file line number Diff line number Diff line change
Expand Up @@ -127,8 +127,8 @@ gateway_app_id = "any"
[core.onboard]
enabled = true
auto_bootstrap_domain = ""
# Keep KMS.GetKmsKey on the public RPC listener for older onboarding clients.
# New deployments should point onboarding at the admin listener instead.
public_key_handover = true
# Also serve KMS.GetKmsKey on the public RPC listener, for onboarding clients
# that cannot reach the admin listener (KMS 0.5.x, or no source admin token).
public_key_handover = false
address = "0.0.0.0"
port = 8000
8 changes: 6 additions & 2 deletions dstack/kms/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -241,8 +241,8 @@ pub(crate) struct RootKeyPaths {
pub(crate) struct OnboardConfig {
pub enabled: bool,
pub auto_bootstrap_domain: String,
/// Keep GetKmsKey available on the public RPC listener for compatibility.
#[serde(default = "default_true")]
/// Also serve GetKmsKey on the public RPC listener, for legacy onboarding clients.
#[serde(default)]
pub public_key_handover: bool,
}

Expand All @@ -264,6 +264,10 @@ mod tests {
"default admin token must be empty (fail-closed)"
);
assert!(!config.admin.insecure_no_auth);
assert!(
!config.onboard.public_key_handover,
"public key handover must be off by default"
);
}

#[test]
Expand Down
5 changes: 4 additions & 1 deletion dstack/kms/src/main_service.rs
Original file line number Diff line number Diff line change
Expand Up @@ -934,7 +934,10 @@ impl KmsRpc for RpcHandler {

async fn get_kms_key(self, request: GetKmsKeyRequest) -> Result<KmsKeyResponse> {
if !self.state.config.onboard.public_key_handover {
bail!("public KMS key handover is disabled; use the admin listener");
bail!(
"public KMS key handover is disabled; onboard through the admin listener with \
the source admin token, or set core.onboard.public_key_handover = true"
);
}
self.handover_keys(KeyRelease::RpcHandover, request).await
}
Expand Down
3 changes: 2 additions & 1 deletion dstack/kms/src/www/onboard.html
Original file line number Diff line number Diff line change
Expand Up @@ -259,7 +259,8 @@ <h2>Onboard from an Existing KMS Instance</h2>
<div class="form-group">
<label for="source_token">Source admin token (optional):</label>
<input type="password" id="source_token" v-model="sourceToken"
autocomplete="off">
autocomplete="off"
placeholder="Without it, the source needs core.onboard.public_key_handover = true">
</div>
<div class="form-group">
<label>
Expand Down
3 changes: 3 additions & 0 deletions dstack/tests/docs/kms-bootstrap-onboard.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,9 @@ Deploy both KMS instances in onboard mode with:
- `core.onboard.auto_bootstrap_domain = ""`
- `core.auth_api.type = "webhook"`

`kms-dst` onboards through the public `KMS.GetKmsKey` without a `source_token`,
so `kms-src` also needs `core.onboard.public_key_handover = true`.

Record:

```bash
Expand Down
3 changes: 3 additions & 0 deletions dstack/tests/docs/kms-self-authorization.md
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,9 @@ Requirements for **both** VMs:
- `core.onboard.auto_bootstrap_domain = ""`
- `core.auth_api.type = "webhook"`

`kms-dst` onboards through the public `KMS.GetKmsKey` without a `source_token`,
so `kms-src` also needs `core.onboard.public_key_handover = true`.

Point them at different auth services. If using host-local `dstack-auth` with QEMU user-mode networking:

- `kms-src` → `http://10.0.2.2:3101`
Expand Down