Update Databricks CLI to v1.20.0

[eng-dev-ecosystem-bot]

Update Databricks CLI to v1.20.0

[github-actions]

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/vscode

Inputs:

• PR number: 2241
• Commit SHA: ab6904a1d612debe8feff2a6de7652f4146b06a5

Checks will be approved automatically on success.

[rugpanov]

Blocking merge: integration tests fail for every e2e job (Linux + Windows) during VSIX packaging, before any test runs.

scripts/setupCLIDependencies.ts:46
SyntaxError: Unexpected token 'D', "Debug info"... is not valid JSON

Cause: CLI v1.20.0 removed databricks bundle debug terraform (databricks/cli#6933), along with the Terraform deployment engine (#6888, #6889). setupCLIDependencies.ts still calls it, gets the bundle debug help text instead of JSON, and fails. Every CLI bump from v1.20.0 on will hit this.

This PR stays blocked until a follow-up PR removes the Terraform bundling step from package-vsix.sh (setupCLIDependencies.ts and the .build CLI fetch). The runtime already handles a missing terraformMetadata. The Terraform-engine warning in BundleEngineManager.ts also needs updating, because engine: terraform is now an error. After the fix merges, rebase this PR and re-run the ITs.

Failed run: https://github.com/databricks-eng/eng-dev-ecosystem/actions/runs/37620470676

[rugpanov]

Executive summary — Databricks CLI v1.19.0 → v1.20.0

⚠️ Breaking: the Terraform deployment engine is removed

• bundle.engine: terraform and DATABRICKS_BUNDLE_ENGINE=terraform now error. A failed migration of existing Terraform state is now an error and no longer falls back to Terraform. Users who must stay on Terraform need CLI v1.19.x. (#6888, #6889)
• The hidden bundle debug terraform command is removed. (#6933)
• bundle.terraform is deprecated, has no effect, and now warns. (#6940)
• bundle deployment migrate no longer fails with "no such file or directory" when the state has no resources, or when the config no longer declares them. (#6958)

Bundles

• New run_as.group_name at bundle and target level, for jobs and pipelines. (#6676)
• bundle run / pipelines run now send the per-update development parameter in development-mode targets. Setting development on a pipeline is deprecated; use mode: development. (#6863)
• Direct engine: explicit zero-value bools and floats added after the first deploy are now detected and applied (#6882). Recreating a secret scope that was deleted outside the bundle now works (#6970).
• Boolean variables accept True/False, and Python bundles accept the same boolean strings as YAML (yes/no, on/off, …). (#6942)

CLI

• aitools install supports Kiro. (#6908)
• databricks api keeps integers larger than 2^53 exact. (#6884)
• apps init:
  • --auth-mode / --set <plugin>.<resourceKey>.authMode=obo|sp|both (#6886)
  • a value is now required for every field that an SP resource binding references (#6903)
  • --package-manager <npm|pnpm> (#6902)
• apps validate / apps deploy pick npm or pnpm from packageManager and lockfiles. (#6892)
• auth docker host no longer reports the credential helper as configured when the helper is missing from PATH. (#6880)
• The CLI warns when its binary was built more than 6 months ago. (#6898)

AI Runtime

• Experimental rank-partitioned container images (#6841)
• code_source snapshot fields can go directly under code_source (#6927)
• AIR priority and UC image fields are mapped when a run config is converted to a bundle (#6905)
• Workspace backend validation in air run --dry-run (#6934)

Dependency updates

• databricks-sdk-go v0.182.0 → v0.185.0. (#6928)

What could be integrated into the VS Code extension experience

0. Blocker for this bump: VSIX packaging calls the removed bundle debug terraform

Required cleanup. Must land with or before this bump.

• scripts/setupCLIDependencies.ts:39-48 runs databricks bundle debug terraform --output json. It then runs JSON.parse on stdout and asserts on dependencies.terraform. v1.20.0 removed that command (#6933), so this step will fail.
• scripts/package-vsix.sh:60-62 calls this step for every arch except win32-arm64. These paths use it:
  • package:all (create-build-artifacts.yml:37)
  • test:integ:prepare → scripts/package-vsix-tests.sh, which the e2e VSIX build uses
• Suggested fix: remove the setupCLIDependencies.ts step from package-vsix.sh, plus the Terraform binary and provider download that it does. This also makes each VSIX smaller.
• The runtime already handles missing terraformMetadata:
  • getCLIDependenciesEnvVars returns {} (src/utils/envVarGenerators.ts:202-205)
  • updateTerraformCliConfig returns early (src/utils/terraformUtils.ts:33)
  • Removing the packaging step is therefore safe before the follow-up cleanup in item 1.
• Please check the "package" and e2e jobs on this PR before merge.

1. Remove the Terraform plumbing from the runtime

Cleanup, low effort, follows item 0. None of this has any effect on v1.20.0:

• src/extension.ts:580-599 puts DATABRICKS_TF_VERSION, DATABRICKS_TF_EXEC_PATH, DATABRICKS_TF_PROVIDER_VERSION and DATABRICKS_TF_CLI_CONFIG_FILE into every terminal's environment collection. It also writes a Terraform CLI config.
• src/utils/envVarGenerators.ts:197,202-216 adds the same variables to every bundle command's environment.
• src/utils/terraformUtils.ts (whole file) and its TerraformUtils export from src/utils/index.ts:8 can go.
• src/cli/CliWrapper.ts:1035-1037: the comment on bundle summary --force-pull gives Terraform state and provider version matching as the reason for the flag. The comment needs new wording, or the flag needs a fresh justification for the direct engine.

2. Change the Terraform-engine deprecation warning (BundleEngineManager)

Cleanup or small UX change on existing plumbing.

• src/bundle/BundleEngineManager.ts:18-21 says Terraform "will stop working in a future Databricks CLI version". With this bump, that version is the bundled one, so the text is now wrong.
• The manager reacts to engine === "terraform" in validate output (BundleEngineManager.ts:74, read at src/bundle/models/BundleValidateModel.ts:93-94). Because bundle.engine: terraform now errors, users will probably get a validate failure, not this warning. That makes the manager mostly dead code.
• Options:
  • (a) Delete the manager, its wiring (src/extension.ts:783), the databricks.bundle.hideTerraformEngineWarning state key (src/vscode-objs/StateStorage.ts:121-128), and Events.BUNDLE_TERRAFORM_ENGINE_WARNING (src/telemetry/constants.ts:22,55,388).
  • (b) Reuse its prompter seam and the "Read migration guide" action (DIRECT_ENGINE_DOCS_URL, BundleEngineManager.ts:11-13). Show it when validate or deploy fails with the new engine-removed or migration-failed error, so users get a migration path and not only raw stderr.
• Option (b) has the most value of anything in this release. Terraform users hit a hard failure on upgrade.

3. Regenerate BundleSchema.ts for run_as.group_name

Flag-flip on existing plumbing (a codegen re-run).

• src/bundle/BundleSchema.ts is generated from the bundled CLI by package:bundle-schema:write (package.json:1962).
• The run_as type BundleSchem6 (BundleSchema.ts:1784-1795) has only service_principal_name and user_name.
• A re-run against v1.20.0 adds group_name (#6676). It may also pick up the bundle.terraform deprecation (#6940) and the development deprecation (#6863).
• The file was last regenerated in Revert "Revert #1857" - Re-apply rename "Databricks Asset Bundles" → "Declarative Automation Bundles" #1864, so expect unrelated drift in the diff.

4. databricks apps init options (--auth-mode, --package-manager, required bindings)

Net-new feature work.

• The extension does not wrap apps init today.
• Apps show only as a resource type in the bundle explorer, and that entry still has a TODO … 'as any' (src/ui/bundle-resource-explorer/types.ts:43).
• A guided "New Databricks App" flow could expose:
  • OBO / SP / both for each resource (#6886)
  • the npm/pnpm choice (#6902)
• Note: #6903 makes the non-interactive apps init fail when a binding value is missing. A wrapper must collect every --set value up front. It cannot rely on a TTY prompt.

No-ops for the extension (transparent)

• Kiro in aitools install (#6908):
  • The agent list is not hard-coded. It comes from aitools list --output json (src/cli/CliWrapper.ts:205-227) and maps to UI rows in src/aitools/AiToolsManager.ts:83-98.
  • Kiro should appear in the AI tools section and in databricks.aitools.installAgent with no code change.
  • The only special-cased agent is Cursor (AiToolsManager.ts:51).
• Pipeline development parameter (#6863):
  • The extension runs plain bundle run --target <t> <key> … (src/cli/CliWrapper.ts:1278-1284), including the --refresh / --full-refresh partial runs (src/ui/bundle-resource-explorer/BundleCommands.ts:287-293).
  • Development-mode targets get the new behavior automatically.
  • The deprecation warning goes to the output channel through stderr (CliWrapper.ts:1012-1013).
• databricks api big-integer fix (#6884): the extension never runs databricks api. Workspace calls go through the SDK.
• Stale-binary warning (#6898):
  • The bundled binary is a fresh release.
  • Users who set databricks.cli.path to an old CLI may now see the warning in output, in addition to the extension's own drift warning (src/cli/CliWrapper.ts:1293).
• Direct-engine fixes (#6882, #6970), deployment migrate fix (#6958), title-case booleans (#6942): these are CLI-side deploy and validate behavior. The extension needs no change.
• apps validate / apps deploy package-manager detection (#6892), auth docker host (#6880), all AI Runtime items (#6841, #6927, #6905, #6934): the extension uses none of these surfaces.
• databricks-sdk-go bump (#6928): internal to the CLI. The extension uses the JS SDK.

_{🤖 Auto-generated executive summary of the CLI v1.19.0 → v1.20.0 changelog. Integration notes are opportunities, not commitments — verify before acting.}