experimental/bundletest: base for pytest-style DABs isolation testing (DuckDB local backend) - #6597
Draft
Sankalp-Mittal wants to merge 20 commits into
Draft
Sankalp-Mittal wants to merge 20 commits into
Sankalp-Mittal wants to merge 20 commits into
Conversation
Base package layout for a pytest-style isolation testing framework for DABs bundles: pyproject (hatchling, src layout, pytest dev extra), README describing the seed-and-run isolation model and the two-backend design, and an empty package module. Backends and the test API land in follow-up commits on this branch. Co-authored-by: Isaac <no-reply@databricks.com>
Define the Backend protocol and RunResult dataclass. The protocol is the single abstraction with more than one implementation: deploy/teardown, seed_table (stand in for an upstream resource), run_job (run one resource), execute_sql + table_schema (the data plane the assertion layer builds on), get_resource, and put_file. Co-authored-by: Isaac <no-reply@databricks.com>
BundleEnv plus TableHandle/ColumnHandle/JobHandle/VolumeHandle, all written once against the backend seam. bundle_env() is a contextmanager that builds a backend (selected by BUNDLETEST_BACKEND, default "memory"), deploys, yields and tears down. The pytest plugin registers the cloud_only marker and skips those tests with a visible reason on any non-cloud backend. The cloud backend factory branch raises NotImplementedError until its follow-up PR lands. Co-authored-by: Isaac <no-reply@databricks.com>
sqlite-backed implementation of the Backend protocol. It is a real engine, not a mock: seeded rows are stored, run_job executes the job's registered logic (from the bundle's transforms.py JOBS dict) against them, and execute_sql/table_schema query the real output. Dotted table refs are rewritten to flat sqlite names, anchored to FROM/JOIN/INTO/UPDATE/TABLE so numeric literals are left alone. Failing jobs surface as RunResult(FAILED) rather than crashing, so bad-data tests can assert on state. Co-authored-by: Isaac <no-reply@databricks.com>
An orders_bundle example whose transform_orders job dedupes and null-filters bronze into silver, with an isolation test that seeds bronze, runs the real job, and asserts on silver (row count, no-nulls, uniqueness). A cloud_only schema/type assertion demonstrates the loud-skip guard. Framework tests cover the assertion layer, the numeric-literal rewrite edge case, unknown-job KeyError, and the failed-run path. Co-authored-by: Isaac <no-reply@databricks.com>
Commit the uv lockfile so `uv run pytest` resolves the same pytest version for everyone working on the base. Co-authored-by: Isaac <no-reply@databricks.com>
…he real SQL
A design review found the sqlite + Python-callable backend tested a
reimplementation of the job, not the deployed artifact — so its headline
catch ("wrong table name") could pass locally and still fail on deploy, and
sqlite's lax typing risked silent false greens.
Replace it with a DuckDB backend that runs the job's actual sql_task artifact
(read from databricks.yml). Invariants: same source of truth (real .sql, never
a reimplementation); names bound at the environment level via ATTACH/CREATE
SCHEMA so the query body is never rewritten; three-way routing via the new
LocalUnsupported signal — missing table/column is red, while a Databricks-only
function, a notebook/Python task, or a reserved catalog skips loudly through a
pytest hookwrapper. Backend kind renamed "memory" -> "local".
Co-authored-by: Isaac <no-reply@databricks.com>
Replace the orders_bundle's Python transforms.py callable with the real deployed artifact: src/transform_orders.sql, referenced by the job's sql_task in databricks.yml. The isolation test seeds shop.bronze.raw_orders, runs the job's real SQL, and asserts on shop.silver.orders. Uses a non-reserved catalog so the local backend can host it. Co-authored-by: Isaac <no-reply@databricks.com>
Framework tests now run on the DuckDB backend and pin the routing contract: a missing table in the real artifact fails red (the headline catch), while a notebook task, a Databricks-only function, and a reserved catalog each skip via LocalUnsupported. Also covers env-level name binding (two- and three-part names) and that numeric literals are not misread as table refs. README updated to describe the local (DuckDB) backend and the honesty invariants. Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
changed the title
Collaborator
Integration test reportCommit: 58b20e9
Top 3 slowest tests (at least 2 minutes):
|
One example file per resource/capability supported today, each with a precise CAN / CANNOT-test-locally header, so the folder doubles as a record of capability as the framework grows. Covers: SQL job run+assert (test_job_sql), resource config/wiring read-back (test_job_config), the non-SQL job loud-skip boundary (test_job_nonsql, via a new notebook job in the example bundle), and the volume upload stub (test_volume). Shared env fixture moved to conftest.py. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Formatting-only churn on files not otherwise touched this change. Co-authored-by: Isaac <no-reply@databricks.com>
Volume upload was a no-op stub. Give the local backend a real volume: put_file copies the source into a temp volume filesystem, and read_volume_file reads it back via DuckDB (read_csv/json/parquet). A new FileHandle exposes exists(), row_count(), and columns, so env.volume(name).file(...) can be asserted on. The volume example now uploads a CSV and checks its rows/columns; the non-SQL job example now asserts the LocalUnsupported guard fires rather than just skipping. Every example test now checks something real. Co-authored-by: Isaac <no-reply@databricks.com>
Add a second SQL job (aggregate_orders: silver -> gold) and a test_pipeline_end_to_end that chains bronze -> silver -> gold in one test, asserting the intermediate and final tables. Demonstrates that an all-SQL pipeline runs hop by hop locally: the first job's output persists in the env's shared DuckDB connection for the next to read. The test sequences run_job calls itself (no automatic dependency ordering yet). Co-authored-by: Isaac <no-reply@databricks.com>
The lockfile is regenerated behind the internal PyPI proxy, which bakes pypi-proxy.cloud.databricks.com URLs into every entry and trips the check-lockfiles guard. Gitignore it for this experimental package. Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
added this pull request to stack #6604
September 10, 2026 13:58
…transform The cloud_only test asserts silver.orders.total_price is decimal(10,2), but the transform never cast it, so the pipeline produced double end-to-end — the assertion would fail on the cloud backend. Cast total_price to DECIMAL(10,2) in transform_orders.sql so the pipeline genuinely produces decimal; the assertion is now truthful on cloud and stays cloud_only (Databricks 'decimal(10,2)' vs DuckDB 'DECIMAL(10,2)' spelling). Surfaced by the cloud-backend work. Co-authored-by: Isaac <no-reply@databricks.com>
A failed job run now raises JobRunFailed (carrying the RunResult) at the handle layer, so a wired-up-wrong job cannot slip past unnoticed. Tests that deliberately run a failing job opt out with check=False and inspect the result. Backends still just return RunResult, so both tiers get this for free through the seam. Co-authored-by: Isaac <no-reply@databricks.com>
Nothing ran the bundletest suite before. Add an independent 'bundletest' job to the python build workflow that runs 'uv run --extra dev pytest' in experimental/bundletest. Scoped to that dir; not folded into ./task test. uv.lock is gitignored (internal-proxy URLs), so the runner resolves fresh against pypi.org. Co-authored-by: Isaac <no-reply@databricks.com>
A minimal bundle whose one SQL job runs unchanged on both backends, selected by BUNDLETEST_BACKEND. The single test's assertions are backend-agnostic (portable SQL semantics only); the cloud tier, which needs per-run namespace isolation, lands with the cloud backend PR and is scoped out here rather than shipped with a colliding fixed-namespace fixture. Demonstrates the same-test-two-tiers claim instead of only asserting it. Co-authored-by: Isaac <no-reply@databricks.com>
…esting Reword the local-backend bullet so a green local run reads as 'the SQL is portable and structurally sound,' not 'this passes on Databricks.' DuckDB is not a Databricks SQL emulator; dialect-dependent checks belong on cloud. Wording only. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Base scaffolding for
bundletest— an experimental Python library for pytest-style isolation testing of DABs bundles. It tests a single deployed resource in isolation: stand in for the resource's upstream neighbors withenv.seed(...), run that one resource's real deployed SQL, and assert on its real output.This is the base of a stacked-PR series. It ships the scaffolding + a zero-infra local backend; the cloud (deco) backend is the next stacked PR on top.
Why
A bundle can pass 100% of its unit tests and still be broken on deploy — wrong table referenced, a resource wired to the wrong upstream, a transform regression. Those only surface after
databricks bundle deploy+ a real run. A cloud-only harness would just be automated UAT; the value here is fast local feedback that catches the "validates but doesn't run" bug class.Design
backend.py): aBackendprotocol is the only abstraction with more than one implementation. Every assertion is derived above it fromexecute_sql/table_schemaand never branches on backend type.env.seed("shop.bronze.raw_orders", ...)stands in for the upstream ingest job; the realtransform_ordersSQL runs. Faking the output of the thing under test is a tautology and is not supported.BUNDLETEST_BACKEND:local(default) andcloud(follow-up PR; the factory currently raisesNotImplementedError).The local backend (DuckDB) stays honest
An earlier iteration used sqlite + a hand-written Python callable per job. A design review showed that tested a reimplementation, not the deployed artifact — so the headline "wrong table name" catch could pass locally and still fail on deploy. Replaced with a DuckDB backend that:
sql_taskartifact (read fromdatabricks.yml) — same source of truth.catalog.schema.tablenames viaATTACH/CREATE SCHEMA, so the unmodified SQL resolves.main/temp/system) can't be judged locally →LocalUnsupported→ the test skips with a reason (via a pytest hookwrapper).@pytest.mark.cloud_onlyfences known-cloud-only assertions the same way.Layout
Verification
Notable tests:
test_wrong_table_name_fails_red(headline catch against the real artifact), and notebook-task / Databricks-function / reserved-catalog all skip viaLocalUnsupported.Reviewing this stack
Commits are small and ordered: scaffold → seam → shared API → (sqlite backend, then its replacement by DuckDB) → example → tests. The sqlite→DuckDB pivot is preserved as its own commit so the design change is legible.
This pull request and its description were written by Isaac.