ci: bootstrap branch-specific prebuilt CI images by BrendanWalsh · Pull Request #2529 · microsoft/SynapseML

Brendan Walsh (BrendanWalsh) · 2026-03-27T05:28:32Z

Summary

Moves repeated CI environment setup into an immutable, content-addressed image published at mcr.microsoft.com/mmlspark/build-demo under collision-resistant ci-* tags. Trusted runs bootstrap missing images through the existing SynapseML Build ARM service connection, while every consumer—including fork PRs—pulls the resulting image publicly without registry credentials.

How it works

Derives the image tag from every effective image input and validates both pipeline tag locations.
Derives Java, Spark, Spark checksum, PyTorch, and torchvision values from the checked-out branch.
Reuses an existing public MCR tag immediately; otherwise builds or reuses the matching public/mmlspark/build-demo ACR manifest.
Waits for MCR to expose the exact ACR manifest digest before dependent jobs can start.
Keeps registry credentials out of consumer and fork jobs; fork guards reject image-input changes and verify the public tag exists.
Preserves shared sbt caches, preloads the checksummed test dataset, and uses the image-provided Spark distribution for R tests.

Supported branches

The same implementation was replayed against the current Spark 4.0 and Spark 4.1 branch tips. Their image tags and runtime arguments are derived independently, preserving Java 17 and each branch's Spark, Python, and CPU PyTorch versions. Shared release branches still receive this change by merging master, not by rebasing.

Exact-head fixes

Exports sparklyr's resolved SPARK_HOME after fallback archive installation, resolving the current-head suppressed review finding.
Compiles Java sources with --release 8; Databricks failures were caused by NativeLoader.class major version 55 on Java 8 clusters. A regression now proves major version 52 when built on JDK 11 and JDK 17.
Replaces the valid CMYK OpenCV fixture with deterministic malformed Spark image metadata, so decoding-error behavior no longer depends on the host decoder.
Refreshes the immutable image tag to ci-2a90442c0282 after the build input change.

Validation

Rebased onto master at 7c72c49f6fd7f8064ea19662add399d222545b85 (16 commits ahead, 0 behind).
Public MCR bootstrap and anonymous consumption were proven in Azure build 233104290; ACR and MCR served the same digest for ci-112db6614927.
Databricks final-retry outputs for all eight failed notebooks identified the same Java 11 bytecode versus Java 8 runtime incompatibility; the installed SynapseML jar/wheel was present.
Scala 2.12/JDK 11: core/OpenCV scalastyle, test scalastyle, compile, test compile, and both focused regressions passed; NativeLoader.class is major version 52.
Scala 2.13/JDK 17: Spark 4.0 and Spark 4.1 bytecode regressions passed with major version 52; prior OpenCV and pipeline helper validation also passed on both branches.
CI helper/pipeline tests: 90 passed. Black 22.3.0: 203 files unchanged.
The sparklyr 1.8.1 fallback was exercised with a temporary Spark-shaped archive and exported the extracted Spark home.

The remaining gate is the Azure Pipelines run and automated review on exact head ec73e2dff8837f89f996dabec0e38d97760ae5c9.

github-actions · 2026-03-27T05:28:44Z

Hey Brendan Walsh (@BrendanWalsh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

fix: Fix LightGBM crashes with empty partitions
feat: Make HTTP on Spark back-offs configurable
docs: Update Spark Serving usage
build: Add codecov support
perf: improve LightGBM memory usage
refactor: make python code generation rely on classes
style: Remove nulls from CNTKModel
test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

github-actions · 2026-03-27T05:28:46Z

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

Copilot

Pull request overview

Containerizes most Azure DevOps CI jobs to run inside a pre-built Docker image, aiming to reduce end-to-end CI time by pre-baking toolchains, caches, and datasets.

Changes:

Add a new CI Docker image definition and a pipeline job to build/retag it using a content-hash tag.
Move major CI jobs (style/tests/publish) to run with container: ci, add disk cleanup, and scope compilation for unit tests.
Add dataset cache support in build.sbt and introduce a global ScalaTest per-test timeout in TestBase.

Reviewed changes

Copilot reviewed 5 out of 7 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
tools/docker/ci/Dockerfile	Defines the CI container image (JDK8, conda env, Spark, SBT warmup, datasets).
templates/free_disk.yml	Adds a reusable host disk cleanup step for container jobs.
pipeline.yaml	Introduces `ci` container resource, BuildCIImage job, and migrates many jobs to containers/scoped compilation.
build.sbt	Uses `DATASET_CACHE` to avoid re-downloading test datasets in CI.
core/src/test/.../TestBase.scala	Wraps all tests with a global `failAfter` timeout.
project/CodegenPlugin.scala	Removes redundant `LocalRootProject` `publishLocal` from `installPipPackage`.
.gitignore	Ignores `pipeline.yaml.bak`.

Brendan Walsh (BrendanWalsh) · 2026-03-27T19:25:53Z

/azp run

azure-pipelines · 2026-03-27T19:26:07Z

Azure Pipelines successfully started running 1 pipeline(s).

Codecov Comments Bot (codecov-commenter) · 2026-03-27T19:45:57Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.93%. Comparing base (0a2b0b4) to head (4d6cdc8).
⚠️ Report is 12 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2529      +/-   ##
==========================================
+ Coverage   84.66%   84.93%   +0.27%     
==========================================
  Files         335      335              
  Lines       17747    17747              
  Branches     1595     1595              
==========================================
+ Hits        15025    15073      +48     
+ Misses       2722     2674      -48

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Brendan Walsh (BrendanWalsh) · 2026-04-02T07:58:12Z

/azp run

azure-pipelines · 2026-04-02T07:58:23Z

Azure Pipelines successfully started running 1 pipeline(s).

Brendan Walsh (BrendanWalsh) · 2026-04-02T08:20:28Z

/azp run

azure-pipelines · 2026-04-02T08:20:40Z

Azure Pipelines successfully started running 1 pipeline(s).

Brendan Walsh (BrendanWalsh) · 2026-04-02T08:44:42Z

/azp run

azure-pipelines · 2026-04-02T08:44:54Z

Azure Pipelines successfully started running 1 pipeline(s).

Brendan Walsh (BrendanWalsh) · 2026-04-02T09:39:51Z

/azp run

azure-pipelines · 2026-04-02T09:40:03Z

Azure Pipelines successfully started running 1 pipeline(s).

## Summary Rebase PR #2529 onto current master, preserve master pipeline updates, add the CI container image flow, and fix blocking CI image publication and timeout defects. ## Prompting Intent Engineer asked to rebase microsoft/SynapseML PR #2529 onto current master in a new local worktree without pushing or commenting, resolve pipeline.yaml conflicts while preserving master-side changes, verify and fix genuine blockers around untrusted image publication, mutable image consumption, and UnitTests timeout regression, validate the result, and prepare one local commit. ## Linked Sources - GitHub PR: #2529 - Upstream branch: brwals/containerize-ci - Rebase target: microsoft/SynapseML master at ffe123a - Local validation: yaml.safe_load plus duplicate-key detector; sbt scalastyle Test/scalastyle; sbt core/Test/compile ## Rationale Kept the Java 8 CI image because current SynapseML CI sbt logs use Java 8 and ReleaseBranchCompat is the only job that explicitly switches JDKs. Restricted image build/push to trusted non-PR master builds so fork and same-repo PRs cannot publish a poisoned shared image. Pinned downstream jobs to a content-hash tag and added a hash guard so a given run consumes the intended image instead of racing on ci-latest. Raised UnitTests to 100 minutes so its 90-minute test task plus setup can complete, while preserving current master matrix additions and timeouts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rebase PR #2529 onto current master, preserve master pipeline updates, add the CI container image flow, and fix blocking CI image publication and timeout defects. Engineer asked to rebase microsoft/SynapseML PR #2529 onto current master in a new local worktree without pushing or commenting, resolve pipeline.yaml conflicts while preserving master-side changes, verify and fix genuine blockers around untrusted image publication, mutable image consumption, and UnitTests timeout regression, validate the result, and prepare one local commit. - GitHub PR: #2529 - Upstream branch: brwals/containerize-ci - Rebase target: microsoft/SynapseML master at ffe123a - Local validation: yaml.safe_load plus duplicate-key detector; sbt scalastyle Test/scalastyle; sbt core/Test/compile Kept the Java 8 CI image because current SynapseML CI sbt logs use Java 8 and ReleaseBranchCompat is the only job that explicitly switches JDKs. Restricted image build/push to trusted non-PR master builds so fork and same-repo PRs cannot publish a poisoned shared image. Pinned downstream jobs to a content-hash tag and added a hash guard so a given run consumes the intended image instead of racing on ci-latest. Raised UnitTests to 100 minutes so its 90-minute test task plus setup can complete, while preserving current master matrix additions and timeouts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The new global failAfter in TestBase relied on ScalaTest's default signaler, DoNotSignal, which does not touch the running thread. It only reports a timeout *after* the test body returns on its own - so a genuinely hung test still hangs forever, which is the exact failure mode this timeout was added to contain. Verified with a probe suite (testTimeoutInSeconds = 3, body sleeping 12s): before: PROBE_ELAPSED_MS=12001 (ran to completion, then reported failed) after: PROBE_ELAPSED_MS=3004 (interrupted at the deadline) Declaring ThreadSignaler as the implicit signaler makes the deadline interrupt the test thread. Tests finishing inside the limit are unaffected, since the signaler only fires at the deadline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Fix verified CI containerization review findings by removing world-writable R library permissions, hashing the full Docker build inputs, validating CI image tag consistency during PR builds, preserving sbt cache/succeeded() pipeline invariants, restoring Python test retries, and removing an unused Spark download from R tests. Engineer asked to audit suppressed Copilot review findings for GitHub PR #2529, empirically verify each against current code and the PR diff, fix only real issues, treat tools/ci/tests/test_pipeline_yaml.py as authoritative for pipeline invariants, validate pipeline YAML parsing and available pipeline tests, and commit locally without pushing. - GitHub PR: #2529 - Azure Pipelines container resource documentation: https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/resources-containers-container - Azure Pipelines Sprint 212 release notes: https://learn.microsoft.com/en-us/azure/devops/release-notes/2022/sprint-212-update#template-expressions-in-container-resource-definition Used UID 1001 ownership instead of mode 777 because Azure Pipelines commonly runs container jobs as that user and world-writable site libraries are unnecessary. Kept the container image literal in resources because Azure documentation only guarantees template expressions for selected container properties, not image, so a cheap validation step is safer than risking pipeline validation failure. Made BuildCIImage a successful no-op outside trusted master builds so containerized jobs can keep succeeded() dependency gating while still building/pushing only from master. Widened hashing to all files copied from project/ so tags remain content-addressed for the actual Docker context, restored sbt cache and pinned black checks required by pipeline invariants, restored removed retry behavior, and deleted the redundant Spark tarball download because Spark is already pre-baked into the CI image. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Corrects the pipeline-invariant regressions this branch introduced, and updates two invariants that containerization legitimately changes. Pipeline fixes (real bugs in this branch): - Restored `succeeded()` gating on every job that fans out from BuildAndCacheSbt/BuildCIImage. The branch had weakened these to `not(failed()), not(canceled())` to work around BuildCIImage being skipped off master; `not(failed())` is also true when a dependency is *skipped*, so jobs could run on stale/absent impact-detection output. BuildCIImage now always runs and gates its expensive Docker steps on master instead, so the job succeeds as a no-op on PRs. - Restored `templates/sbt_cache.yml` on the containerized sbt jobs that had lost it, so they keep hitting the prewarmed bootstrap cache. Test-invariant updates (containerization genuinely invalidates these): - Publish no longer requires `templates/conda.yml`. Inside `container: ci` the synapseml env is baked into the image, while conda.yml resolves $(CONDA_CACHE_DIR) to the hosted-agent path (/usr/share/miniconda/envs) rather than the image path (/opt/conda/envs) and runs `conda env remove` + `conda env create` -- which would delete the prebaked env. The assertion now applies only to a non-containerized job. - Style no longer needs an inline `pip install black[jupyter]==22.3.0`. environment.yml already pins black==22.3.0 exactly and the image bakes it in, so the test asserts the pin at its authoritative source for a containerized job instead of adding a network install to every run. Also fixed the sbt-job detector in test_pipeline_yaml.py, which used a bare `"sbt " in text` substring check and therefore treated any job hashing `build.sbt sonatype.sbt ...` as an sbt-running job. It now ignores filenames ending in `.sbt`. Verified the tightened pattern changes detection for BuildCIImage only -- all nine genuine sbt jobs are still detected. ## Prompting Intent Engineer proved empirically that four `test_pipeline_yaml.py` failures were caused by this branch rather than pre-existing (swapping in master's pipeline.yaml under the same test file yields zero failures), and asked for each to be root-caused and fixed -- either by correcting pipeline.yaml or, where containerization genuinely invalidates an invariant, by updating the test with justification. Explicitly instructed not to paper over a real regression by loosening a test. ## Linked Sources - Pull request under audit: #2529 - Suppressed Copilot review comments on the same PR ## Rationale Preferred fixing pipeline.yaml wherever the invariant still held, since the gating and cache-warming invariants are about correctness and cost and survive containerization untouched. Took the test-update path only for the two conda-related invariants, where satisfying the test literally would have been actively harmful: re-adding conda.yml to a containerized job risks destroying the prebaked environment it exists to provide. Fixed the `sbt ` detector at its root instead of reformatting the YAML to dodge the substring match, so the workaround does not silently rot. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Fixes the Python Style Check failure on this branch. The assertion added in the previous commit ran to 92 characters, over black's 88-column limit, so `black --check` flagged the file and the GitHub Actions style job failed: would reformat tools/ci/tests/test_pipeline_yaml.py 1 file would be reformatted, 188 files would be left unchanged. Purely a line wrap - no change to the assertion's logic. `black --check --extend-exclude 'docs/' .` now reports 189 files unchanged, and the suite still passes. ## Prompting Intent Engineer asked to review CI status on the open PRs and fix failures. This failure was self-inflicted by my own earlier edit to this file, so it is in scope for this branch rather than a separate PR. ## Linked Sources - Pull request: #2529 - Failing job: https://github.com/microsoft/SynapseML/actions/runs/31730493501/job/94549468954 ## Rationale Ran black rather than hand-wrapping so the result matches exactly what CI computes. Used the repo-pinned 22.3.0 (environment.yml, and the same version the workflow installs) - a newer black produces different output and would reintroduce the failure. Note for reviewers: `test_release_compat_replays_prerequisite_before_pr_patch` is intermittently flaky when run from a Windows drive mounted into WSL, failing with `fatal: stash failed`. It creates its scratch repo under `target/`, and on v9fs the coarse mtime granularity trips git's racy-index detection, so `git merge` attempts an autostash that has nothing to stash. Measured with identical content: 3/12 failures on v9fs versus 0/12 on ext4. Pre-existing (the test arrives via #2604) and environment-specific, so it is untouched here. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Adds a comment above `resources.containers` recording the two conditions that must hold before this pipeline can run at all: 1. A Docker Registry service connection named exactly `SynapseML MCR`, pointing at `mmlsparkmcr.azurecr.io` and authorized for this pipeline. 2. The referenced image tag must already exist in the registry. Both fail during resource validation, before any job starts, so neither appears as a job failure in the timeline - the build just reports a validation error with no logs to inspect. This branch is currently blocked on exactly that, so the note is worth having next to the code that causes it. Engineer asked to review suppressed Copilot review comments on every PR and fix anything real. Copilot suggested making this prerequisite discoverable inline at pipeline.yaml:5. - Pull request: #2529 - Copilot review (suppressed comments): #2529 (review) Documented the tag's provenance alongside the service connection because the two failures look identical from the build UI but have different fixes. The tag is content addressed and published only by BuildCIImage on master, so any change to environment.yml, build.sbt, sonatype.sbt, project/, or the CI Dockerfile requires a master build to publish the new image before a PR can consume it. That ordering is not obvious from the literal tag string. The other suppressed comment on this PR, that the allowlist-style .dockerignore re-includes `project/` and `docs/` without their contents, was checked and is not correct, so nothing changed there. Built an image against a fixture using this exact .dockerignore: `project/*.sbt`, `project/build.properties`, `project/build/Nested.scala`, `docs/index.md` and `docs/sub/deep.md` were all present in the context, including the nested paths, while an unlisted `secret.txt` was correctly excluded. Comment-only change. Verified pipeline.yaml still parses and the resource block is unchanged; pipeline invariant suite passes 17/17. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Removes the Node.js 16 + yarn layer from `tools/docker/ci/Dockerfile`. Nothing that runs in this image uses it. The layer was labelled "for website deployment", but website deployment runs in `.github/workflows/website-deploy.yml` on a GitHub-hosted runner, not in this ADO container. Checked every consumer: none of the nine containerized jobs (Style, Publish, DatabricksCPUE2E, DatabricksGPUE2E, FabricE2E, PythonTests, RTests, WebsiteSamplesTests, UnitTests) reference node, npm, or yarn; neither do `templates/`, `project/`, or build.sbt. `WebsiteSamplesTests` despite its name runs an sbt suite. The only `yarn` hits in Scala are Spark's YARN cluster manager. Also removes the one `curl ... | bash -` in the image, which was the remaining unpinned remote-code-execution step in the build. Every other fetch is HTTPS and version-pinned (Adoptium via GPG keyring, Miniconda py311_24.11.1-0, Spark 3.5.0, sbt 1.10.11, a specific libssl1.1 .deb). The image tag is content addressed over environment.yml, build.sbt, sonatype.sbt, project/, and this Dockerfile, so it is recomputed: `ci-9d15e6d9bfde` -> `ci-060baf1aac44`, updated in both the `resources.containers` image reference and the `CI_IMAGE_TAG` variable that the two hash-validation steps compare against. Engineer asked to review unresolved and suppressed Copilot review comments on every PR and fix anything real. Copilot flagged the `curl | bash` NodeSource install at tools/docker/ci/Dockerfile:61 as an unpinned supply-chain risk. - Pull request: #2529 - Review thread: tools/docker/ci/Dockerfile:61 Removed the dependency rather than hardening the install. The reviewer's suggested fix was to pin the NodeSource repo via a keyring, mirroring the Adoptium block already in this file. That would have been reasonable, but verifying the consumers showed nothing needs Node at all, and deleting an unused layer removes the risk outright while shrinking the image. Adding a keyring for software nobody runs would be strictly worse. Note the same `curl | bash` NodeSource pattern exists at tools/helm/zeppelin/Dockerfile:76. That is a pre-existing image unrelated to CI, so it is out of scope here and left for a separate change. Verified: recomputed hash matches both pinned locations (image reference and CI_IMAGE_TAG agree, so the pipeline's own tag-validation steps pass); `docker build --check` reports no warnings; pipeline invariant suite passes 17/17. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Harden the container image and pipeline against UID-remapped cache failures, stale image tags, duplicate packaging, and premature E2E timeouts. Bring GitHub PR #2529 to merge-ready confidence by verifying every inline review finding against current code, fixing only live defects, preserving deterministic containerized CI, and validating Docker, YAML, Python, and Scala behavior without pushing or changing the PR. - GitHub PR: #2529 - Inline review findings: GitHub PR #2529 review threads Use ownership by the hosted agent UID with non-world-writable modes rather than 777 or root-owned 755. Hash only tracked build inputs and enforce the tag in a regression test. Override the global test timeout only for E2E suites whose documented inner limits exceed ten minutes, and avoid rebuilding core twice in its own Python matrix leg. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The tag is documented as content addressed, but .dockerignore selects the docker build context and so can change the built image without changing any hashed input. This PR edits .dockerignore itself, so a later change to it would have silently reused a stale image. Hash it alongside the other build inputs in both tag computations and in the test helper. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Containerizing Publish left behind an apt-get install of graphviz and doxygen that cannot succeed in the image: the Dockerfile already installs both and then removes /var/lib/apt/lists, so apt cannot resolve them, and the job no longer runs as root. Drop the step, add a test so no containerized job reintroduces one, and record .dockerignore in the content-addressed tag comment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The CI image bakes in a dataset tarball that feeds tests, but fetched it without checking its contents, unlike the libssl deb a few lines above. Pin and verify its sha256 so a changed or truncated blob fails the image build loudly instead of silently altering test inputs. Digest confirmed stable across two independent downloads; the size in the comment was also wrong (117MB, not 50MB). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The image fetched Miniconda, Spark and sbt over the network and used them unchecked; the Miniconda one is piped straight into bash as root. Pin each to the digest its own vendor publishes, so a corrupted or substituted artifact fails the image build instead of silently becoming the toolchain every CI job runs on. Each digest was confirmed against the upstream publisher, not just computed locally: Miniconda - matches the sha256 on repo.anaconda.com/miniconda/ Spark - matches Apache's published spark-3.5.0-bin-hadoop3.tgz.sha512, which also confirms the blob mirror is byte-identical upstream sbt - matches the sbt-1.10.11.tgz.sha256 asset on the GitHub release Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Summary Make Azure CI self-bootstrap an immutable branch-specific image in the existing ACR. Derive Java, Spark, and PyTorch inputs from each branch, protect fork builds, preserve dataset and sbt caches, and use the bundled Spark distribution for R tests. ## Prompting Intent Revive and rebase SynapseML PR #2529, remove its stale registry-service blocker, and make containerized CI dependable and easy to maintain across the supported Spark branches. ## Linked Sources - Pull request: #2529 - SynapseML release guide: https://msdata.visualstudio.com/A365/_wiki/wikis/Osmos%20Team%20Wiki/130638/SynapseML-Fabric-Release-Guide-v2 - Azure Pipelines container schema: https://learn.microsoft.com/azure/devops/pipelines/yaml-schema/resources-containers-container ## Rationale Use content-addressed tags and the authorized SynapseML Build ARM connection instead of the nonexistent Docker registry endpoint. Build missing images only on trusted runs, reject unpublishable fork input changes explicitly, derive runtime arguments from branch-owned files, and retain local fallbacks plus focused regressions so master, Spark 4.0, and Spark 4.1 remain portable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rana Singh (ranadeepsingh) · 2026-08-27T13:33:49Z

/azp run

Copilot

Review details

Files reviewed: 16/17 changed files
Comments generated: 2
Review effort level: Lite

+if (!nzchar(Sys.getenv("SPARK_HOME", ""))) {
+  spark_install_tar(paste(getwd(), "/../../../../../../spark-3.5.0-bin-hadoop3.tgz", sep = ""))
+}


+- job: BuildCIImage
+  displayName: 'Publish missing trusted CI image'
+  cancelTimeoutInMinutes: 0
+  timeoutInMinutes: 90
+  condition: or(ne(variables.isPR, true), ne(variables['System.PullRequest.IsFork'], 'True'))


azure-pipelines · 2026-08-27T13:34:06Z

Azure Pipelines: Successfully started running 1 pipeline(s).

## Summary Use Azure Pipelines' documented lowercase `acr` resource discriminator so the first-class ACR resolver can derive authenticated private-image access from the existing `SynapseML Build` ARM service connection. Enforce the case-sensitive contract in the pipeline regression test. ## Prompting Intent Revive #2529 on current master and make its branch-specific prebuilt CI image usable in trusted, fork, and Spark-port validation without introducing static registry credentials. ## Linked Sources - Pull request: #2529 - Failed Azure validation: https://dev.azure.com/msdata/A365/_build/results?buildId=233088781 - ACR resource documentation: https://learn.microsoft.com/azure/devops/pipelines/process/resources?view=azure-devops#azure-container-registry-resource-type - Container job authentication documentation: https://learn.microsoft.com/azure/devops/pipelines/process/container-phases?view=azure-devops#service-endpoints ## Rationale The prior uppercase value parsed as YAML but left the agent's registry endpoint empty, causing an anonymous pull from private ACR. The current first-class resource documentation consistently uses lowercase `acr`; preserving the ARM-backed resource avoids a separate standing Docker Registry credential and keeps the existing trusted publisher/fork guard design intact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rana Singh (ranadeepsingh) · 2026-08-27T14:36:03Z

/azp run

azure-pipelines · 2026-08-27T14:36:20Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot

Review details

Suppressed comments (1)

tools/tests/run_r_tests.R:8

When SPARK_HOME is unset, this script installs Spark via spark_install_tar(...) but never updates SPARK_HOME. The generated R tests now read Sys.getenv("SPARK_HOME") for spark_connect(...), so local runs can still see an empty Spark home even after the install. Capture the installed path and export it via Sys.setenv() so the subsequent tests reliably pick it up.

if (!nzchar(Sys.getenv("SPARK_HOME", ""))) {
  spark_install_tar(paste(getwd(), "/../../../../../../spark-3.5.0-bin-hadoop3.tgz", sep = ""))
}

Files reviewed: 16/17 changed files
Comments generated: 0 new
Review effort level: Lite

## Summary Publish immutable ci-* manifests through the existing public/mmlspark/build-demo ACR-to-MCR mapping and run container jobs from the anonymously readable MCR mirror. Wait for the public manifest digest to match ACR before releasing dependent jobs, and verify public image availability for fork pull requests. ## Prompting Intent Revive #2529 on current master and make branch-specific prebuilt CI images reliable for trusted pull requests, forks, and Spark release branches without adding static registry credentials. ## Linked Sources - Pull request: #2529 - Private-pull failure: https://dev.azure.com/msdata/A365/_build/results?buildId=233096609 - Azure container-job authentication: https://learn.microsoft.com/azure/devops/pipelines/process/container-phases?view=azure-devops#service-endpoints - Public image manifest: https://mcr.microsoft.com/v2/mmlspark/build-demo/manifests/ci-112db6614927 ## Rationale Azure container jobs require a Docker Registry service connection for private pulls, and A365 has no such endpoint or permission for this identity to create one. The existing MCR mapping already publishes SynapseML build images, and a server-side copy proved that the CI manifest appears publicly with the exact ACR digest. Reusing that mapping avoids registry-wide anonymous access, static credentials, and manual container wrappers while retaining immutable branch-specific tags and fork compatibility. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rana Singh (ranadeepsingh) · 2026-08-27T15:38:36Z

/azp run

azure-pipelines · 2026-08-27T15:39:03Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot

Review details

Suppressed comments (1)

tools/tests/run_r_tests.R:8

RTestGen now generates spark_connect(..., spark_home = Sys.getenv("SPARK_HOME"), ...), but when SPARK_HOME is unset this runner only calls spark_install_tar(...) and never sets SPARK_HOME. That makes local/non-container runs likely to pass an empty spark_home into spark_connect, causing R tests to fail even though Spark was installed.

if (!nzchar(Sys.getenv("SPARK_HOME", ""))) {
  spark_install_tar(paste(getwd(), "/../../../../../../spark-3.5.0-bin-hadoop3.tgz", sep = ""))
}

Files reviewed: 16/17 changed files
Comments generated: 0 new
Review effort level: Lite

## Summary Pin javac output to Java 8 bytecode, add a class-version regression, export SPARK_HOME after sparklyr fallback installation, and make the OpenCV decoding-error fixture deterministic. Refresh the immutable CI-image tag after the build input change. ## Prompting Intent Revive PR #2529 on the current target, resolve every current-head review and exact-head Azure failure, and prove the prebuilt CI path remains usable across Spark 3.5, 4.0, and 4.1. ## Linked Sources - Pull request: #2529 - Exact-head automated review: #2529 (review) - Exact-head Azure build: https://dev.azure.com/msdata/A365/_build/results?buildId=233104290 ## Rationale Targeting Java 8 bytecode preserves Databricks runtime compatibility without downgrading the Java 11 and 17 build environments. A malformed Spark image row is deterministic across decoder implementations, unlike the valid CMYK fixture. Explicitly setting sparklyr's resolved Spark home makes the archive fallback usable by generated R tests rather than merely extracting it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rana Singh (ranadeepsingh) · 2026-08-27T18:48:52Z

ec73e2d fixes every actionable failure from Azure build 233104290 and the current-head suppressed review:

sparklyr fallback now exports and validates the installed SPARK_HOME;
OpenCV decoding-error coverage uses deterministic malformed image metadata;
Databricks run outputs showed all eight failures shared NativeLoader class-file version 55 on Java 8 runtimes, so javac now uses --release 8 and a regression proves major version 52 on JDK 11, Spark 4.0/JDK 17, and Spark 4.1/JDK 17.

Local checks are green: targeted Scala style/compile/tests, 90 pipeline helper tests, and Black 22.3.0 across 203 files.

Rana Singh (ranadeepsingh) · 2026-08-27T18:48:53Z

/azp run

azure-pipelines · 2026-08-27T18:49:08Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Copilot AI lite review requested due to automatic review settings March 27, 2026 05:28

Copilot started reviewing on behalf of Brendan Walsh (BrendanWalsh) March 27, 2026 05:29 View session

Copilot AI reviewed Mar 27, 2026

View reviewed changes

Brendan Walsh (BrendanWalsh) force-pushed the brwals/containerize-ci branch from 6f18643 to bbb23c6 Compare March 27, 2026 19:12

Brendan Walsh (BrendanWalsh) force-pushed the brwals/containerize-ci branch 4 times, most recently from aec4efd to 6678299 Compare April 2, 2026 07:49

Brendan Walsh (BrendanWalsh) force-pushed the brwals/containerize-ci branch 3 times, most recently from 369043d to be65e36 Compare April 2, 2026 08:16

Brendan Walsh (BrendanWalsh) force-pushed the brwals/containerize-ci branch from be65e36 to 61b4e8f Compare April 2, 2026 08:39

Brendan Walsh (BrendanWalsh) force-pushed the brwals/containerize-ci branch from 61b4e8f to 4d6cdc8 Compare April 2, 2026 09:32

Copilot AI review requested due to automatic review settings August 13, 2026 02:38

Rana Singh (ranadeepsingh) force-pushed the brwals/containerize-ci branch from 4d6cdc8 to 5f0e7b5 Compare August 13, 2026 02:38

Rana Singh (ranadeepsingh) self-requested a review as a code owner August 13, 2026 02:38

Brendan Walsh (BrendanWalsh) and others added 13 commits August 27, 2026 02:53

Copilot AI review requested due to automatic review settings August 27, 2026 13:26

Rana Singh (ranadeepsingh) force-pushed the brwals/containerize-ci branch from 8c82618 to cd345c1 Compare August 27, 2026 13:26

Copilot AI reviewed Aug 27, 2026

View reviewed changes

Uh oh!

Conversation

Brendan Walsh (BrendanWalsh) commented Mar 27, 2026 • edited by ranadeepsingh Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How it works

Supported branches

Exact-head fixes

Validation

Uh oh!

github-actions Bot commented Mar 27, 2026

Uh oh!

github-actions Bot commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependency Review

Scanned Files

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Brendan Walsh (BrendanWalsh) commented Mar 27, 2026

Uh oh!

azure-pipelines Bot commented Mar 27, 2026

Uh oh!

Codecov Comments Bot (codecov-commenter) commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Brendan Walsh (BrendanWalsh) commented Apr 2, 2026

Uh oh!

azure-pipelines Bot commented Apr 2, 2026

Uh oh!

Brendan Walsh (BrendanWalsh) commented Apr 2, 2026

Uh oh!

azure-pipelines Bot commented Apr 2, 2026

Uh oh!

Brendan Walsh (BrendanWalsh) commented Apr 2, 2026

Uh oh!

azure-pipelines Bot commented Apr 2, 2026

Uh oh!

Brendan Walsh (BrendanWalsh) commented Apr 2, 2026

Uh oh!

azure-pipelines Bot commented Apr 2, 2026

Uh oh!

Rana Singh (ranadeepsingh) commented Aug 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

azure-pipelines Bot commented Aug 27, 2026

Uh oh!

Rana Singh (ranadeepsingh) commented Aug 27, 2026

Uh oh!

azure-pipelines Bot commented Aug 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Rana Singh (ranadeepsingh) commented Aug 27, 2026

Uh oh!

azure-pipelines Bot commented Aug 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Rana Singh (ranadeepsingh) commented Aug 27, 2026

Brendan Walsh (BrendanWalsh) commented Mar 27, 2026 •

edited by ranadeepsingh

Loading

github-actions Bot commented Mar 27, 2026 •

edited

Loading

Codecov Comments Bot (codecov-commenter) commented Mar 27, 2026 •

edited

Loading