Skip to content

ci: test OCP 4.12 XPN workaround for Terraform SA race - #83406

Draft
davdhacs wants to merge 2 commits into
openshift:mainfrom
stackrox:davdhacs/test-ocp4-xpn-workaround
Draft

ci: test OCP 4.12 XPN workaround for Terraform SA race#83406
davdhacs wants to merge 2 commits into
openshift:mainfrom
stackrox:davdhacs/test-ocp4-xpn-workaround

Conversation

@davdhacs

@davdhacs davdhacs commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

DO NOT MERGE — rehearsal test only.

Tests the automation-flavors XPN workaround (stackrox/automation-flavors#362) that bypasses the terraform-provider-google service account race condition (OCPBUGS-42014) on OCP 4.12.

Changes:

  • Use openshift-4-latest image tag (contains the XPN workaround + bootstrap SA cache warming)
  • Set SKIP_TERRAFORM_SA_CREATION=true for all 4.12 jobs

The workaround:

  1. Creates a pre-existing VPC with subnets + Cloud Router + Cloud NAT
  2. Sets networkProjectID = projectID with TechPreview to trigger the installer's XPN code path
  3. The installer passes the credential's client_email to Terraform, skipping master/worker SA creation (count=0)
  4. Pre-creates and deletes the bootstrap SA to warm GCP's IAM cache

Result from simulate testing: 0 Terraform SA failures, first-attempt success, 31 min provisioning (vs ~30% failure rate and 90-min timeouts previously).

Summary by CodeRabbit

StackRox OpenShift 4.12 CI jobs now use openshift-4-latest and set SKIP_TERRAFORM_SA_CREATION="true". The OCP 4 e2e workflow passes this setting to test the Terraform Google provider service-account race-condition workaround in XPN environments. This configuration is for rehearsal testing only and must not be merged.

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 13, 2026
@openshift-ci

openshift-ci Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@coderabbitai

coderabbitai Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 28eee43c-94f9-4ebe-b105-c92f3d632be0

📥 Commits

Reviewing files that changed from the base of the PR and between c8d353d and 689b66b.

📒 Files selected for processing (1)
  • ci-operator/config/stackrox/stackrox/stackrox-stackrox-master__ocp-4-12.yaml
🚧 Files skipped from review as they are similar to previous changes (1)
  • ci-operator/config/stackrox/stackrox/stackrox-stackrox-master__ocp-4-12.yaml

Walkthrough

The StackRox OCP 4.12 CI configuration now uses the openshift-4-latest base image. The OCP 4 E2E workflow defines SKIP_TERRAFORM_SA_CREATION, and all listed E2E jobs set it to "true".

Changes

StackRox CI configuration

Layer / File(s) Summary
E2E workflow flag
ci-operator/step-registry/stackrox/automation-flavors/ocp-4-e2e/stackrox-automation-flavors-ocp-4-e2e-workflow.yaml
The OCP 4 E2E workflow adds SKIP_TERRAFORM_SA_CREATION with an empty default value.
Base image and job environment updates
ci-operator/config/stackrox/stackrox/stackrox-stackrox-master__ocp-4-12.yaml
The base image tag changes to openshift-4-latest. The listed E2E environments set SKIP_TERRAFORM_SA_CREATION to "true".

Estimated code review effort: 1 (Trivial) | ~3 minutes

Mergeability Score: ⚪ Minimal · up to 689b6

This PR updates OCP 4.12 CI configuration to exercise the Terraform service-account workaround; no actionable merge-blocking risk is identified at the current head, subject to the repository's do-not-merge rehearsal controls.

Possibly related PRs

Suggested labels: rehearsals-ack

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the CI test for the OCP 4.12 XPN workaround targeting the Terraform service-account race.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The diff changes only CI YAML image and environment settings; it adds no Ginkgo declarations or test titles, so this check is not applicable.
Test Structure And Quality ✅ Passed The diff changes only two YAML CI configuration files; it adds CI environment settings and an image tag, with no Ginkgo It blocks or test-code changes to assess.
Microshift Test Compatibility ✅ Passed The PR diff changes only CI YAML configuration and adds no new Ginkgo e2e tests, so MicroShift test compatibility checks do not apply.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The PR changes only two CI YAML files; the aggregate diff adds no Ginkgo tests or test source, so SNO compatibility checks do not apply.
Topology-Aware Scheduling Compatibility ✅ Passed The diff changes only CI image and environment variables; it adds no deployment, controller, affinity, topology-spread, replica, node-selector, toleration, or PDB constraints.
Ote Binary Stdout Contract ✅ Passed The PR diff changes only two YAML configurations: an image tag and environment variables. It adds no OTE binary or process-level stdout code.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The full PR range changes only two YAML CI configuration files; it adds no Ginkgo test declarations or test code, so this IPv4/disconnected test check is not applicable.
No-Weak-Crypto ✅ Passed The diff changes only YAML image and environment settings; added lines contain no MD5, SHA1, DES, RC4, Blowfish, ECB, crypto code, or secret comparisons.
Container-Privileges ✅ Passed The diff only changes an image tag and adds SKIP_TERRAFORM_SA_CREATION; no changed or referenced YAML contains privileged, host namespaces, SYS_ADMIN, root, or allowPrivilegeEscalation settings.
No-Sensitive-Data-In-Logs ✅ Passed The PR adds only an image tag and environment declarations. It adds no logging command or sensitive value; existing echo and credential references are unchanged.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 13, 2026
@davdhacs

Copy link
Copy Markdown
Contributor Author

/test all

Temporary change to test the automation-flavors XPN workaround that
bypasses the terraform-provider-google service account race condition.

Changes:
- Use openshift-4-latest image tag (contains the XPN workaround)
- Set SKIP_TERRAFORM_SA_CREATION=true for all 4.12 jobs

This PR is for rehearsal testing only — do not merge.
@davdhacs
davdhacs force-pushed the davdhacs/test-ocp4-xpn-workaround branch from 1ccd624 to c8d353d Compare August 13, 2026 20:47
@coderabbitai

coderabbitai Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@openshift-ci

openshift-ci Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: davdhacs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@davdhacs

Copy link
Copy Markdown
Contributor Author

/test all

Use double quotes and alphabetical order for SKIP_TERRAFORM_SA_CREATION
to match the output of `make ci-operator-config`.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@davdhacs

Copy link
Copy Markdown
Contributor Author

/test all

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@davdhacs: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-stackrox-stackrox-master-ocp-4-12-compliance-e2e-tests stackrox/stackrox presubmit Ci-operator config changed
pull-ci-stackrox-stackrox-master-ocp-4-12-nongroovy-e2e-tests stackrox/stackrox presubmit Ci-operator config changed
pull-ci-stackrox-stackrox-master-ocp-4-12-operator-e2e-tests stackrox/stackrox presubmit Ci-operator config changed
pull-ci-stackrox-stackrox-master-ocp-4-12-qa-e2e-tests stackrox/stackrox presubmit Ci-operator config changed
pull-ci-stackrox-stackrox-master-ocp-4-12-scanner-v4-install-tests stackrox/stackrox presubmit Ci-operator config changed
pull-ci-stackrox-stackrox-master-ocp-4-12-ui-e2e-tests stackrox/stackrox presubmit Ci-operator config changed
pull-ci-stackrox-stackrox-master-ocp-4-22-qa-e2e-tests stackrox/stackrox presubmit Registry content changed
pull-ci-stackrox-stackrox-master-ocp-4-22-scanner-v4-install-tests stackrox/stackrox presubmit Registry content changed
pull-ci-stackrox-stackrox-master-ocp-4-22-operator-e2e-tests stackrox/stackrox presubmit Registry content changed
pull-ci-stackrox-stackrox-master-ocp-4-22-ui-e2e-tests stackrox/stackrox presubmit Registry content changed
pull-ci-stackrox-stackrox-master-ocp-4-22-fips-qa-e2e-tests stackrox/stackrox presubmit Registry content changed
pull-ci-stackrox-stackrox-master-ocp-4-22-compliance-e2e-tests stackrox/stackrox presubmit Registry content changed
pull-ci-stackrox-stackrox-master-ocp-4-22-nongroovy-e2e-tests stackrox/stackrox presubmit Registry content changed
pull-ci-stackrox-stackrox-master-ocp-4-22-vm-scanning-e2e-tests stackrox/stackrox presubmit Registry content changed
pull-ci-stackrox-stackrox-master-ocp-next-candidate-qa-e2e-tests stackrox/stackrox presubmit Registry content changed
pull-ci-stackrox-stackrox-master-ocp-next-candidate-scanner-v4-install-tests stackrox/stackrox presubmit Registry content changed
pull-ci-stackrox-stackrox-master-ocp-next-candidate-operator-e2e-tests stackrox/stackrox presubmit Registry content changed
pull-ci-stackrox-stackrox-master-ocp-next-candidate-ui-e2e-tests stackrox/stackrox presubmit Registry content changed
pull-ci-stackrox-stackrox-master-ocp-next-candidate-fips-qa-e2e-tests stackrox/stackrox presubmit Registry content changed
pull-ci-stackrox-stackrox-master-ocp-next-candidate-compliance-e2e-tests stackrox/stackrox presubmit Registry content changed
pull-ci-stackrox-stackrox-master-ocp-next-candidate-nongroovy-e2e-tests stackrox/stackrox presubmit Registry content changed
pull-ci-stackrox-stackrox-master-ocp-stable-scanner-v4-install-qa-e2e-tests stackrox/stackrox presubmit Registry content changed
pull-ci-stackrox-stackrox-master-ocp-stable-scanner-v4-install-scanner-v4-install-tests stackrox/stackrox presubmit Registry content changed
pull-ci-stackrox-stackrox-master-ocp-stable-scanner-v4-install-operator-e2e-tests stackrox/stackrox presubmit Registry content changed
pull-ci-stackrox-stackrox-master-ocp-stable-scanner-v4-install-ui-e2e-tests stackrox/stackrox presubmit Registry content changed

A total of 156 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@davdhacs

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-stackrox-stackrox-master-ocp-4-12-qa-e2e-tests pull-ci-stackrox-stackrox-master-ocp-4-12-nongroovy-e2e-tests pull-ci-stackrox-stackrox-master-ocp-4-12-operator-e2e-tests pull-ci-stackrox-stackrox-master-ocp-4-12-compliance-e2e-tests pull-ci-stackrox-stackrox-master-ocp-4-12-scanner-v4-install-tests pull-ci-stackrox-stackrox-master-ocp-4-12-ui-e2e-tests

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@davdhacs: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci

openshift-ci Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

@davdhacs: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/stackrox/stackrox/master/ocp-4-12-nongroovy-e2e-tests 689b66b link unknown /pj-rehearse pull-ci-stackrox-stackrox-master-ocp-4-12-nongroovy-e2e-tests
ci/rehearse/stackrox/stackrox/master/ocp-4-12-qa-e2e-tests 689b66b link unknown /pj-rehearse pull-ci-stackrox-stackrox-master-ocp-4-12-qa-e2e-tests
ci/rehearse/stackrox/stackrox/master/ocp-4-12-operator-e2e-tests 689b66b link unknown /pj-rehearse pull-ci-stackrox-stackrox-master-ocp-4-12-operator-e2e-tests
ci/rehearse/stackrox/stackrox/master/ocp-4-12-compliance-e2e-tests 689b66b link unknown /pj-rehearse pull-ci-stackrox-stackrox-master-ocp-4-12-compliance-e2e-tests

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@davdhacs

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-stackrox-stackrox-master-ocp-4-12-qa-e2e-tests pull-ci-stackrox-stackrox-master-ocp-4-12-nongroovy-e2e-tests pull-ci-stackrox-stackrox-master-ocp-4-12-operator-e2e-tests pull-ci-stackrox-stackrox-master-ocp-4-12-compliance-e2e-tests pull-ci-stackrox-stackrox-master-ocp-4-12-scanner-v4-install-tests pull-ci-stackrox-stackrox-master-ocp-4-12-ui-e2e-tests pull-ci-stackrox-stackrox-master-ocp-4-22-qa-e2e-tests pull-ci-stackrox-stackrox-master-ocp-4-22-nongroovy-e2e-tests pull-ci-stackrox-stackrox-master-ocp-4-22-operator-e2e-tests

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@davdhacs: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant