[Cosmos] Backport PPCB fixes to 4.81.1 - #50282
Merged
Abhijeet Mohanty (jeet1995) merged 16 commits intoSep 4, 2026
Merged
Conversation
…ientCfgs The partitionLevelCircuitBreakerCfg field disappeared from the clientCfgs section of CosmosDiagnostics when a customer explicitly enabled Per-Partition Circuit Breaker (PPCB) client-side. Root cause: the diagnostics write was coupled to the Per-Partition Automatic Failover (PPAF) initialization path, so the field was only populated when the service mandated PPAF, not when PPCB was configured client-side. Fix: move the diagnostics write into initializePerPartitionCircuitBreaker(), which is invoked unconditionally at client init, so the field appears whenever the circuit breaker is configured client-side. Adds a CI-runnable regression test asserting all clientCfgs keys are present, including partitionLevelCircuitBreakerCfg. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…roperty Remove trailing comma before closing brace in the COSMOS.PARTITION_LEVEL_CIRCUIT_BREAKER_CONFIG JSON so the value is strictly valid and clearer as a customer reference. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…kerE2ETests Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… keys The partitionLevelCircuitBreakerCfg field now appears in clientCfgs for every client regardless of PPCB configuration, so the E2E test no longer sets the COSMOS.PARTITION_LEVEL_CIRCUIT_BREAKER_CONFIG system property. It just builds a plain client and asserts all expected clientCfgs keys (including partitionLevelCircuitBreakerCfg) are present in CosmosDiagnostics. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Remove PPCB System.setProperty/clearProperty and the value-specific assertion; assert only that all clientCfgs keys are present, matching the E2E test simplification. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Fix PPCB failback with missing or stale addresses --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Improve PPCB failback diagnostics * Validate PPCB state in diagnostics E2E tests * Limit PPCB diagnostics assertion to data-plane requests * Log PPCB failback backlog progress * Add PPCB failback remaining meter * Clarify PPCB failback meter name * Reduce PPCB failback meter allocations * Track PPCB pending recoveries by collection * Correlate PPCB failback recovery diagnostics * Harden PPCB failback recovery tests * Fix PPCB all-region diagnostics assertion * Optimize PPCB diagnostics snapshots * Cache PPCB diagnostics snapshots * Align PPCB failback flow with main * Add benchmark fault injection support. * Harden PPCB failback telemetry Use a single injectable logger, prevent backlog telemetry failures from escaping the recovery flow, and align the pending failback metric name. * Scope PR to PPCB diagnostics Remove benchmark, fault-injection, metric, and recovery behavior changes. Retain immutable CosmosDiagnostics PPCB snapshots, lifecycle E2E assertions, and WARN logging for every failback failure. * Reduce PPCB diagnostics overhead Reuse immutable PPCB map references in response snapshots and shorten per-region diagnostic field names. * Refactoring * Simplify PPCB diagnostics snapshots Represent holder state with one volatile immutable-map reference and align the compact timestamp serialization test. * Ignore updates to empty PPCB diagnostics Make updates to the shared uninitialized diagnostics sentinel a no-op and verify it remains null-serializing. * Avoid copying PPCB diagnostics state Retain the live PPCB diagnostics map reference to avoid per-publication map and wrapper allocations, accepting weak consistency. * Log PPCB diagnostics lifecycle snapshots Emit one full CosmosDiagnostics JSON payload for failed, post-failover, and post-failback E2E phases for PR evidence. * Document PPCB diagnostics improvements Add the unreleased changelog entry for per-region PPCB snapshots and failback WARN logging. * Add PPCB failback outcome diagnostics Track the latest background failback attempt time, outcome, and failure reason per partition-region; validate lifecycle state in E2E and focused recovery tests. * Reset unavailable timestamp after PPCB failback Use the available-state sentinel when recovery moves a region to HealthyTentative and cover it in the scheduled recovery test. * Refine PPCB failback diagnostics Keep failback attempt metadata partition-scoped, retain only the latest full failure message per region, and clear retained messages when recovery backlog drains.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 16ce0941-555c-4190-8c4d-96c705086350
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 16ce0941-555c-4190-8c4d-96c705086350
|
Azure Pipelines: 34 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Member
Author
|
/azp run java - cosmos - tests |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Abhijeet Mohanty (jeet1995)
deleted the
release_azure-cosmos-4.81.1-hotfix
branch
August 31, 2026 13:16
Abhijeet Mohanty (jeet1995)
restored the
release_azure-cosmos-4.81.1-hotfix
branch
August 31, 2026 13:26
|
Azure Pipelines: 34 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Abhijeet Mohanty (jeet1995)
changed the base branch from
ppcb-hotfix-baseline/azure-cosmos-4.81.0
to
hotfix/jeet1995/4.81.1-hotfix
August 31, 2026 13:28
Abhijeet Mohanty (jeet1995)
changed the base branch from
hotfix/jeet1995/4.81.1-hotfix
to
hotfix/jeet1995/azure-cosmos_4.81.0
August 31, 2026 15:05
Abhijeet Mohanty (jeet1995)
changed the base branch from
hotfix/jeet1995/azure-cosmos_4.81.0
to
hotfix/jeet1995/azure-cosmos_4.81.1-hotfix
August 31, 2026 15:13
Abhijeet Mohanty (jeet1995)
marked this pull request as ready for review
September 1, 2026 15:18
Abhijeet Mohanty (jeet1995)
requested review from
a team and
Kiran Kumar Kolli (kirankumarkolli)
as code owners
September 1, 2026 15:18
|
Azure Pipelines: Successfully started running 2 pipeline(s). 32 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 16ce0941-555c-4190-8c4d-96c705086350
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 16ce0941-555c-4190-8c4d-96c705086350
Abhijeet Mohanty (jeet1995)
changed the title
Tomas Varon (tvaron3)
approved these changes
Sep 3, 2026
Fabian Meiswinkel (FabianMeiswinkel)
approved these changes
Sep 3, 2026
Fabian Meiswinkel (FabianMeiswinkel)
left a comment
Fabian Meiswinkel (FabianMeiswinkel)
left a comment
Member
There was a problem hiding this comment.
LGTM
Member
Author
|
/azp run java - cosmos - tests |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Abhijeet Mohanty (jeet1995)
merged commit Sep 4, 2026
c9d65ef
into
Azure:hotfix/jeet1995/azure-cosmos_4.81.1-hotfix
106 of 108 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Backports the required Per-Partition Circuit Breaker (PPCB) fixes to
azure-cosmos4.81.1 and adapts release validation for network-isolated hosted agents.PR branches
Azure:hotfix/jeet1995/azure-cosmos_4.81.1-hotfix987c15b6ee19f2a51d054c795394331caf2119cajeet1995:release_azure-cosmos-4.81.1-hotfixProduct changes
partitionLevelCircuitBreakerCfgin the diagnosticsclientCfgssection.Pipeline changes
-amdotherwise reaches those tests and attempts blocked public image pulls; all normal NotFromSource Cosmos jobs remain enabled.Test changes
nonCanonicalPreferredRegions_ppcbShouldStillRouteCorrectlyonly on this release, where the supporting region-name normalization product change exists.Release metadata
Prepare-Release.ps1forcom.azure:azure-cosmos.4.81.1-hotfix) and dates the changelog2026-09-03.eng/versioning/version_client.txtand all dependent Cosmos module POM references, including the benchmark module.