Skip to content

test: Sentinel failover drills (just test-dr) - #117

Merged
lesnik512 merged 1 commit into
mainfrom
test/dr-failover-drills
Oct 28, 2026
Merged

lesnik512 merged 1 commit into
mainfrom
test/dr-failover-drills

Conversation

@lesnik512

Copy link
Copy Markdown
Member

Adds an opt-in failover suite, just test-dr, modeled on faststream-outbox's tests_dr. It doesn't touch library code and doesn't run in CI.

Cluster. docker-compose.dr.yml starts a Redis primary, one replica and three Sentinels (quorum 2, primary marked down after 2s). The drill connects through Sentinel.master_for, the way a Sentinel deployment would.

Drill. Each scenario gets a fresh cluster. The test publishes 50 timers that are due now and waits for them, then 50 more that fall due over the next 10s, and confirms the replica has them with WAIT 1. Then it fails the primary, waits for Sentinel to promote the replica, and publishes 50 more, retrying until each one is accepted. It passes when all 150 are delivered and nothing is left pending on the new primary.

Scenarios.

  • kill: the primary container is killed.
  • pause: the primary container is frozen with docker pause. Connects still complete, but no reply ever comes back. Outbox drops packets with iptables, but here that would only cut the test container off. Sentinel would keep seeing a healthy primary and never fail over.

Results from just test-dr (redis-py 8.1):

Scenario Promoted Publish recovered Subscriber recovered Redeliveries
kill 3.5s 3.6s 10.9s 0
pause 4.2s 9.8s 11.0s 0

The pause drill passes only because redis-py 8 defaults socket_timeout to 5s. Versions 5.0.1 through 7.4.0 default to None, and with that the subscriber stops delivering for good after the primary freezes. That gets its own PR.

@lesnik512
lesnik512 merged commit 7bb9607 into main Oct 28, 2026
17 checks passed
@lesnik512
lesnik512 deleted the test/dr-failover-drills branch October 28, 2026 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant