Repository navigation
test: Sentinel failover drills (just test-dr) - #117
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an opt-in failover suite,
just test-dr, modeled on faststream-outbox'stests_dr. It doesn't touch library code and doesn't run in CI.Cluster.
docker-compose.dr.ymlstarts a Redis primary, one replica and three Sentinels (quorum 2, primary marked down after 2s). The drill connects throughSentinel.master_for, the way a Sentinel deployment would.Drill. Each scenario gets a fresh cluster. The test publishes 50 timers that are due now and waits for them, then 50 more that fall due over the next 10s, and confirms the replica has them with
WAIT 1. Then it fails the primary, waits for Sentinel to promote the replica, and publishes 50 more, retrying until each one is accepted. It passes when all 150 are delivered and nothing is left pending on the new primary.Scenarios.
kill: the primary container is killed.pause: the primary container is frozen withdocker pause. Connects still complete, but no reply ever comes back. Outbox drops packets with iptables, but here that would only cut the test container off. Sentinel would keep seeing a healthy primary and never fail over.Results from
just test-dr(redis-py 8.1):The
pausedrill passes only because redis-py 8 defaultssocket_timeoutto 5s. Versions 5.0.1 through 7.4.0 default toNone, and with that the subscriber stops delivering for good after the primary freezes. That gets its own PR.