Repository navigation
fix: prevent cluster list pagination cycles - #296
perfloop[bot] wants to merge 4 commits into
Conversation
[Medium impact] The PR appears safe to merge; both previous blocking findings are addressed. SummaryFixes cluster listing so successful pages resume after the last returned key instead of restarting shards.
Both numbered findings from the previous review are addressed. No new actionable issues were found. Tests were not run during this review. Reviews (4) · Last reviewed commit: "fix: gate compact empty cursors and reta..." · Reviewed by Greptile |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 43370c5. Configure here.
|
Perfloop stopped work on this pull request with comments still to address. |
|
Perfloop paused work on this pull request before it answered the open comments. The work reached the usage limit set for this one change, not the workspace's overall limit. An admin of the Perfloop workspace that opened it can continue the work from the Inbox. |
Signed-off-by: Perfloop Agent <agent@perfloop.ai>
Signed-off-by: Perfloop Agent <agent@perfloop.ai>
Signed-off-by: Perfloop Agent <agent@perfloop.ai>
43370c5 to
228ee93
Compare
|
The reported header issue is addressed by queuing capability metadata with |
Signed-off-by: Perfloop Agent <agent@perfloop.ai>

Summary
Cluster-wide
ListWithValuesresumes after the last emitted key, preventing successful pages from restarting shards.The two-node test enumerates
a,b,c,zonce at limit one and reachesHasMore=false; a duplicate empty-key page with an expired tail returns the live key and ends.Changes
Listand embedded listing use the shared merge; rawStartKeyremains exclusive.grpc.SetHeader, preserving the interceptor's ring epoch.-enable-cluster-list-empty-key-pagination(restart required); embedded config isembedded.Config.EnableEmptyKeyListPagination(docs/configuration.md, docs/embedded.md). Enable every token-issuing coordinator only after all token receivers support it, and keep it on while tokens remain. A five-second probe fails closed on stalled or unsupported members; it cannot inventory every receiver or guarantee a page.Stored values are unchanged.
Partial fan-outs are best-effort and may replay a key already returned by a responding shard; unmarked legacy tokens may omit keys below their greatest cursor. Order assumes fixed shards and successful replies per page. Live-cluster frequency is unknown; public continuations remain subject to the 128 MiB limit.
Sparse limit-case tokens can replay omitted peers on older readers.
Resource costs
Eleven paired runs measured increased Go-heap allocation. Empty two-shard
Listmedians were 25,106 → 27,228 B/op and 442 → 466 allocs/op.With an empty-key frontier, 1,000 expired rows, then a live row, median time rose from 4,262 → 546,482 ns/op for
Listand 7,374 → 540,797 ns/op forListWithValues. The scan distinguishes terminal pages from pages requiring a gated empty-key token; nonempty frontiers avoid it. These costs retain progress and older-reader compatibility. Cluster elapsed-time differences remain inconclusive.Allocation results, rollout details, and test coverage
All following before/after values are medians from the same eleven paired runs. Bytes and allocation counts cover the Go heap, not total process memory. These are measured costs, not speedups:
ListWithValues: 23,483 → 25,733 B/op; 439 → 463 allocs/op.List: 24,104 → 26,282 B/op; 449 → 469 allocs/op.ListWithValues: 32,612 → 34,774 B/op; 451 → 471 allocs/op.List: 25,360 → 27,863 B/op; 485 → 506 allocs/op.ListWithValues: 42,382 → 44,738 B/op; 488 → 509 allocs/op.List: 25,120 → 27,844 B/op; 488 → 507 allocs/op.ListWithValues: 42,490 → 45,081 B/op; 493 → 512 allocs/op.List: 25,368 → 27,905 B/op; 482 → 511 allocs/op.ListWithValues: 37,952 → 40,462 B/op; 481 → 510 allocs/op.Listallocates 840 → 64,573 B/op and 26 → 4,025 allocs/op;ListWithValuesallocates 5,248 → 68,981 B/op and 30 → 4,029 allocs/op.Capability headers and cumulative cursors add allocations. Cursor calls reuse the context when no marker needs changing. Empty-key frontiers inspect expiry without decoding payloads, including when no live row follows. Without this lookahead, an expired suffix can demand an empty-key token after enumeration has finished, causing
FailedPreconditionwith the default-off gate.At the response-size limit, compact tokens retain the base page's sparse cursors without dropping rows or values.
Testing
Go tests passed for server list responses, both node-local response-header paths, cursor metadata, storage pagination, cluster pagination, embedded pagination, and the existing streamed-put helper caller. Lint and dependency-license validation passed.
TestIntegration_Compaction/CompactionSuite/Test_SegmentRecompaction_ThresholdBehaviordid not pass reliably on either revision and is left to target CI.TestIntegration_Coordinator/CoordinatorSuite/Test_Coordinator_GracefulNodeDeparturealso had unreliable startup on both revisions and is left to target CI.ListWithValuesPaginationchecks order, values, duplicates, legacy tokens, plain starts, and completion.ListCompactEmptyCursorRolloutGatechecks both RPCs at the compact-token boundary with issuance disabled, enabled, and an unsupported leaving member.ListDuplicateEmptyKeyTerminalPagechecks both public RPCs and embedded operations with the expired tail.ListLegacyReaderCompatibilitymodels the base decoder over node-local RPCs.ListLegacyServerRPCintests/integration/list_legacy_rpc_test.goresumes complete and partial pages through pinned older server handlers for both public RPCs with a nonempty prefix; the static compatibility build carries the original RocksDB artifact path into the historical tree.ListPartialFailureRecovery,ListPartialCursorAtResponseLimit, andListLargeCursorResponsecover recovery, exactly fitting pages, and large distinct or duplicate cursors.TestEmbeddedClient_ListPageWithValuesandTestPaginationCursorAfterEmptyKeycheck embedded and storage cursors.tests/integration/integration_test.goskips added cases in short mode.Generated by Perfloop.
Note
Medium Risk
Changes core cluster list merge, token encoding, and rollout semantics across server, storage, and embedded clients; misconfiguration or mixed versions can surface FailedPrecondition or alter pagination behavior.
Overview
Cluster-wide List/ListWithValues now resume correctly after the last emitted key (including the empty key) without cycling or skipping shards. Continuation tokens gain explicit global, peer, and empty-cursor modes, cumulative per-node cursors for older readers, and compact tokens when a full cursor map would exceed the gRPC message budget.
Empty-key continuation is opt-in via
-enable-cluster-list-empty-key-pagination/embedded.Config.EnableEmptyKeyListPagination(default false). When disabled or when ring members lack capability, pages that need an empty-key cursor returnFailedPreconditioninstead of issuing incompatible tokens. Node-local handlers advertise support through gRPC metadata; coordinators probe healthy (including LEAVING) members with a bounded timeout.Storage and RPC paths treat start cursors as present-or-absent (
*string/listinputcontext) so an exclusive emptyStartKeyis not confused with “start from beginning.” Local list skips expired tail keys when deciding whether another page is needed after an empty frontier.Docs, config wiring, and broad integration coverage (legacy token compatibility, partial fan-out, rollout gates, pinned old-server RPC) exercise the new behavior.
Reviewed by Cursor Bugbot for commit 5760eb2. Bugbot is set up for automated code reviews on this repo. Configure here.