Skip to content

fix: prevent cluster list pagination cycles - #296

Open
perfloop[bot] wants to merge 4 commits into
mainfrom
perfloop-pr-open-e46pcxng7j
Open

perfloop[bot] wants to merge 4 commits into
mainfrom
perfloop-pr-open-e46pcxng7j

Conversation

@perfloop

@perfloop perfloop Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Cluster-wide ListWithValues resumes after the last emitted key, preventing successful pages from restarting shards.

The two-node test enumerates a,b,c,z once at limit one and reaches HasMore=false; a duplicate empty-key page with an expired tail returns the live key and ends.

Changes

  • Complete-page tokens retain legacy per-node cursors; partial pages preserve peer progress. Compact tokens preserve rows and values at the response-size limit and recheck the rollout gate and healthy-member capabilities if compaction restores an empty cursor.
  • Key-only List and embedded listing use the shared merge; raw StartKey remains exclusive.
  • Node-local capability metadata uses grpc.SetHeader, preserving the interceptor's ring epoch.
  • Empty-key continuation is default-off. The startup flag is -enable-cluster-list-empty-key-pagination (restart required); embedded config is embedded.Config.EnableEmptyKeyListPagination (docs/configuration.md, docs/embedded.md). Enable every token-issuing coordinator only after all token receivers support it, and keep it on while tokens remain. A five-second probe fails closed on stalled or unsupported members; it cannot inventory every receiver or guarantee a page.

Stored values are unchanged.

Partial fan-outs are best-effort and may replay a key already returned by a responding shard; unmarked legacy tokens may omit keys below their greatest cursor. Order assumes fixed shards and successful replies per page. Live-cluster frequency is unknown; public continuations remain subject to the 128 MiB limit.

Sparse limit-case tokens can replay omitted peers on older readers.

Resource costs

Eleven paired runs measured increased Go-heap allocation. Empty two-shard List medians were 25,106 → 27,228 B/op and 442 → 466 allocs/op.

With an empty-key frontier, 1,000 expired rows, then a live row, median time rose from 4,262 → 546,482 ns/op for List and 7,374 → 540,797 ns/op for ListWithValues. The scan distinguishes terminal pages from pages requiring a gated empty-key token; nonempty frontiers avoid it. These costs retain progress and older-reader compatibility. Cluster elapsed-time differences remain inconclusive.

Allocation results, rollout details, and test coverage

All following before/after values are medians from the same eleven paired runs. Bytes and allocation counts cover the Go heap, not total process memory. These are measured costs, not speedups:

  • Empty two-shard ListWithValues: 23,483 → 25,733 B/op; 439 → 463 allocs/op.
  • Single-key List: 24,104 → 26,282 B/op; 449 → 469 allocs/op. ListWithValues: 32,612 → 34,774 B/op; 451 → 471 allocs/op.
  • Distinct-frontier List: 25,360 → 27,863 B/op; 485 → 506 allocs/op. ListWithValues: 42,382 → 44,738 B/op; 488 → 509 allocs/op.
  • Duplicate-frontier List: 25,120 → 27,844 B/op; 488 → 507 allocs/op. ListWithValues: 42,490 → 45,081 B/op; 493 → 512 allocs/op.
  • Legacy-resume List: 25,368 → 27,905 B/op; 482 → 511 allocs/op. ListWithValues: 37,952 → 40,462 B/op; 481 → 510 allocs/op.
  • Key-only 1,000-key page: 501,289 → 503,236 B/op; 14,011 → 14,031 allocs/op.
  • Empty-key frontier followed by 1,000 expired rows and a live row: List allocates 840 → 64,573 B/op and 26 → 4,025 allocs/op; ListWithValues allocates 5,248 → 68,981 B/op and 30 → 4,029 allocs/op.

Capability headers and cumulative cursors add allocations. Cursor calls reuse the context when no marker needs changing. Empty-key frontiers inspect expiry without decoding payloads, including when no live row follows. Without this lookahead, an expired suffix can demand an empty-key token after enumeration has finished, causing FailedPrecondition with the default-off gate.

At the response-size limit, compact tokens retain the base page's sparse cursors without dropping rows or values.

Testing

make -o proto build
make -o proto test-server TESTRUN=EmptyStartKey
make -o proto test-server TEST=TestLoadConfig_EnableEmptyKeyListPagination
make -o proto test-coordinator
make -o proto test-embedded
make -o proto test-storage TESTRUN=Pagination
make -o proto test-integration-short TEST=TestIntegration_Cluster
make -o proto test-integration-cluster TEST=TestIntegration_Cluster/List
make -o proto test-integration-cluster TEST=TestIntegration_Cluster/ClusterSuite/Test_ClusterList_Pagination
make -o proto test-integration TEST=TestEmbeddedClient_ListPage
make -o proto lint
make -o proto license-check

Go tests passed for server list responses, both node-local response-header paths, cursor metadata, storage pagination, cluster pagination, embedded pagination, and the existing streamed-put helper caller. Lint and dependency-license validation passed.

TestIntegration_Compaction/CompactionSuite/Test_SegmentRecompaction_ThresholdBehavior did not pass reliably on either revision and is left to target CI.

TestIntegration_Coordinator/CoordinatorSuite/Test_Coordinator_GracefulNodeDeparture also had unreliable startup on both revisions and is left to target CI.

ListWithValuesPagination checks order, values, duplicates, legacy tokens, plain starts, and completion. ListCompactEmptyCursorRolloutGate checks both RPCs at the compact-token boundary with issuance disabled, enabled, and an unsupported leaving member. ListDuplicateEmptyKeyTerminalPage checks both public RPCs and embedded operations with the expired tail. ListLegacyReaderCompatibility models the base decoder over node-local RPCs. ListLegacyServerRPC in tests/integration/list_legacy_rpc_test.go resumes complete and partial pages through pinned older server handlers for both public RPCs with a nonempty prefix; the static compatibility build carries the original RocksDB artifact path into the historical tree. ListPartialFailureRecovery, ListPartialCursorAtResponseLimit, and ListLargeCursorResponse cover recovery, exactly fitting pages, and large distinct or duplicate cursors. TestEmbeddedClient_ListPageWithValues and TestPaginationCursorAfterEmptyKey check embedded and storage cursors. tests/integration/integration_test.go skips added cases in short mode.


Generated by Perfloop.


Note

Medium Risk
Changes core cluster list merge, token encoding, and rollout semantics across server, storage, and embedded clients; misconfiguration or mixed versions can surface FailedPrecondition or alter pagination behavior.

Overview
Cluster-wide List/ListWithValues now resume correctly after the last emitted key (including the empty key) without cycling or skipping shards. Continuation tokens gain explicit global, peer, and empty-cursor modes, cumulative per-node cursors for older readers, and compact tokens when a full cursor map would exceed the gRPC message budget.

Empty-key continuation is opt-in via -enable-cluster-list-empty-key-pagination / embedded.Config.EnableEmptyKeyListPagination (default false). When disabled or when ring members lack capability, pages that need an empty-key cursor return FailedPrecondition instead of issuing incompatible tokens. Node-local handlers advertise support through gRPC metadata; coordinators probe healthy (including LEAVING) members with a bounded timeout.

Storage and RPC paths treat start cursors as present-or-absent (*string / listinput context) so an exclusive empty StartKey is not confused with “start from beginning.” Local list skips expired tail keys when deciding whether another page is needed after an empty frontier.

Docs, config wiring, and broad integration coverage (legacy token compatibility, partial fan-out, rollout gates, pinned old-server RPC) exercise the new behavior.

Reviewed by Cursor Bugbot for commit 5760eb2. Bugbot is set up for automated code reviews on this repo. Configure here.

@greptile-apps

greptile-apps Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; both previous blocking findings are addressed.

Summary

Fixes cluster listing so successful pages resume after the last returned key instead of restarting shards.

  • Cluster list pages resume from their last emitted key.
  • List pages can continue after an empty key when receivers support it.

Both numbered findings from the previous review are addressed. No new actionable issues were found. Tests were not run during this review.

Reviews (4) · Last reviewed commit: "fix: gate compact empty cursors and reta..." · Reviewed by Greptile

Comment thread server/operations/list.go Resolved
Comment thread server/internal/listinput/context.go Resolved

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 43370c5. Configure here.

Comment thread server/internal/listinput/context.go Resolved
@perfloop

perfloop Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

Perfloop stopped work on this pull request with comments still to address.

@perfloop

perfloop Bot commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

Perfloop paused work on this pull request before it answered the open comments. The work reached the usage limit set for this one change, not the workspace's overall limit. An admin of the Perfloop workspace that opened it can continue the work from the Inbox.

Signed-off-by: Perfloop Agent <agent@perfloop.ai>
Signed-off-by: Perfloop Agent <agent@perfloop.ai>
Signed-off-by: Perfloop Agent <agent@perfloop.ai>
@perfloop
perfloop Bot force-pushed the perfloop-pr-open-e46pcxng7j branch from 43370c5 to 228ee93 Compare November 16, 2026 19:36
@perfloop

perfloop Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

The reported header issue is addressed by queuing capability metadata with grpc.SetHeader. The response-header regression test exercises both node-local list RPCs with the epoch interceptor installed.

Comment thread server/operations/list.go Resolved
Signed-off-by: Perfloop Agent <agent@perfloop.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant