Dev/v6.x#167
Merged
Merged
Conversation
szmyd
force-pushed
the
dev/v6.x
branch
2 times, most recently
from
July 9, 2026 14:02
dd27272 to
4ce88c1
Compare
Adds dlsn_tracker -- the client-side dLSN bookkeeping for one CRAFT partition --
and rewires craft_client onto it. It owns the dLSN counter, the contiguous
commit frontier F that client_hdr.commit_lsn piggybacks, and the per-read
horizon H.
The spine is sisl::StreamTracker keyed by dLSN. The load-bearing choice is that
its "completed" bit means RESOLVED (acked or empty), not "acked", which yields
the invariant "every slot <= F is resolved". So "is anything unresolved at or
below H?" collapses to "F < H": the common read is two atomic loads and never
touches the tracker. It also means a sub-quorum write pins F, which is exactly
the commit_lsn the protocol wants, since commit must not pass a slot the leader
has not yet filled or Emptied. Advancing F is completed_upto() over the
completion bitset, so the std::mutex around a std::set that used to fold acked
dLSNs on every ack is gone from the I/O path.
The horizon is a per-LBA property, and a read carries one. For each block, with
A the highest acked slot covering it and u* the lowest unresolved slot covering
it above A, that block is safe at H = u*-1: nothing unresolved remains at or
below it, and A < u*, so no acked write is hidden. With no such u* the block is
safe at the full horizon, because a replica serves the highest dLSN <= H per
LBA and an unresolved slot beneath a durable one is superseded -- it can never
be the version a read sees. When blocks in a range disagree, the read
partitions into segments issued in parallel, each landing in its own slice of
the caller's buffer via sisl::sg_iterator. That is CRAFT-Design.md's split
read, and it is why an unresolved write never blocks a read: a safe per-block
horizon always exists, so in_flight and failed slots fence identically.
Reads wait only in three pathological branches -- an index whose create() is not
yet visible, a window past the scan cap, or a plan wider than the fan-out cap --
on a re-armable resolution_gate whose Dekker interlock lets signal() skip its
allocation when nobody is stalled without losing a wakeup. A failed slot in one
of those branches fails instead of waiting, since nothing resolves it without a
leader resolution round.
Three bugs found while reading the client:
- A recovery login seeded next_dlsn but left the horizon at -1, so every read
carried H = -1 and returned zeros. reset_at() now seeds both watermarks from
the set's rs_commit_lsn.
- make_hdr() sent its own frontier as all_committed_lsn. That is an upper
bound on the set-wide minimum, never the minimum, so a replica could reclaim
journal a lagging peer still needs. It now sends -1 (unknown).
- A follower login returns a value with term == 0 and leader_hint, not an
error. The client turned that into a hard NOT_LEADER and discarded the hint;
it now follows the redirect.
Requires the companion sisl commit "AtomicBitset: publish with release, observe
with acquire". The read path scans slots the write path is concurrently
creating, and the slot bit is the only thing that publishes the payload.
NOT included, and drawn explicitly in the new README: read routing. Every
segment goes to replicas_[leader_] and that peer's error is surfaced directly.
There is no failover, no eligibility check, and no per-replica Missing map. The
tracker answers which version a read may see; the Missing map answers which
replica can serve it. Only the first is built, so read() is correct only when
the leader is in the write quorum of every winner in the range.
Tests: test_craft_dlsn_tracker drives reserve/resolve directly (14 cases; links
without homestore::homestore) and ends with a concurrent reader/writer stress
meant for TSAN -- readers scan slots writers are creating, which reports ~12
races with relaxed bit ordering and none with release/acquire. test_craft_client
adds the end-to-end cases against the reference model, including one where the
leader physically holds a sub-quorum write spanning both blocks of a split read.
craft_test_util.hpp collects the scaffolding the four craft test binaries had
each copied.
Verified end to end: craft_ublk --replicas 3 exposes /dev/ublkb0; unwritten
blocks read as zeros, direct and buffered writes round-trip, overwrite takes,
8-block IO round-trips, FLUSH succeeds, and DISCARD is refused.
Also folds in a clang-format pass over src/test/craft, src/test/legacy_ublk and
src/lib/craft (no behaviour change), and fixes the version-bump CI check, which
globbed a src/tests/ directory that is actually named src/test/.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.