Corgi hash key lane - #832
Merged
Merged
Conversation
Corgi must hand the proxy seam an integer key, and the seam works properly only when the arrangement is already sorted by that integer. Until now corgi supplied the integer but not the order: `ids()` derived it per retire and threw it away, so for anything but a primitive-integer key the arrangement was in structural key order and the identifier was in some other order. Everything downstream paid for that mismatch — the identifier was re-hashed every retire, and `changed` could not be turned into needles, so the accumulated history was scanned in full. The rule is now applied once, at ingest (`CorgiChunker::flush`): a key that is already a primitive integer is used as it stands; any other key shape is hashed and the hash PREPENDED, making the key `Prod([hash, key])`. `from_columns` sorts lexicographically over lanes, so that is hash order with the real key as tie-break, and the identifier is lane 0 either way. The real key stays in the column, so colliding keys land adjacent and sub-sorted rather than indistinguishable, and reads recover it by dropping a lane (an `Arc` bump). The hash is computed once and thereafter moves as data: merge, advance and settle permute key columns with `gather_lanes`, so no transducer recomputes it. This is the distinction the earlier hash-ordering attempt missed — a coordinate is carried, a sidecar is recomputed. What this deletes: * `collect_present`'s scan branch, and with it `seek_needles`' `Option`, the seek-vs-scan heuristic and `SEEK_ADVANTAGE`. Every key shape now seeks, and seeks by `find_ranges` over a `u64` leaf — corgi's fast path. * `ids()` on keys. It reads lane 0. Egress strips the lane in the two places a key leaves an arrangement: the container built by `as_collection`, and the key handed to the join's projection, which is written against the key the program declared. Values are untouched — value order is load-bearing for Min/Collect, and value identifiers are a separate question. 53 tests, plus the 4 heavy release ones. `pair_keys` caught the join egress. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YH7iq9JoXmf7ATpq1gZaQS
… cheap Deleting the scan branch cost the primitive-integer programs: scc +1.2%, ident +3.4%, both of which this change should not have touched at all. The heuristic was earning its keep — broad-churn retires (loads, label cascades) present most of the key space, and a gallop per changed key loses to a flat membership test there. Restored, with both branches reading the identifier lane. That is strictly better than the version this branch started from: the seek is `find_ranges` over a `u64` leaf for EVERY key shape, and the scan BORROWS the lane rather than materializing it — a structural key used to be forced onto the scan (its on-the-fly hash could not be inverted into a needle) and forced to allocate a hash vector per chunk per retire to take it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YH7iq9JoXmf7ATpq1gZaQS
…umes `advance_leaf` was the only regime that blocks: it seeks `from`, ends at a key boundary, and writes back where to resume. It was reachable only by keys that were a single u64 lane. `advance_lanes` and `advance_structured` opened with `*from = None` and staged the whole intersection in one call, which is the join half of the windowing problem. Now that every arrangement leads with its identifier and is sorted by it, the leaf walk applies to every key shape — it seeks and resumes on that lane. The dispatch tries it first and the two whole-key walks become the fallback. Three sites moved off whole-key-shaped needles onto the lane: the block seek, the batched probe, and `LeafView`'s pull. `needle_like` goes with them — building a needle in the key's own shape was only ever a way to search a column that had no integer to search by. The one exposure is a hash collision, which would cross-product two distinct keys sharing an identifier. `one_key` checks per matched key, not per row: runs are contiguous and sub-sorted by the real key, so a run holds one key exactly when its first and last rows agree, and the same pass over both sides' runs confirms they matched on the key rather than the hash. On failure the call is redone by the whole-key walks, which compare the real key and cannot be fooled. That path cannot be provoked with a real hash, so it is mutation-tested both ways: forcing the fallback for every hashed key leaves all 53 tests passing (it computes the same answer), and panicking in the fallback also leaves them passing (it is never taken, so the leaf walk really is handling these keys). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YH7iq9JoXmf7ATpq1gZaQS
The block bound used to be discovered from what had already been read: pull a fixed number of rows per chunk, then take the least last-pulled identifier, since anything at or beyond it might have more rows outside somebody's buffer. Whatever sat past that bound was discarded and read again by the next block. The waste is structural rather than incidental — a chunk whose keys are sparse spans a wide identifier range with its rows and a dense one spans a narrow range, and the bound is the minimum, so the sparse chunks re-read most of what they touched, every block. A totally ordered identifier makes the other order possible. `block_horizon` reads ONE value per chunk — the identifier a budget's worth of rows in, bumped past its own run so a key is never split — and takes the least. `block_ends` then binary-searches that bound in each chunk. Each chunk is read once, over exactly the rows the block needs, and nothing is read twice. This is why it could not have been written this way before: choosing a bound before reading requires a single ordered value you can both index by position and seek by value, and until the arrangement led with its identifier, a non-scalar key had neither. Falling out of it: * `pull_horizon` and its extend loop are gone. The degenerate case they existed for — a run longer than the budget denying all progress — cannot arise, because the bound is taken past the run at the budget, so the chunk that set it always contributes at least a budget of rows. Progress is by construction, not by retry. * `LeafView` borrows its identifiers straight from the chunk's lane instead of gathering them into a `Vec`. The block reads keys; it no longer copies them. * Seeking the resume point is a `partition_point` on that slice rather than a column probe. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YH7iq9JoXmf7ATpq1gZaQS
…-lane # Conflicts: # differential-dataflow/src/operators/int_proxy/reduce.rs # interactive/src/corgi/reduce.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Encourages DDIR's Corgi backend to use hashes in place of non-primitive keys (anything other than integers). Generally good news, but some slowdown reported with narrow compound keys, where adding the hash increases the sort times because of the radix passes required.