Add a foundation-stereo option for warping detections between stereo cameras - #1875
Open
mattdawkins wants to merge 1 commit into
Open
Add a foundation-stereo option for warping detections between stereo cameras#1875mattdawkins wants to merge 1 commit into
mattdawkins wants to merge 1 commit into
Conversation
…cameras The existing client-side transfer matches each point by NCC template matching along its epipolar curve, which needs the source patch to be photometrically matchable in the other view. Where that fails -- obstructed viewpoints, repetitive substrate, low contrast -- it mismatches or declines. Adds a second correspondence method that runs a Fast-FoundationStereo ONNX export instead: rectify the pair once per frame, run a dense disparity network, and read each point's shift out of the disparity map. Both methods satisfy a shared StereoMatcher interface, so box/line/polygon warping, measurement and bulk transfer are unchanged either way, and a Correspondence method dropdown under Track Settings -> Stereo Settings picks between them. The foundation model is not bundled -- the exports run ~100 MB -- so it is supplied by URL and the method no-ops with a warning when none is served. NCC remains the default.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a second stereo correspondence method — a Fast-FoundationStereo ONNX
export — alongside the existing NCC template matcher, with a dropdown under
Track Settings → Stereo Settings → Correspondence method to pick between
them. NCC remains the default.
Why
The existing matcher correlates the source patch along its epipolar curve, so it
needs that patch to be photometrically matchable in the other view. Where that
breaks down — obstructed viewpoints, repetitive substrate, low contrast — it
either mismatches or declines to answer.
A dense disparity network doesn't depend on patch correlation, and it costs one
pass per frame regardless of how many points are warped, so bulk-warping a whole
camera amortises well.
How
Both methods satisfy a shared
StereoMatcherinterface, so box / line / polygonwarping, measurement and bulk transfer are untouched and nothing downstream
branches on the choice.
The foundation path:
rectify.ts, ported fromOpenCV
cvStereoRectify), sized to the network's input resolution.pixel and bilinear-sampling — rectify and resize fused, so cost scales with
the network's resolution, not the frame's.
x,unrectify into the target image.
Step 4 pools rather than reading one pixel deliberately: a head or tail tip is a
couple of pixels wide at the network's working resolution, so the disparity
exactly at the tip is frequently the background's.
The network has no confidence channel, so
scoreis the fraction of the pooledwindow carrying a finite positive disparity; a match is accepted when that
clears a threshold and the implied disparity is inside the configured search
range — the same range that already bounds the NCC search.
The model is not bundled
Fast-FoundationStereo exports run ~100 MB, well past what belongs in the repo.
It's supplied by URL:
foundationModelSpecmust match the export — the graph fixes its inputresolution, and the sidecar
.yamlgives it asimage_size: [H, W]. With nomodel served, selecting the method logs a warning and the warp no-ops, the same
way a missing calibration already does.
Testing
tests/rectify.spec.ts(new, 6 cases): Rodrigues round-trip, orthonormality ofthe rectifying rotations, the defining rectification property (a 3D point lands
on the same row in both rectified views), disparity positive and decreasing
with range, and pixel round-trip through rectify/unrectify with and without
distortion.
Not tested:
StereoFoundationMatcherend-to-end, which needs the ~100 MBmodel the repo doesn't carry. The geometry it rests on is covered above, but the
network call, the disparity pooling and the dropdown have not been exercised
against a real export in a running viewer. Worth a look from someone who can
point it at a model.