Skip to content

Add a foundation-stereo option for warping detections between stereo cameras - #1875

Open
mattdawkins wants to merge 1 commit into
mainfrom
feat/stereo-foundation-transfer
Open

Add a foundation-stereo option for warping detections between stereo cameras#1875
mattdawkins wants to merge 1 commit into
mainfrom
feat/stereo-foundation-transfer

Conversation

@mattdawkins

Copy link
Copy Markdown
Member

What

Adds a second stereo correspondence method — a Fast-FoundationStereo ONNX
export — alongside the existing NCC template matcher, with a dropdown under
Track Settings → Stereo Settings → Correspondence method to pick between
them. NCC remains the default.

Why

The existing matcher correlates the source patch along its epipolar curve, so it
needs that patch to be photometrically matchable in the other view. Where that
breaks down — obstructed viewpoints, repetitive substrate, low contrast — it
either mismatches or declines to answer.

A dense disparity network doesn't depend on patch correlation, and it costs one
pass per frame regardless of how many points are warped, so bulk-warping a whole
camera amortises well.

How

Both methods satisfy a shared StereoMatcher interface, so box / line / polygon
warping, measurement and bulk transfer are untouched and nothing downstream
branches on the choice.

The foundation path:

  1. Solve the rectifying rotations once per calibration (rectify.ts, ported from
    OpenCV cvStereoRectify), sized to the network's input resolution.
  2. Build the rectified pair by inverse-mapping each output pixel to its source
    pixel and bilinear-sampling — rectify and resize fused, so cost scales with
    the network's resolution, not the frame's.
  3. Run the network for dense disparity.
  4. Per point: rectify, pool disparities in a small window by median, shift x,
    unrectify into the target image.

Step 4 pools rather than reading one pixel deliberately: a head or tail tip is a
couple of pixels wide at the network's working resolution, so the disparity
exactly at the tip is frequently the background's.

The network has no confidence channel, so score is the fraction of the pooled
window carrying a finite positive disparity; a match is accepted when that
clears a threshold and the implied disparity is inside the configured search
range — the same range that already bounds the NCC search.

The model is not bundled

Fast-FoundationStereo exports run ~100 MB, well past what belongs in the repo.
It's supplied by URL:

useStereoOnnxWeb({
  ...,
  foundationModelUrl: '/models/stereo_foundation.onnx',
  foundationModelSpec: { height: 576, width: 960 },  // the export's sidecar image_size
});

foundationModelSpec must match the export — the graph fixes its input
resolution, and the sidecar .yaml gives it as image_size: [H, W]. With no
model served, selecting the method logs a warning and the warp no-ops, the same
way a missing calibration already does.

Testing

  • tests/rectify.spec.ts (new, 6 cases): Rodrigues round-trip, orthonormality of
    the rectifying rotations, the defining rectification property (a 3D point lands
    on the same row in both rectified views), disparity positive and decreasing
    with range, and pixel round-trip through rectify/unrectify with and without
    distortion.
  • Full stereo suite passes (28/28); lint clean.

Not tested: StereoFoundationMatcher end-to-end, which needs the ~100 MB
model the repo doesn't carry. The geometry it rests on is covered above, but the
network call, the disparity pooling and the dropdown have not been exercised
against a real export in a running viewer. Worth a look from someone who can
point it at a model.

…cameras

The existing client-side transfer matches each point by NCC template matching
along its epipolar curve, which needs the source patch to be photometrically
matchable in the other view. Where that fails -- obstructed viewpoints,
repetitive substrate, low contrast -- it mismatches or declines.

Adds a second correspondence method that runs a Fast-FoundationStereo ONNX
export instead: rectify the pair once per frame, run a dense disparity network,
and read each point's shift out of the disparity map. Both methods satisfy a
shared StereoMatcher interface, so box/line/polygon warping, measurement and
bulk transfer are unchanged either way, and a Correspondence method dropdown
under Track Settings -> Stereo Settings picks between them.

The foundation model is not bundled -- the exports run ~100 MB -- so it is
supplied by URL and the method no-ops with a warning when none is served.
NCC remains the default.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant