[RFC] What filtered search algorithms should DiskANN support?

[magdalendobson]

This PR adds an RFC on which filtered search algorithms the DiskANN repo should support.

[Copilot]

Pull request overview

Adds an RFC documenting an empirical evaluation of DiskANN filtered-search algorithm variants, with a recommendation for which algorithms the repo should support going forward.

Changes:

• Introduces a new RFC describing existing and proposed filtered-search algorithms (inline, beta, multi-hop, two-queue, adaptive-L).
• Records benchmark results on two datasets (Caselaw and YFCC) with accompanying plots.
• Proposes a path forward: add inline (optionally adaptive-L), retain multi-hop, deprecate beta, and close two-queue.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.45%. Comparing base (68cc3c4) to head (0d4eadb).
⚠️ Report is 8 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1128      +/-   ##
==========================================
+ Coverage   88.87%   89.45%   +0.57%     
==========================================
  Files         485      484       -1     
  Lines       92112    91407     -705     
==========================================
- Hits        81868    81767     -101     
+ Misses      10244     9640     -604     

Flag	Coverage Δ
miri	89.45% <ø> (+0.57%)	⬆️
unittests	89.10% <ø> (+0.57%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 47 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[harsha-simhadri]

Could this be in the wiki? The plots are a quite large addition to the repo.

[magdalendobson]

Could this be in the wiki? The plots are a quite large addition to the repo.

Do we want an RFC that points to benchmarks in the wiki then? Because folks were pretty clear that they wanted an RFC on filtered search.

[magdalendobson]

Could this be in the wiki? The plots are a quite large addition to the repo.

Do we want an RFC that points to benchmarks in the wiki then? Because folks were pretty clear that they wanted an RFC on filtered search.

I have moved the benchmark discussion section to the DiskANN wiki: https://github.com/microsoft/DiskANN/wiki/Evaluation-of-Filtered-Search-Algorithms

[arrayka]

LGTM. Approving, assuming the QPS and latency comparison between Adaptive L and the beta filter for disk search scenarios looks promising.

[suri-kumkaran]

Thanks for the great work. The RFC recommendations around deprecating beta filter, adding inline filtered search, and adding adaptive L search make sense to me.

One follow up implementation question is how we should expose these similar and overlapping algorithms at the right level of granularity, so clients can reuse them through providers, with or without their own query planners, depending on their use cases.