Flag studies the project already has when adding, and fix the matching edge cases

[InfinityBowman]

Closes #820.

What changed

Staged studies are now matched against the studies the project already holds. A match is flagged, never acted on: the row names the study it duplicates and why, sits out of the upload by default, and carries an Add anyway toggle.

That restraint is deliberate. One production project deliberately builds 14 studies from 4 byte-identical PDFs, one study per site the report covers, so auto-skipping or auto-merging would have broken a live workflow.

Matching, rewritten against real data

I pulled 90 PDFs out of production projects and ran the real extractors over them offline before changing anything. The issue's premise turned out to be slightly off: the filename fallback it worried about essentially never fires, and the real hazard is junk titles, which 27% of PDFs produce.

• matchEntries reports how two entries match (doi, file, title, title-substring) instead of returning a boolean
• Two DOIs settle it both ways, so different DOIs are different papers whatever the titles say
• PDF byte size identifies a file. pdfs.size matched R2 bytes 90/90 with zero collisions, so no hashing and no schema change
• isWeakTitle rejects the boilerplate extraction actually produces, so untitled and RE S E AR C H RE P O R T no longer merge unrelated papers
• Substring matching catches titles cut off where they wrapped, and studies named Author Year- <title>
• No fuzzy matching. I tried it; it merges sibling drug trials that differ by a few words, and it is not needed once the above is in place

Merging: a file-only match no longer merges, so the multi-site workflow keeps one study per site. A second distinct PDF is attached via extraPdfs rather than dropped. Merged rows report the fields their sources disagree on.

Bugs fixed along the way

• Retrying a failed extraction re-read the file through handlePdfSelect, which filtered it out as already staged and left the row spinning forever
• DOIs broken across a line break resolved to nothing while still looking authoritative; they are rejoined before the regex runs (1 of 90 PDFs)
• Crossref registers some titles as escaped JATS, so a study could be named <p>MRI Changes After.... Entities are decoded before tags are stripped (1 of 43 resolvable DOIs)
• The canonical title from the DOI lookup is used when the page gave nothing usable or only a truncated prefix
• During submit, staged rows briefly flagged themselves against the studies they were creating. The check now compares against a snapshot taken when upload starts

Verified against production

	Before	After
Re-upload of a PDF the project holds (90 real PDFs)	0 detected	90 / 90
Flagged against an unrelated study	n/a	0
Unrelated same-title pairs merged	all	0
Study named with the published title (41 resolvable)	24	40
Anderson 2017 multi-site batch	1 study, 3 PDFs lost	5 studies, 5 PDFs
DOIs changed by the extractor fix	n/a	1 of 90

Re-importing all 23 PDFs of a real project flags 23 of 23 (19 by DOI, 4 by file) and adds nothing silently. Matching 50 staged rows against a 300-study project takes 10ms, so no indexing was added.

Testing

duplicates.test.ts is new and its fixtures are strings production actually produced: the junk titles extraction returns, and the published titles those papers carry. 681 web, 322 web-server and 127 workers tests pass; typecheck and oxlint clean.

Notes for review

• collectMode in useAddStudies and AddStudiesForm is now dead code, since CreateProjectModal no longer adds studies. That makes the issue's third open question moot. Flagged rather than deleted.
• StudyMetadata.authors is typed string[] but the lookup returns a formatted string and addBatch casts it back. Pre-existing; I left one localised cast with a comment rather than widening the change.
• There is an orphaned pdfs row in one production project pointing at a missing R2 key. Unrelated to this work.

https://claude.ai/code/session_01JySapHzgzwPW1yBWHccZYf

Summary by CodeRabbit

• New Features

  • Detects studies already in a project and clearly identifies why they match.
  • Shows duplicate, merged-source, extra-PDF, and conflicting-information details during upload.
  • Lets you add flagged duplicates intentionally or undo that choice.
  • Displays accurate counts of studies ready to add and those already present.
  • Supports uploading multiple PDFs merged into one study.
  • Improves DOI and publication-title extraction, including split DOIs and formatted metadata.

• Bug Fixes

  • Prevents duplicate files from being staged repeatedly and provides clearer empty-state messages.

[coderabbitai]

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: d1684f1f-4606-45e3-b53a-8cf1ef1333d4

📥 Commits

Reviewing files that changed from the base of the PR and between 8cf6987 and 0a9737c.

📒 Files selected for processing (18)

• packages/web/src/components/project/add-studies/AddStudiesForm.tsx
• packages/web/src/components/project/add-studies/AddStudiesSheet.tsx
• packages/web/src/components/project/add-studies/StagedStudiesSection.tsx
• packages/web/src/components/project/all-studies-tab/AllStudiesTab.tsx
• packages/web/src/hooks/__tests__/useAddStudies/deduplication.test.ts
• packages/web/src/hooks/__tests__/useAddStudies/duplicates.test.ts
• packages/web/src/hooks/__tests__/useAddStudies/matching.test.ts
• packages/web/src/hooks/useAddStudies/deduplication.ts
• packages/web/src/hooks/useAddStudies/existing.ts
• packages/web/src/hooks/useAddStudies/index.ts
• packages/web/src/hooks/useAddStudies/matching.ts
• packages/web/src/hooks/useAddStudies/pdfs.ts
• packages/web/src/lib/__tests__/pdfUtils.test.ts
• packages/web/src/lib/__tests__/referenceLookup.test.ts
• packages/web/src/lib/pdfUtils.ts
• packages/web/src/lib/referenceLookup.ts
• packages/web/src/project/actions/studies.ts
• packages/web/src/project/workspace-data.ts

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}