Skip to content

Protect the source authority register behind the RAG impact gate, and log the red canary - #2732

Open
claude[bot] wants to merge 1 commit into
mainfrom
claude/protect-source-authority-registry
Open

Protect the source authority register behind the RAG impact gate, and log the red canary#2732
claude[bot] wants to merge 1 commit into
mainfrom
claude/protect-source-authority-registry

Conversation

@claude

@claude claude Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Two pieces of RAG-safeguard hygiene found while preparing Register nine Australian clinical publishers in the source authority register #2711.
  • The gate did not ask. scripts/pr-policy.mjs classified src/lib/source-authority-registry.ts as an ordinary file, so Register nine Australian clinical publishers in the source authority register #2711 carried its RAG impact: line voluntarily rather than because the check demanded it. That file and src/lib/australian-source-priority.ts are on the ordering path: classifySourceAuthority assigns a result's tier, australianSourceTier reads it, and selectAustralianClinicalContext orders and trims the model's context by it. Registering or re-scoping a publisher moves that ordering without touching a score, which is precisely the shape this gate exists to catch. Both are now in ragRankingPatterns, asserted in the self-test, and listed in the protected-surface table in docs/rag-behaviour/safeguards.md.
  • The safeguards note is explicit that this is authority classification, not a governance ranking penalty or boost. That shape stays refuted (refuted-approaches.md § Refutation 3, ledger #032), and saying so here stops a later reader mistaking this entry for permission to revisit it.
  • The canary has been red for a fortnight and nothing tracked it. Adds one outstanding-issues inbox request. Golden retrieval passed 36/36 with both recalls at 1.0 in every one of the last five canary runs. All five are red purely on the separate live answer-quality subset, on the same three cases (clozapine-monitoring, clozapine-monitoring-paraphrase, nocc-requirements, source_backed_review_fallback unaccounted), going back to run 32656925301 on 2026-08-23. The request records the evidence, points at #J8SJQ9 as the overlapping row to check first, and explicitly forbids the two tempting non-fixes: adding acceptSourceOnly to the eval cases, or relaxing the threshold.

Filed as an inbox request, not a direct ledger-table edit, per the write-discipline rule. Run npm run issues:reconcile from a dedicated ledger branch after this lands.

Why this is not just paperwork

The canary is the pair-comparison instrument for every protected RAG change, including the one #2711 is about to be measured with. A permanently red run trains readers to skim past it, and a real retrieval regression would then land inside an already-red signal.

Verification

  • npm run check:pr-policy — self-test passed, workflow guard passed. The two new assertions fail against the old pattern list, so the change is red-proven.
  • npm run check:ledger-write-discipline — passed for 0177bed18446..HEAD. This gate compares committed refs, so it was re-run after the commit, not before.
  • npm run check:outstanding-issues — snapshot in step, 121 open, 0 pending.
  • npm run docs:check-links — 7392 repo path references resolve.
  • npx prettier --check . — whole tree clean, formatting committed.
  • npm run verify:ui — not applicable, no UI, routing or styling change.
  • npm run eval:retrieval:qualityVerification not run: not applicable. No retrieval, ranking, selection or scoring code path changes here. This PR only changes which files the PR-body gate classifies, plus documentation and one ledger request.

Risk and rollout

  • Risk: low. The only executable change is a regex added to a classifier list in the PR-policy script. Its effect is that future PRs touching two named files must carry a RAG impact: line. It cannot change retrieval, ranking, answers, or anything at runtime.
  • The tightening is deliberate and will make some future PRs fail until their author adds the declaration. That is the point.
  • Rollback: single-commit revert. Nothing depends on the new patterns.
  • Provider or production effects: None.
  • RAG impact: no retrieval behaviour change — gate classification, documentation, and one ledger request only; no retrieval, ranking, selection, scoring or eval code executes differently, and the imputation formulas and release-comparator key order are untouched.

Clinical Governance Preflight

  • Source-backed claims still require linked source verification before clinical use
  • No patient-identifiable document workflow was introduced or expanded without explicit governance approval
  • Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy)
  • Service-role keys and private document access remain server-only
  • Demo/synthetic content remains clearly separated from real clinical sources
  • Source metadata, review status, and outdated/unknown-source behavior remain conservative
  • Deployment classification/TGA SaMD impact was checked when clinical decision-support behavior changed

This PR adds a guard and a tracking record. It removes no control and relaxes no threshold.

Notes

🤖 Generated with Claude Code

https://claude.ai/code/session_01DXnbx8Pa4VMp2dzBrYFH4J


Generated by Claude Code

… log the red canary

Two pieces of RAG-safeguard hygiene found while preparing PR #2711.

pr-policy classified `src/lib/source-authority-registry.ts` as an ordinary
file, so PR #2711 declared its RAG impact voluntarily rather than because the
gate asked. That file and `src/lib/australian-source-priority.ts` do sit on the
ordering path: classifySourceAuthority assigns a result's tier,
australianSourceTier reads it, and selectAustralianClinicalContext orders and
trims the model's context by it. Registering or re-scoping a publisher moves
that ordering without touching a score, which is exactly the shape the gate
exists to catch. Both are now in ragRankingPatterns, asserted in the self-test,
and listed in the protected-surface table.

This is authority classification, not a governance ranking penalty or boost.
That shape stays refuted and the safeguards note says so, so a later reader
cannot mistake this entry for permission to revisit Refutation 3.

The second piece is an outstanding-issues inbox request. The eval canary has
been red on the same three answer-quality cases since at least 2026-08-23, over
five consecutive runs, while golden retrieval passed 36/36 in every one. Nothing
in the ledger tracked it. A permanently red canary is a poor instrument to
validate the next protected change against.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DXnbx8Pa4VMp2dzBrYFH4J
@claude
claude Bot requested a review from BigSimmo as a code owner September 7, 2026 11:32
@supabase

supabase Bot commented Sep 7, 2026

Copy link
Copy Markdown

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.


Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant