Skip to content

feat(dependencies)!: assess advisories through the SDK and CLI - #1136

Open
mldangelo-oai wants to merge 235 commits into
mainfrom
mdangelo/codex/sca-mvp
Open

mldangelo-oai wants to merge 235 commits into
mainfrom
mdangelo/codex/sca-mvp

Conversation

@mldangelo-oai

@mldangelo-oai mldangelo-oai commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Add local dependency assessment to the CLI with cs scan dependencies [repository] and to the SDK with scanDependencies(). Operators can inspect the evaluated inputs, the original OSV advisory matches, and an assessment of each match against local source. Main’s scanner import accepts existing findings. This command performs matching itself.

Changes

  • Discover supported lockfiles and manifests, retain original paths and hashes, and report incomplete coverage when an input is unavailable or a package’s origin cannot be established. Preserve OSV evidence separately from model conclusions.
  • Defer model authentication until an advisory needs assessment. Reuse the source scan’s provider, model, authentication and budget settings. Checkpoint completed assessments and verify the source snapshot before accepting a verdict.
  • Bind each verdict to the selected package, resolved version when known, source lockfile and advisory. Version comparison respects Python epochs and build suffixes in the assessment and its evaluation. A mismatched verdict leaves the scanner facts and valid sibling assessments available.
  • Add reports, comparisons, selected-match handoff, CLI help, examples and SDK types. Print a limitation that also appears in diagnostics once on stderr while retaining both fields in structured output.

Testing

  • On e93481d3, 775 affected SDK and CLI tests passed across 16 files, with one Windows-only case skipped. The 38 deterministic dependency evaluation tests and the Deep Scan worker settings test at the child-process boundary passed.
  • The offline OSV-Scanner 2.6.0 contract harness passed all 114 synthetic cases. The five portable checks passed in repository order, including all nine compatibility tests. SDK and MCP types, formatting, SDK and bundled plugin builds, and built CLI help for dependency scans and positional imports passed.
  • Reproduced the mismatched epoch, mismatched build suffix in the evaluation, and repeated stderr diagnostic before their fixes. The regression cases passed after the fixes.
  • Both complete Linux SDK package runs passed on e93481d3. The seeded run used 12345 and the default run selected 2618295243. Each recorded 13,661 passes, 172 skips, one todo, zero failures and 148,231 assertions across 328 files.
  • Earlier qualification on 109c8e16 recorded 13,429 passes, 173 skips and one todo in each of the two complete installed-package SDK test orders. These historical totals apply to that revision.

Risk and rollout

Breaking CLI change. The bare argument in cs scan dependencies selects a dependency scan of the current directory. Use cs scan ./dependencies for a source scan of a directory named dependencies.

OSV-Scanner must be installed on PATH. The default output uses persistent dependency state. A completed scan returns 0 even when it finds advisories. Partial or failed scans and budget stops return 2, and interrupts retain 130 or 143. Cost limits use estimated spend.

Online matching sends package identities to OSV. Model assessment uses local source and can leave applicability unresolved. This refresh used synthetic inputs. Live matching and live model qualification remain outstanding. Go manifests with toolchain default remain unsupported because the previously tested OSV-Scanner 2.6.0 also reads unselected go.sum entries in that case.

Public disclosure review

Reviewed the public text, changes, commit messages, discussion, available linked logs and retained artifacts. Some cited CI runs have expired artifacts, so the comprehensive historical attestation remains open.

  • No customer, partner, prospect, or user identities, data, or identifying details are included.
  • No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
  • I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.

@github-actions github-actions Bot added the enhancement New feature or request label Oct 1, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-20T23:03:33.549484Z 9532604 New commits
🔒 Security Review ✅ Completed 2026-11-20T23:04:32.073110Z 9532604 New commits

Security findings

Blocking findings (5)

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@soyeon-oai soyeon-oai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Withholding approval pending the newly added native cancellation test failure below. Reviewed the runtime modules, API integration, input/package/advisory handling, provenance and partial-result/cancellation paths, schemas and key tests. No confirmed additional production defect from this source pass. No independent execution; ecosystem correctness and all 134 changed files have not been exhaustively validated.

Comment thread sdk/typescript/tests-ts/sca-osv.test.ts Resolved
@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 87913c05d7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread sdk/typescript/src/sca-inputs.ts Resolved
Comment thread sdk/typescript/src/sca-osv.ts Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Broad cross-ecosystem process execution and model orchestration require final human review, and one misleading dependency-output error remains.

Review effort: Balanced
Findings: 1 Low severity

Open (1)
What changed in this PR

Adds SDK-level software composition analysis using an installed OSV-Scanner, followed by optional static advisory triage.

Changes:

  • Adds dependency discovery, OSV normalization, durable artifacts, assessment, reporting, comparison, and update handoff APIs.
  • Introduces versioned SCA schemas, public types, documentation, and examples.
  • Adds extensive ecosystem, process-boundary, failure-retention, and evaluation coverage.
File Description
sdk/​typescript/​tests-ts/​sca-triage.test.ts Tests triage contracts and prompts.
sdk/​typescript/​tests-ts/​sca-process-boundary.test.ts Tests executable and environment isolation.
sdk/​typescript/​tests-ts/​sca-invocations.test.ts Tests per-file invocation retention.
sdk/​typescript/​tests-ts/​sca-git-scope.test.ts Tests Git-scoped discovery.
sdk/​typescript/​tests-ts/​sca-exclusions.test.ts Tests exclusion handling.
sdk/​typescript/​tests-ts/​sca-ecosystems.test.ts Tests cross-ecosystem semantics.
sdk/​typescript/​src/​thread-source.ts Adds dependency-triage thread source.
sdk/​typescript/​src/​targets.ts Exposes configurable Git isolation.
sdk/​typescript/​src/​sca.ts Builds and persists SCA results.
sdk/​typescript/​src/​sca-types.ts Defines generated SCA contracts.
sdk/​typescript/​src/​sca-triage.ts Implements model triage contract.
sdk/​typescript/​src/​runtime.ts Adds persistent dependency outputs.
sdk/​typescript/​src/​index.ts Exports the public SCA API.
sdk/​typescript/​src/​api.ts Implements scanDependencies.
sdk/​typescript/​scripts/​sca-osv-ecosystem-fixtures.mts Defines ecosystem scanner fixtures.
sdk/​typescript/​scripts/​generate-sca-models.cjs Generates SCA TypeScript models.
sdk/​typescript/​scripts/​check-package.mjs Verifies packaged SCA modules.
sdk/​typescript/​README.md Documents SCA usage and limitations.
sdk/​typescript/​pnpm-lock.yaml Promotes saxes to runtime.
sdk/​typescript/​package.json Adds generation and runtime dependency changes.
plugins/​codex-security/​skills/​triage-finding/​references/​triage-result-contract.md Documents the shared schema.
plugins/​codex-security/​schemas/​triage-result.schema.json Promotes the triage schema.
plugins/​codex-security/​schemas/​sca-result.schema.json Defines persisted SCA results.
plugins/​codex-security/​plugin-files.json Packages the new schemas.
examples/​sca/​run.mjs Adds a runnable scan example.
examples/​sca/​README.md Documents SCA examples.
examples/​sca/​handoff.mjs Demonstrates update handoffs.
examples/​sca/​compare.mjs Demonstrates result comparison.
examples/​sca/​ci.yml Shows CI artifact integration.
evals/​triage-finding/​sca/​scripts/​sca-result.js Scores SCA evaluations.
evals/​triage-finding/​sca/​scripts/​run-promptfoo.js Stages isolated evaluation runs.
evals/​triage-finding/​sca/​scripts/​generate-tests.js Generates synthetic evaluation cases.
evals/​triage-finding/​sca/​scripts/​baselines.js Produces scanner baselines.
evals/​triage-finding/​sca/​README.md Documents evaluation methodology.
evals/​triage-finding/​sca/​QA.md Records evaluation QA.
evals/​triage-finding/​sca/​prompts/​triage-baseline.txt Defines the baseline prompt.
evals/​triage-finding/​sca/​prompts/​sca-wrapper.txt Defines the typed-match prompt.
evals/​triage-finding/​sca/​promptfooconfig.yaml Configures model evaluation.
evals/​triage-finding/​sca/​fixtures/​sca-001/​src/​application.mjs Adds affected depth-parser usage.
evals/​triage-finding/​sca/​fixtures/​sca-001/​package.json Declares fixture dependency.
evals/​triage-finding/​sca/​fixtures/​sca-001/​package-lock.json Locks fixture dependency.
evals/​triage-finding/​sca/​fixtures/​sca-001/​osv.json Captures synthetic advisory data.
evals/​triage-finding/​sca/​fixtures/​sca-001/​input.json Defines typed advisory input.
evals/​triage-finding/​sca/​fixtures/​sca-001/​APPLICATION.md Defines fixture deployment context.
evals/​triage-finding/​sca/​fixtures/​sca-002/​src/​application.mjs Adds mitigated depth-parser usage.
evals/​triage-finding/​sca/​fixtures/​sca-002/​package.json Declares fixture dependency.
evals/​triage-finding/​sca/​fixtures/​sca-002/​package-lock.json Locks fixture dependency.
evals/​triage-finding/​sca/​fixtures/​sca-002/​osv.json Captures synthetic advisory data.
evals/​triage-finding/​sca/​fixtures/​sca-002/​input.json Defines typed advisory input.
evals/​triage-finding/​sca/​fixtures/​sca-002/​APPLICATION.md Defines fixture deployment context.
evals/​triage-finding/​sca/​fixtures/​sca-003/​src/​application.mjs Adds unresolved depth-parser usage.
evals/​triage-finding/​sca/​fixtures/​sca-003/​package.json Declares fixture dependency.
evals/​triage-finding/​sca/​fixtures/​sca-003/​package-lock.json Locks fixture dependency.
evals/​triage-finding/​sca/​fixtures/​sca-003/​osv.json Captures synthetic advisory data.
evals/​triage-finding/​sca/​fixtures/​sca-003/​input.json Defines typed advisory input.
evals/​triage-finding/​sca/​fixtures/​sca-003/​APPLICATION.md Defines fixture deployment context.
evals/​triage-finding/​sca/​fixtures/​sca-004/​src/​application.mjs Adds affected template usage.
evals/​triage-finding/​sca/​fixtures/​sca-004/​package.json Declares fixture dependency.
evals/​triage-finding/​sca/​fixtures/​sca-004/​package-lock.json Locks fixture dependency.
evals/​triage-finding/​sca/​fixtures/​sca-004/​osv.json Captures synthetic advisory data.
evals/​triage-finding/​sca/​fixtures/​sca-004/​input.json Defines typed advisory input.
evals/​triage-finding/​sca/​fixtures/​sca-004/​APPLICATION.md Defines fixture deployment context.
evals/​triage-finding/​sca/​fixtures/​sca-005/​src/​application.mjs Adds mitigated template usage.
evals/​triage-finding/​sca/​fixtures/​sca-005/​package.json Declares fixture dependency.
evals/​triage-finding/​sca/​fixtures/​sca-005/​package-lock.json Locks fixture dependency.
evals/​triage-finding/​sca/​fixtures/​sca-005/​osv.json Captures synthetic advisory data.
evals/​triage-finding/​sca/​fixtures/​sca-005/​input.json Defines typed advisory input.
evals/​triage-finding/​sca/​fixtures/​sca-005/​APPLICATION.md Defines fixture deployment context.
evals/​triage-finding/​sca/​fixtures/​sca-006/​src/​application.mjs Adds unresolved template usage.
evals/​triage-finding/​sca/​fixtures/​sca-006/​package.json Declares fixture dependency.
evals/​triage-finding/​sca/​fixtures/​sca-006/​package-lock.json Locks fixture dependency.
evals/​triage-finding/​sca/​fixtures/​sca-006/​osv.json Captures synthetic advisory data.
evals/​triage-finding/​sca/​fixtures/​sca-006/​input.json Defines typed advisory input.
evals/​triage-finding/​sca/​fixtures/​sca-006/​APPLICATION.md Defines fixture deployment context.
evals/​triage-finding/​sca/​fixtures/​sca-007/​src/​application.mjs Adds affected cache usage.
evals/​triage-finding/​sca/​fixtures/​sca-007/​package.json Declares fixture dependency.
evals/​triage-finding/​sca/​fixtures/​sca-007/​package-lock.json Locks fixture dependency.
evals/​triage-finding/​sca/​fixtures/​sca-007/​osv.json Captures synthetic advisory data.
evals/​triage-finding/​sca/​fixtures/​sca-007/​input.json Defines typed advisory input.
evals/​triage-finding/​sca/​fixtures/​sca-007/​APPLICATION.md Defines fixture deployment context.
evals/​triage-finding/​sca/​fixtures/​sca-008/​src/​application.mjs Adds mitigated cache usage.
evals/​triage-finding/​sca/​fixtures/​sca-008/​package.json Declares fixture dependency.
evals/​triage-finding/​sca/​fixtures/​sca-008/​package-lock.json Locks fixture dependency.
evals/​triage-finding/​sca/​fixtures/​sca-008/​osv.json Captures synthetic advisory data.
evals/​triage-finding/​sca/​fixtures/​sca-008/​input.json Defines typed advisory input.
evals/​triage-finding/​sca/​fixtures/​sca-008/​APPLICATION.md Defines fixture deployment context.
evals/​triage-finding/​sca/​fixtures/​sca-009/​src/​application.mjs Adds unresolved cache usage.
evals/​triage-finding/​sca/​fixtures/​sca-009/​package.json Declares fixture dependency.
evals/​triage-finding/​sca/​fixtures/​sca-009/​package-lock.json Locks fixture dependency.
evals/​triage-finding/​sca/​fixtures/​sca-009/​osv.json Captures synthetic advisory data.
evals/​triage-finding/​sca/​fixtures/​sca-009/​input.json Defines typed advisory input.
evals/​triage-finding/​sca/​fixtures/​sca-009/​APPLICATION.md Defines fixture deployment context.
evals/​triage-finding/​sca/​fixtures/​sca-010/​src/​application.mjs Adds affected filter usage.
evals/​triage-finding/​sca/​fixtures/​sca-010/​package.json Declares fixture dependency.
evals/​triage-finding/​sca/​fixtures/​sca-010/​package-lock.json Locks fixture dependency.
evals/​triage-finding/​sca/​fixtures/​sca-010/​osv.json Captures synthetic advisory data.
evals/​triage-finding/​sca/​fixtures/​sca-010/​input.json Defines typed advisory input.
evals/​triage-finding/​sca/​fixtures/​sca-010/​APPLICATION.md Defines fixture deployment context.
evals/​triage-finding/​sca/​fixtures/​sca-011/​src/​application.mjs Adds mitigated filter usage.
evals/​triage-finding/​sca/​fixtures/​sca-011/​package.json Declares fixture dependency.
evals/​triage-finding/​sca/​fixtures/​sca-011/​package-lock.json Locks fixture dependency.
evals/​triage-finding/​sca/​fixtures/​sca-011/​osv.json Captures synthetic advisory data.
evals/​triage-finding/​sca/​fixtures/​sca-011/​input.json Defines typed advisory input.
evals/​triage-finding/​sca/​fixtures/​sca-011/​APPLICATION.md Defines fixture deployment context.
evals/​triage-finding/​sca/​fixtures/​sca-012/​src/​application.mjs Adds unresolved filter usage.
evals/​triage-finding/​sca/​fixtures/​sca-012/​package.json Declares fixture dependency.
evals/​triage-finding/​sca/​fixtures/​sca-012/​package-lock.json Locks fixture dependency.
evals/​triage-finding/​sca/​fixtures/​sca-012/​osv.json Captures synthetic advisory data.
evals/​triage-finding/​sca/​fixtures/​sca-012/​input.json Defines typed advisory input.
evals/​triage-finding/​sca/​fixtures/​sca-012/​APPLICATION.md Defines fixture deployment context.
evals/​triage-finding/​sca/​corpus/​manifest.json Plans human-adjudicated corpus work.
evals/​triage-finding/​sca/​corpus/​developer-pilot.csv Adds pilot recording template.
evals/​triage-finding/​sca/​corpus/​adjudication-template.json Adds adjudication template.
evals/​triage-finding/​sca/​assertions/​sca-metrics.js Records evaluation metrics.
evals/​triage-finding/​sca/​assertions/​sca-evidence.js Validates verdicts and citations.
evals/​triage-finding/​sastbench/​promptfooconfig.sastbench.yaml Uses the promoted schema.
evals/​triage-finding/​README.md Updates schema documentation.
evals/​triage-finding/​promptfooconfig.yaml Uses the promoted schema.
evals/​triage-finding/​promptfooconfig.calibration.yaml Uses the promoted schema.
evals/​triage-finding/​promptfooconfig.calibration-smoke.yaml Uses the promoted schema.
evals/​triage-finding/​package.json Adds deterministic and model SCA scripts.
docs/​sca-qa.md Documents implementation verification and risks.
Files not reviewed (13)
  • evals/triage-finding/sca/fixtures/sca-001/package-lock.json: Generated file
  • evals/triage-finding/sca/fixtures/sca-002/package-lock.json: Generated file
  • evals/triage-finding/sca/fixtures/sca-003/package-lock.json: Generated file
  • evals/triage-finding/sca/fixtures/sca-004/package-lock.json: Generated file
  • evals/triage-finding/sca/fixtures/sca-005/package-lock.json: Generated file
  • evals/triage-finding/sca/fixtures/sca-006/package-lock.json: Generated file
  • evals/triage-finding/sca/fixtures/sca-007/package-lock.json: Generated file
  • evals/triage-finding/sca/fixtures/sca-008/package-lock.json: Generated file
  • evals/triage-finding/sca/fixtures/sca-009/package-lock.json: Generated file
  • evals/triage-finding/sca/fixtures/sca-010/package-lock.json: Generated file
  • evals/triage-finding/sca/fixtures/sca-011/package-lock.json: Generated file
  • evals/triage-finding/sca/fixtures/sca-012/package-lock.json: Generated file
  • sdk/typescript/pnpm-lock.yaml: Generated file

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread sdk/typescript/src/runtime.ts Resolved
@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 24763698a3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread sdk/typescript/src/sca-inputs.ts Resolved
Comment thread sdk/typescript/src/api.ts Resolved

@alandelong-oai alandelong-oai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks—the Pipenv fixture now declares its public source, and the maintained OSV-Scanner 2.6.0 harness passes all 114 cases. This owner-level merge-safety review of 09b02507 found one additional issue in the full dependency-command contribution: a symlink followed by .. can select a different repository, as detailed inline. The fixture-only follow-up is sound.

Fresh validation passed 726 SCA/CLI tests (one Windows-only skip), 38 deterministic evals, 41 Deep Scan leaf cases, and nine portable tests; builds, types, formatting, and built CLI checks passed. The path issue was reproduced through source and built CLI modules with real filesystem paths and the SDK call intercepted before scanning. No full SDK suite, live model/provider assessment, or Windows execution is claimed.

Comment thread sdk/typescript/src/cli.ts Resolved

@alandelong-oai alandelong-oai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix. This owner-level merge-safety review of 7eb287dd found no material unresolved issue. Dependency scans now preserve physical symlink/.. repository selection through the shared path helper. The documented breaking scan dependencies selector remains intentional.

Fresh validation passed 726 dependency/CLI tests (one Windows-only skip), 114 offline OSV contract cases, 38 deterministic evaluations, 41 Deep Scan leaf cases, and nine source-checker tests. Builds, types, formatting, eight source/built Node CLI path controls, and eight independent resolver checks passed.

The broader shared path fixture remains blocked by local default-state-directory permissions after its dependency assertion. No full SDK suite, live advisory-service/model assessment, or native Windows/macOS execution is claimed.

@alandelong-oai alandelong-oai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the thorough implementation. No blocking findings after an owner-level merge-safety review of all 156 changed files at 109c8e1, including the current main integration and resolved discussion. The intentional reservation of scan dependencies is documented; literal -- still selects a source path. Advisory facts remain separate from static assessments, with source identity, credential boundaries and partial-result checkpoints preserved.

Validation passed: 774 focused tests (one Windows-only skip), 228 CLI dispatch tests, 38 deterministic evals, 41 Deep Scan cases, 114 real OSV 2.6.0 offline contracts, eight source/built path checks, and a compiled CLI → real OSV → saved-artifacts check. Fresh builds, generated models, types, Ruff and formatting passed. The broader shared suite had 445 passes, 37 platform skips and three failures; all three reproduce on the exact base, and the two Python-path assertions pass on both commits with ordinary Python discovery.

Limits: no full suite, native Windows/macOS execution, live advisory/model run, or installed-package smoke. Broad eval type checking could not resolve the separate yaml/promptfoo dependencies. CI currently has 74 successful checks and two still running. Approving with those validation limits recorded.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Unknown error
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

mldangelo-oai added a commit that referenced this pull request Nov 20, 2026
mldangelo-oai added a commit that referenced this pull request Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking-change Breaking change called out in generated release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants