Skip to content

feat: Filter tool declarations and recheck eligibility before execution - #6143

Open
markstuart-oai wants to merge 3 commits into
feat/simple-codexfrom
feat/prepared-tool-eligibility
Open

markstuart-oai wants to merge 3 commits into
feat/simple-codexfrom
feat/prepared-tool-eligibility

Conversation

@markstuart-oai

@markstuart-oai markstuart-oai commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

This pull request adds an experimental Runner policy for direct functions, sandbox capability functions and tools discovered through agent.mcp_servers. For example, an application can withhold a publish tool until an inspection succeeds and revoke access while an approved call waits to run.

Summary

Set ToolExecutionConfig.prepare_tool_declaration to an async callback. It receives the run context and an immutable ToolDeclaration. Return a description to offer the tool or None to withhold it. The selection governs both the model request and the calls the Runner accepts. The callback cannot replace the executor, schema or approval policy.

The Runner rechecks eligibility before execution, including after queueing, approval and MCP metadata resolution. Withholding a named tool choice or leaving a required choice with no tools raises an error before the model request.

With this policy enabled, a function that allows both direct and programmatic callers raises UserError during selection. Programmatic-only functions stay outside the policy. Changing their caller list after selection cannot grant direct access for that request.

For sandbox functions, the Runner checks the session's tool inventory and passes the public agent to is_enabled. Removing a direct function from that agent blocks execution even if the sandbox still lists it. A pending capability approval records the capability type, position and tool key as a digest. An approval cannot move between a capability and a direct function with the same name.

Keep the callback free of side effects and supply it again on resume. Restoring a serialized approval relies on the application's current authorized tools. The policy does not order parallel calls or cover provider-managed tools. It is off by default.

Test plan

  • Exercised both Runner modes, forced choices, revocation and MCP discovery on both supported MCP versions.
  • Verified live and serialized approvals for sandbox shell and patch functions. A changed patch approval policy blocked the write. Missing capability provenance blocked view_image.
  • Verified that mixed callers stop before sampling. Checked caller changes during selection, in on_llm_start and across live and serialized approvals.
  • Checked streamed argument isolation, runtime type hints and Pydantic validation.
  • Ran the full local verification script (13,335 passed, 37 skipped) and checked the declaration links, caller restrictions and sandbox guidance in all four versions of the generated API reference.

Checks

  • Added regression tests.
  • Ran .agents/skills/code-change-verification/scripts/run.sh.
  • Confirmed all local verification steps passed.
  • Completed independent review of the full change.

@markstuart-oai
markstuart-oai requested review from a team, rm-openai and seratch as code owners November 21, 2026 14:53
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-21T16:04:20.453671Z 8a26839 New commits
🔒 Security Review ✅ Completed 2026-11-21T16:04:41.651690Z 8a26839 New commits

Security findings

Advisory findings (1)

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4d1a13abde

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@jbeckwith-oai jbeckwith-oai left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes at 4d1a13abde47568fb17ac93d55ebb260e3aae32e. I independently confirmed the existing P2 sandbox inventory finding. prepare_sandbox_agent() adds capability FunctionTools such as ExecCommandTool to a cloned execution agent, while both run loops pass the public agent to prepare_declarations(). The new membership check therefore rejects these tools even with a pass-through declaration callback, before the model request. Please validate against the resolved execution inventory while preserving public agent identity for callbacks, and add regressions for SandboxAgent capability tools in both Runner modes, including approval/resume.

Reviewed the remaining declaration binding, forced-choice, stream isolation, MCP filtering and resume paths with an independent review; no additional actionable findings. Referencing the existing inline finding rather than duplicating it. Hosted CI is complete: 33 checks passed and 4 skipped; no failures. The sandbox inventory finding above remains unresolved. Static review only; no candidate code or local tests executed.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛡️ Codex Security Review · Automatically triggered

Here are some automated security review suggestions for this pull request.

Reviewed commit: 464021ccec

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@jbeckwith-oai jbeckwith-oai left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by request changes: confirmed mixed direct/programmatic caller policy gap.

Approved after static re-review and two independent reviews. The sandbox inventory finding is fixed: declaration preparation validates the execution inventory while preserving public agent identity for callbacks. Reviewed capability source tracking through preparation and caching, namespaced tool copy semantics, exact method/type/schema/approval rebinding, the SDK patch adapter, serialized capability-source digests, and direct-tool revocation. Pending approvals cannot move between direct tools and capabilities or change capability position/type/key. Regression coverage exercises streamed and nonstreamed execution, in-memory and serialized resume, replacement executors and approval policies, and source changes. No remaining actionable findings.

Hosted CI snapshot: 17 passed and 7 running; remaining checks require follow-up. Static review only; no candidate code or local tests executed.

@jbeckwith-oai jbeckwith-oai left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: supersedes my approval at this commit. I missed the mixed-caller gap identified in the existing P2 finding; independent static checks now confirm it. _covered excludes every function containing programmatic in allowed_callers, including functions also allowing direct calls. Such a tool remains advertised without invoking prepare_tool_declaration. A call with no caller metadata is accepted as direct, and the missing prepared binding skips the execution-time policy recheck. Cover mixed tools for direct invocation, or reject this configuration when declaration preparation is enabled, with a denial regression. Referencing the existing finding rather than duplicating it.

The sandbox inventory and approval/resume fixes remain sound. Hosted CI completed: 33 passed, 10 skipped. The mixed-caller correctness finding remains unresolved. Static review only; no candidate code or local tests executed.

@jbeckwith-oai jbeckwith-oai left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after static re-review and an independent second review. The mixed-caller bypass identified in the previous review is fixed: enabling declaration preparation rejects mixed direct/programmatic tools, including live and restored approvals. Inventory validation prevents unprepared programmatic-only functions from gaining direct access during selection or request hooks. Reviewed caller-list detachment, both Runner modes, active empty-inventory versus disabled-policy handling, and the denial regressions. No new actionable findings.

Hosted CI completed: 33 passed, 8 skipped, no failures. The macOS build and downstream installed-wheel verification succeeded after the earlier Rust-download DNS failure. Static review only; no candidate code or local tests executed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants