Skip to content

feat: let agent hooks request another turn before accepting a final answer - #6124

Open
markstuart-oai wants to merge 2 commits into
feat/simple-codexfrom
feat/settled-step-control
Open

markstuart-oai wants to merge 2 commits into
feat/simple-codexfrom
feat/settled-step-control

Conversation

@markstuart-oai

@markstuart-oai markstuart-oai commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

This pull request adds an optional agent hook to reject premature answers. For example, one tool measures a width and another computes the area. The hook can require both results before the run ends.

Use Agent(name="Surveyor", hooks=AgentHooks(step_policy=decide)) to check the answer against application state. The callback receives the run context and a SettledStep. It can return "continue" to request another turn or "finish" to accept the answer. Both synchronous and asynchronous callbacks work in streamed and non-streamed runs.

The hook runs once per proposed answer, after pending approvals resolve. Handoffs take precedence. Eligible answers include text, parsed structured output and tool results selected by tool_use_behavior. Without the hook, existing behavior stays the same.

Continuation skips end hooks and keeps the rejected answer in history. Stream events can include that provisional answer. Dynamic instructions can guide the next turn using the updated context. Turn limits still apply, and accepted answers still go through output guardrails.

Test plan

  • Two dependent tools and premature answers in both run modes.
  • Approval and resume, handoff precedence, guardrails, cancellation, and missing versus empty text.
  • All 24 policy tests passed on Python 3.10, including the six cancellation cases.
  • Local repository verification with 13,243 tests passed and 37 skipped.

Checks

  • Added regression tests
  • Ran .agents/skills/code-change-verification/scripts/run.sh
  • Passed formatting, lint, mypy, Pyright and tests
  • Completed the repository's independent implementation review

…nswer

Signed-off-by: markstuart-oai <323302876+markstuart-oai@users.noreply.github.com>
@markstuart-oai
markstuart-oai requested review from a team, rm-openai and seratch as code owners November 21, 2026 05:54
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-21T15:57:30.516280Z 0fbeceb Manual request
🔒 Security Review ✅ Completed 2026-11-21T15:57:36.589838Z 0fbeceb Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@jbeckwith-oai jbeckwith-oai left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by request changes: Python 3.10 cancellation assertion failures.

Approved after comprehensive static review and an independent second review. Reviewed text, structured and final-tool candidates; missing/empty content and refusal handling; approval/resume and handoff precedence; turn numbering and limits; provisional history; error-handler bypass; accepted-output guardrails and end hooks; and cancellation/error propagation. Completed tool effects are checkpointed before resumed policy evaluation and retained on policy failure, while parent cancellation joins the policy task without accepting a late finish. Existing behavior remains unchanged without a policy. No actionable findings.

Hosted check snapshot: {"SUCCESS": 14, "IN_PROGRESS": 9, "FAILURE": 1}. Pending checks require follow-up. Static review only; no candidate code or local tests executed.

@jbeckwith-oai jbeckwith-oai left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: supersedes my approval at this commit. I published that approval before inspecting the newly completed Python 3.10 failure. The hosted job reports six failures, all in new cancellation-message assertions: expected decision failed or local, actual empty string. Make these assertions compatible with supported Python 3.10 while retaining cancellation-type, checkpoint/session persistence and no-replay checks. No additional production correctness findings from the comprehensive static and independent reviews. Hosted CI is now terminal: 31 passed, 2 failed (the known Python 3.10 jobs), and 6 skipped. The request-changes decision remains in effect. No candidate code or local tests executed.

Comment thread tests/test_step_policy.py Resolved

@jbeckwith-oai jbeckwith-oai left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after static re-review and an independent second review. The Python 3.10 cancellation-test finding is fixed: only cancellation-message matching is relaxed on Python versions below 3.11, while RuntimeError messages, cancellation types, suppressed final hooks, exactly-once tool effects, resumed output, and persisted session results remain checked. The incremental change is limited to these tests; no remaining actionable findings.

Hosted CI is complete: 33 passed and 2 skipped; no failures. Static review only; no candidate code or local tests executed.

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 0fbeceb4f8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 0fbeceb4f8

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants