Skip to content

feat(codex): resume pending tool approvals and trace native calls - #6106

Merged
markstuart-oai merged 44 commits into
feat/simple-codexfrom
feature/codex-operation-tracing
Nov 22, 2026
Merged

markstuart-oai merged 44 commits into
feat/simple-codexfrom
feature/codex-operation-tracing

Conversation

@markstuart-oai

@markstuart-oai markstuart-oai commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Workers have been losing pending approvals for built-in commands and file edits on restart. Codex has been keeping these requests only in memory. SDK traces have also been missing the final model request and the time spent running a tool after approval.

Changes

  • Resume a built-in approval. Pass a resumable session to Runner.run(..., session=...). Load the same call on a new worker, then approve or reject it. One attempt per decision, with fresh permission and file checks and confirmation of command exit.
  • Trace native calls. Start timing a tool after approval. Export metadata by default and text only by explicit opt-in.
  • Build and test the SDK wheel. Apply each patch to a temporary copy of Codex. Require a guide for every patch. Keep tests in regular SDK files.

Limits and checks

Use built-in recovery with one local workspace, human review and no native tool hooks. After a crash during a write, check the result before retrying.

We've been testing the Linux CI wheel across 19 local Kubernetes pod replacements with Redis, checking approvals, denials, duplicate delivery, stale edits and OpenTelemetry output. In separate workers, we've been testing late command exit notices and crashes during a write.

Dependencies · #6102, #6103, #6105 and #6115

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-21T15:42:47.601296Z e74c1ee New commits
🔒 Security Review ✅ Completed 2026-11-21T15:42:51.201317Z e74c1ee New commits

Security findings

Advisory findings (4)

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@jbeckwith-oai jbeckwith-oai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved the incremental Codex operation-tracing changes after static review and an independent second review. Reviewed the Python adapter, Rust patch, private connection/turn ownership, capture negotiation, tool authorization boundaries, source timing, retry grouping, interrupted-operation cleanup, and regression coverage. No new actionable findings in this diff.

This PR is stacked on #6105; its outstanding review blockers still need to be fixed in the base. Approval here does not clear those findings.

Hosted checks: {"SUCCESS": 16, "FAILURE": 3}. Python 3.10 fails the new timestamp regression (introduced in #6105); both Windows jobs still have the known UnixLocalSandbox collection failures. No candidate code or local tests were executed; the patched Codex binary was not built or run during this review.

@jbeckwith-oai jbeckwith-oai left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved the incremental changes after static review and an independent second review. The shared timestamp parser fixes Python 3.10 handling while retaining source nanoseconds; the adapter, parity fixture compatibility, and test portability changes have no new actionable findings. This remains stacked on #6105, whose two outstanding findings are not cleared by this approval. Hosted Windows 3.13 logs show the existing selector-event-loop subprocess and Windows file-URI issues, exposed after the collection fix.

Hosted check snapshot: {"SKIPPED": 2, "SUCCESS": 17, "FAILURE": 2}. Both Windows test jobs fail on existing selector-loop subprocess and Windows file-URI issues; the other 17 checks passed. Static review only; no candidate code or local tests were executed.

@jbeckwith-oai jbeckwith-oai left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after comprehensive static re-review and two independent reviews of the incremental changes. Reviewed the Rust borrowed-content projectors, capped typed serialization, oversized message/tool-result adapters, streaming request fingerprint, HTTP/WebSocket and MCP call sites against the pinned Codex types, UTF-8-preserving Python serialization, and parity timestamp assertions. No new actionable findings. The previously approved tracing capture and Windows fixes are incorporated.

Hosted checks completed: 19 passed, 2 skipped, including both Windows test versions and the Windows packaged-contract check. No candidate code or local tests were executed; the Rust patch was not built or run during this review.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛡️ Codex Security Review · Automatically triggered

Here are some automated security review suggestions for this pull request.

Reviewed commit: 85f402e1f7

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

Comment thread src/agents/models/codex.py Resolved

@jbeckwith-oai jbeckwith-oai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: independently confirmed the existing P2 cleanup finding. In the restored native-call path, the unconditional finally await propagates thread.close() errors through _join_close(), replacing an active CancelledError or the original execution failure. Preserve the primary BaseException and suppress secondary cleanup errors when one is already propagating, while still joining thread cleanup and the execution future. Add regression coverage for cancellation plus failing shutdown and execution failure plus failing shutdown. I missed this double-failure case in the previous review; this decision supersedes the prior approval.

The aggregate patch-hashing changes address the previous finding: 32 paths including move destinations, 16 MiB total, 8 MiB per file, and a cooperative 10-second deadline. Reusing the snapshot remains bound to the live approval and exact changes, with fresh preimage validation on restore and before execution. Independent native and Python reviews found no additional issues in these changes or the OpenTelemetry cache/reasoning token attributes.

Hosted check snapshot: {"SKIPPED": 4, "SUCCESS": 17, "IN_PROGRESS": 7}. Pending checks remain for follow-up. Static review only; no candidate code or local tests were executed.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛡️ Codex Security Review · Automatically triggered

Here are some automated security review suggestions for this pull request.

Reviewed commit: 6fdda2c30a

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6fdda2c30a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/agents/models/codex.py Resolved

@jbeckwith-oai jbeckwith-oai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes after static re-review and independent Python and native reviews. The prior cleanup finding is fixed: shutdown is joined while preserving the primary cancellation or execution failure. The checkpoint update also strips deletion preimages and fingerprints update diffs while retaining fresh file validation and legacy compatibility.

Two existing P2 findings remain independently confirmed:

  • Tracing conversion: code_mode_result constructs a full owned JSON value before the content cap. JsonToolOutput clones its Value; MCP conversion serializes the full result before removing _meta. Apply the cap to borrowed/typed output before that allocation.
  • Token accounting: after an earlier sampling pass has populated usage, a later approval-producing pass clears sampling_pending at raw_response_completed, ignores its token_usage, and exits before the cumulative token_count. Preserve cumulative usage across the pause and cover a multi-pass native approval.

Added one new inline finding for the supported Python 3.10 regression: cancellation-message assertions fail even though cleanup completes. The hosted Python 3.10 job reports eight failures in this matrix.

Static review only; no candidate code or local tests executed.

Hosted check snapshot: {"SKIPPED": 8, "SUCCESS": 18, "IN_PROGRESS": 4, "FAILURE": 2}. Unfinished checks remain pending.

Comment thread tests/models/test_codex_provider.py Resolved

@jbeckwith-oai jbeckwith-oai left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after static re-review and an independent second review. The three previous findings are addressed: tool tracing now uses bounded borrowed serialization before owned conversion, excludes MCP private metadata, and safely omits unknown output types; sampling usage is accumulated before approval pauses and replaced by subsequent cumulative counts; Python 3.10 cancellation tests retain type and cleanup checks without requiring unavailable messages. Reviewed wrapper implementations, error formatting, source bounds, usage fields, and session-resume regression coverage. No remaining actionable findings in the reviewed changes.

Hosted CI is complete: 33 passed and 9 skipped; no failures. This includes Python 3.10, native builds, and packaged-contract verification. Static review only; no candidate code or local tests executed.

@markstuart-oai markstuart-oai changed the title feat(codex): trace native operations and recover pending tool approvals feat(codex): trace model calls and tools with stock Codex Nov 21, 2026

@jbeckwith-oai jbeckwith-oai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes after static review of the rewritten stock-Codex integration and independent native and Python reviews. One checkpoint compatibility issue remains: a pending native approval inside an agent-as-tool snapshot is rejected recursively, but the default non-strict nested restore catches that error and discards the child state while accepting the parent. The new fail-closed check must cover the complete nested snapshot. See the inline finding.

The passive Rust observers, bounded metadata buffers, original-stream forwarding, per-thread isolation, shutdown/drain ownership, Python span mapping, sample/cumulative token accounting, and cancellation cleanup otherwise look sound. Native content capture and native approval recovery are explicitly unsupported in this revision; SDK tools remain the supported approval path.

Hosted CI snapshot: 17 checks passed and 7 remain in progress; no failures observed. Native wheel builds and remaining platform tests are pending. Static review only; no candidate code or local tests executed.

Comment thread src/agents/run_state.py Resolved

@jbeckwith-oai jbeckwith-oai left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after static re-review and an independent second review. The nested native-approval checkpoint finding is fixed: the public loader now recursively checks saved agent-as-tool states before restoring application context or sandbox state, so the best-effort nested restore cannot swallow this incompatibility. Regression coverage exercises both RunState and ResumableSession loaders with default options, direct and nested approvals, and with or without a context callback; it checks that the callback is not invoked and the workspace and paused checkpoint remain unchanged. The error also advises checking prior effects before starting a new run. No remaining actionable findings in this revision.

Hosted CI is complete: 33 passed and 4 skipped; no failures. This includes the native wheel builds and installed-wheel verification matrix. Static review only; no candidate code or local tests executed.

@markstuart-oai markstuart-oai changed the title feat(codex): trace model calls and tools with stock Codex feat(codex): resume pending tool approvals and trace native calls Nov 21, 2026

@jbeckwith-oai jbeckwith-oai left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by request changes: confirmed Windows installed-wheel assertion regression.

Approved after comprehensive static re-review and independent native-approval and tracing reviews. Reviewed the two pinned Core patches, fresh-thread approval recovery and immutable checkpoint receipts, permission and file-precondition revalidation, single-use approval boundaries, cleanup and cancellation, bounded opt-in content capture, Python/OpenTelemetry span correlation, and native wheel patch staging. The recursive nested-approval rejection remains intact. No actionable findings in this revision.

Hosted CI snapshot: 19 passed and 5 remain in progress, including native wheel builds and Windows Python 3.10. Those checks remain pending for follow-up. Static review only; no candidate code or local tests executed, and no native build was run during this review.

@jbeckwith-oai jbeckwith-oai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: supersedes my approval at this commit. Hosted Windows installed-wheel verification exposed a test portability regression that I missed in static review. Both Python 3.10 and 3.14 fail the new raw-path substring assertion at verify_installed.py:423 because tool inputs contain JSON-escaped Windows paths. Two independent source checks confirm the serialization mismatch; compare the decoded path instead. See the inline finding.

Hosted CI is complete: 30 passed, 2 failed, 3 skipped. Failures: Windows Python 3.10 and Windows Python 3.14. Both fail at this assertion; these logs do not establish a native runtime failure. Static review and hosted-log inspection only; no candidate code or local tests executed.

Comment thread native/codex/verify_installed.py Resolved

@jbeckwith-oai jbeckwith-oai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after static re-review and an independent second review. The Windows installed-wheel assertion is fixed: it decodes the captured tool arguments and compares the complete path value, so JSON escaping no longer breaks the check. This is the only change since the reviewed commit; no remaining actionable findings.

Hosted CI snapshot: 17 passed, 7 running, and 2 skipped. The new native builds and Windows installed-wheel verification still require follow-up; the prior failed jobs are not evidence that this revision passed. Static review only; no candidate code or local tests executed.

@jbeckwith-oai jbeckwith-oai left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after static re-review and an independent second review. Restored commands now confirm completion from the terminal process result instead of requiring an asynchronously delivered exit event before TurnComplete. Verified the pinned Core result semantics: the call ID must match, no live process handle may remain, and an exit code must be present; nonzero exits still represent completed commands. Saved approval validation remains required, and post-approval errors retain uncertain status rather than checkpointing completion. Reviewed the delayed-exit installed-wheel fixture and documentation. No actionable findings in this increment.

Hosted CI completed: 33 passed, 2 skipped. Static review only; no candidate code, local tests, or native builds executed.

@markstuart-oai
markstuart-oai merged commit beff84a into feat/simple-codex Nov 22, 2026
37 checks passed
@markstuart-oai
markstuart-oai deleted the feature/codex-operation-tracing branch November 22, 2026 02:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants