Skip to content

feat: capture Runner submissions and provider request attempts - #6140

Open
markstuart-oai wants to merge 2 commits into
feat/simple-codexfrom
feat/runner-request-capture
Open

markstuart-oai wants to merge 2 commits into
feat/simple-codexfrom
feat/runner-request-capture

Conversation

@markstuart-oai

@markstuart-oai markstuart-oai commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

This pull request adds RequestCapture to the Python Runner. One model submission can trigger several requests through retries, redirects or authentication. Callers can now inspect local transport counts even when a run fails or is cancelled.

from agents import RequestCapture, Runner

async def run(agent, prompt):
    capture = RequestCapture()
    try:
        with capture:
            return await Runner.run(agent, prompt)
    finally:
        totals = capture.totals
        print(totals.model_submissions, totals.coverage, totals.observed)

One capture covers the first Runner invocation in the block. Drain streams inside the block. Reusing the capture starts fresh, including after approval. Handoffs and corrections stay with the same owner. Attached agent tools share child totals. Automatic session compactions count separately.

  • A stream counts as one submission when the first attempt passes the input guardrail check. If the guardrail blocks that attempt, the capture records no submission. A blocked retry leaves the original count intact.
  • SDK-owned HTTP and SSE count header write attempts, completed header writes and received response headers. WebSocket counts request frame send attempts, successful sends and first replies. Handshakes are excluded.
  • Each route reports coverage. Partial totals retain known counts. HTTP/2 is partial. Authentication can leave gaps when a new request loses its association with the original. Borrowed HTTP clients, custom models and opaque native or hosted activity have unknown transport counts. When coverage is unknown, observed is None.
  • Counts cover local observations only. They cannot confirm delivery, receipt of the full response body or billing. The observer cannot enforce a hard limit on physical requests.

Capture is off by default. Snapshots contain fixed labels and counts. The SDK creates a private client subclass to associate retries with their request while preserving the default User-Agent. Caller-supplied clients remain unchanged.

Verification

Formatting, lint, Mypy and Pyright passed. The full local suite passed 13,254 tests and skipped 37. All 35 capture tests also passed on Python 3.10 and with the minimum supported client version. The guardrail regression covers timed and untimed streams with capture enabled and disabled.

@markstuart-oai
markstuart-oai requested review from a team, rm-openai and seratch as code owners November 21, 2026 14:38
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-21T15:05:07.409864Z b218a98 New commits
🔒 Security Review ✅ Completed 2026-11-21T15:05:35.072782Z b218a98 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 79564c8739

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/agents/models/_run_context.py Resolved

@jbeckwith-oai jbeckwith-oai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: independently confirmed the existing P2 guardrail accounting finding. In streamed runs, model_run_context_stream enters _model_submission before get_model_stream checks the input-guardrail tripwire. If the guardrail trips during awaited preparation, _StreamAttemptSkipped prevents model.stream_response from being called, yet the capture records one submission. Move registration after that gate while retaining one logical submission across retries, and add a regression for a guardrail tripping during input-filter or on_llm_start preparation. Referencing the existing comment rather than adding a duplicate inline finding.

Completed comprehensive static review with two independent reviews covering capture ownership, attached-child isolation, retries, automatic compaction, HTTP authentication/redirects and trace gaps, WebSocket send/reply boundaries, frozen snapshots, and disabled-capture behavior. No additional actionable findings.

Hosted CI snapshot: {"SKIPPED": 2, "SUCCESS": 17, "IN_PROGRESS": 7}. Remaining checks stay pending. Static review only; no candidate code or local tests executed.

@jbeckwith-oai jbeckwith-oai left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after static re-review and an independent second review. The guardrail accounting finding is fixed: streamed submissions are registered only after the input-guardrail skip gate, in the task that actually opens the model stream. The per-stream flag preserves one logical submission across retries, and the capture scope restores the enclosing context. Reviewed timed and untimed retry paths, nonstreaming scope preservation, and the enabled/disabled capture regression matrix. No remaining actionable findings in the reviewed changes.

Hosted CI is complete: 33 checks passed and 4 skipped; no failures. Static review only; no candidate code or local tests executed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants