Improve AgentServer warm response latency and reduce repeated work

[Nathandrake229]

Description

Reduce repeated work in the AgentServer Responses and Invocations SDKs while
preserving request isolation, persistence ordering, validation, replay and
awaited telemetry completion.

Changes

• Reuse request-owned history-ID results across equivalent reads.
• Move streaming telemetry flush off the pre-stream path while still awaiting
  it before final HTTP completion.
• Generate real models lazily and qualify internal type-only references so
  importing the host does not eagerly construct the entire model catalog.
• Coalesce concurrent input-reference materialization within one request.
• Reuse an owned event coercion and remove a completed-item copy-of-copy.
• Copy only required builder seed fields and avoid repeated recovery-graph copies.
• Generate Invocations fallback UUIDs only when needed.
• Await request-owned handler/iterator cleanup on disconnect before flushing,
  without taking ownership of independent stored/background producers.

Current upstream versions, dependencies, session-GUID handling and new model
contracts are retained. No Core changes, native-curl image changes, server
changes or fire-and-forget telemetry behavior are included.

Performance highlights

The benchmarked cumulative SDK candidate improved warm Responses latency
against the released SDK baseline:

P95 metric	Released baseline	Optimized candidate	Improvement
Time to first byte	754.7 ms	357.4 ms	397.4 ms / 52.7% faster
Time to last byte	869.2 ms	691.7 ms	177.4 ms / 20.4% faster

Both variants used the same hosted Echo application and runtime/dependency
configuration apart from the SDK packages, a colocated UK South client,
concurrency 25 and at least 2,000 successful samples per protocol/phase.
Neither image included native curl.

Cycle-block bootstrap 95% intervals support the observed warm reductions:
TTFB 351.8-469.9 ms, TTLB 125.1-253.5 ms.

These measurements describe the tested candidate on released SDK versions.
This PR ports the same optimization intent onto current main; the exact
ported head has not been re-benchmarked. Results are aggregate, not isolated
attribution to individual changes. Full benchmark context is provided in a
separate review note.

Validation

• Offline Linux tests against the ported current-main source: 526 Responses
  tests and 126 Invocations tests passed.
• 379 real model contracts and 109 public method contracts compared against
  current main with no differences.
• Generator/extraction reproducibility, deterministic output, Python 3.10 syntax
  and scoped formatting checks passed.
• The real-pipeline disconnect reproduction now completes handler cleanup before
  orchestration finalization, flush and request return; stored producer behavior
  is covered separately.
• Full SDK CI, full TypeSpec/npm regeneration and the supported-platform matrix
  remain for review/CI; this PR is intentionally opened as a draft.

Review guide

The 53-file diff includes four generated model files. Most line volume is
mechanical generator output; review the generator and runtime changes separately
from their generated output. Tests and changelogs are included. This keeps the
requested cumulative change set together for review; no release is implied.

Contribution checklist

• No breaking changes: maintainer/API review pending; focused parity checks passed.
• Package changelogs updated in their existing unreleased sections.
• Contribution guidelines reviewed.
• Tests cover the included changes.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Nathandrake229]

Benchmark validation context

The description highlights the warm gains. For review, the complete observed
P95 matrix is below so the optimization is not interpreted as an all-latency win.
This is the benchmarked cumulative candidate versus released Responses 2.1.0b2
and Invocations 1.1.0b1, not an isolated measurement of any individual change.

Protocol	Phase	Baseline TTFB	Candidate TTFB	Baseline TTLB	Candidate TTLB
Invocations	Warm	168.9 ms	179.6 ms	170.3 ms	181.3 ms
Responses	Warm	754.7 ms	357.4 ms	869.2 ms	691.7 ms
Invocations	Cold	2313.1 ms	2215.1 ms	2313.6 ms	2217.0 ms
Responses	Cold	2824.1 ms	2737.0 ms	2926.0 ms	3183.4 ms

Material tradeoff: cold Responses P95 completion latency increased 257.3 ms
(8.8%); its block-bootstrap 95% interval supports an observed regression
of 103.3-354.5 ms. The result should not be presented as uniformly improving
latency. Cold Responses TTFB and warm Invocations changes have uncertainty
intervals that include no change.

The candidate's numerical Responses-minus-Invocations TTFB gaps were 177.8 ms
warm and 522.0 ms cold; the 100 ms goal was not reached.

Method and limitations

• Same Echo code, CPU/memory settings, Python/Core dependencies and colocated
  region; neither image included native curl, and no server patch was deployed.
• Every protocol/phase has at least 2,000 successful samples; both Responses cold
  populations have 2,001 valid samples, all retained.
• Baseline had 8,002 attempts/8,001 successes; candidate 8,003 attempts/8,001 successes.
  One versus two Responses service errors remain in failure accounting.
• Actual serving-version proof was mandatory. Metadata-only retries were
  enabled identically but none were needed; no invoke was replayed.
• Full backend trace joins completed and the observed backend build was
  identical across variants.
• Uncertainty used 2,000 replicates of 100-cycle blocks, preserving cold/warm
  pairing within each protocol. This does not eliminate sequential-run,
  placement, cache-state or other operational confounding; no multiplicity
  adjustment or individual-change causal attribution is claimed.
• A separate warm-Invocations check found SDK request time essentially
  unchanged (mean 1.722 -> 1.713 ms); the client-tail movement is not established
  as an SDK processing regression. That does not explain the cold Responses
  completion tradeoff above.

No internal resource identifiers, request traces, access tokens or private
monitoring URLs are included in this public review note.

Port and review scope

The measured prototype used released package versions. The PR is ported onto
current main (Responses 2.2.0b2, Invocations 1.2.0b2, unchanged Core 2.2.0b2),
preserving newer session-GUID behavior and 379 current model contracts. The
ported head passed focused offline checks but was not independently re-run in
the hosted benchmark.

Review-size warning: this cumulative draft contains 53 files and 38,155
changed lines. Four generated files account for 34,234 lines; the remaining
3,921 changed lines include generators, mechanical type-reference updates,
runtime logic, tests and docs. Logical split boundaries, if preferred during
review, are lazy model generation, request/stream work reuse, and Invocations
UUID fallback. The draft keeps the user-requested cumulative set together.

The port also fixes a request-owned disconnect cleanup ordering issue found
during review. That correction is covered by a real ASGI/handler-finalizer
regression and preserves independent stored/background producers; it was not
included in the hosted timing experiment. The final source selection passes
526 Responses and 126 Invocations tests. A union-pickle test assumption was
corrected to follow Python's equality contract while retaining strict canonical
identity checks for real model classes and exported aliases.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🔵 Needs a closer look

It spans concurrency, cancellation, persistence, telemetry, and generated public models while full CI and platform validation remain pending.

Pull request overview

Optimizes AgentServer warm-response latency while preserving request isolation, persistence, cleanup, and model contracts.

Changes:

• Reuses request-local history/input materialization and removes redundant copying/UUID generation.
• Defers streaming telemetry flush while awaiting cleanup before HTTP completion.
• Lazily constructs generated response models with deterministic generation and contract tests.

File summaries

File	Description
sdk/agentserver/azure-ai-agentserver-responses/CHANGELOG.md	Documents Responses optimizations.
sdk/agentserver/azure-ai-agentserver-responses/docs/runtime-work-reuse.md	Explains reuse and cleanup contracts.
sdk/agentserver/azure-ai-agentserver-responses/_scripts/extract_model_contracts.py	Integrates lazy-model emission.
sdk/agentserver/azure-ai-agentserver-responses/_scripts/lazy_model_emitter.py	Generates lazy model factories.
sdk/agentserver/azure-ai-agentserver-responses/_scripts/qualify_model_references.py	Qualifies internal model references.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/__init__.py	Lazily exports root models.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/_response_context.py	Adds request-local caches and single-flight resolution.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/hosting/_acceptance.py	Uses lazy generated annotations.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/hosting/_endpoint_handler.py	Defers flush and coordinates stream cleanup.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/hosting/_event_subject.py	Qualifies lazy event types.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/hosting/_execution_context.py	Qualifies context model types.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/hosting/_orchestrator.py	Reuses work and manages producer ownership.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/hosting/_request_parsing.py	Qualifies request model references.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/hosting/_resilient_input.py	Uses lazy request annotations.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/hosting/_resilient_orchestrator.py	Resets recovered request caches.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/hosting/_routing.py	Closes owned handler iterators.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/hosting/_runtime_state.py	Qualifies output-item types.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/hosting/_validation.py	Qualifies validation model types.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/models/__init__.py	Lazily exposes public models.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/models/_generated/__init__.py	Dispatches generated lazy exports.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/models/_generated/_catalog.py	Catalogs generated exports.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/models/_generated/_unions.py	Lazily constructs generated unions.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/models/_generated/types.py	Lazily constructs generated models.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/models/_helpers.py	Avoids eager model imports.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/models/_lazy_models.py	Provides thread-safe lazy loading.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/models/runtime.py	Qualifies runtime model references.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/store/_base.py	Qualifies provider protocol types.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/store/_file.py	Qualifies file-store types.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/store/_foundry_provider.py	Qualifies Foundry provider types.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/store/_foundry_serializer.py	Qualifies serialization types.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/store/_memory.py	Qualifies in-memory store types.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/streaming/_builders/_base.py	Defers cast type evaluation.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/streaming/_builders/_function.py	Defers function-event cast types.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/streaming/_builders/_message.py	Defers message-event cast types.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/streaming/_builders/_reasoning.py	Defers reasoning-event cast types.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/streaming/_builders/_tools.py	Defers tool-event cast types.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/streaming/_checkpoint.py	Qualifies checkpoint response type.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/streaming/_event_stream.py	Reduces seed copying.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/streaming/_helpers.py	Reuses owned coerced events.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/streaming/_internals.py	Removes redundant item copying.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/streaming/_sse.py	Shields iterator cleanup.
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/streaming/_text_response.py	Qualifies text-response types.
sdk/agentserver/azure-ai-agentserver-responses/tests/contract/test_eager_history_prefetch.py	Tests prefetch reuse and failures.
sdk/agentserver/azure-ai-agentserver-responses/tests/contract/test_history_storage_pull_counts.py	Verifies storage call counts.
sdk/agentserver/azure-ai-agentserver-responses/tests/unit/test_event_copy_reuse.py	Tests copy boundaries.
sdk/agentserver/azure-ai-agentserver-responses/tests/unit/test_input_singleflight.py	Tests concurrent input resolution.
sdk/agentserver/azure-ai-agentserver-responses/tests/unit/test_lazy_generated_models.py	Tests lazy model contracts.
sdk/agentserver/azure-ai-agentserver-responses/tests/unit/test_request_history_resolution.py	Tests history cache isolation.
sdk/agentserver/azure-ai-agentserver-responses/tests/unit/test_streaming_flush.py	Tests flush and cleanup ordering.
sdk/agentserver/azure-ai-agentserver-responses/tests/unit/test_streaming_history_reuse.py	Tests persistence history reuse.
sdk/agentserver/azure-ai-agentserver-invocations/CHANGELOG.md	Documents lazy ID generation.
sdk/agentserver/azure-ai-agentserver-invocations/azure/ai/agentserver/invocations/_invocation.py	Generates fallback UUIDs lazily.
sdk/agentserver/azure-ai-agentserver-invocations/tests/test_lazy_id_fallback.py	Tests fallback ID behavior.

Review details

• Files reviewed: 52/53 changed files
• Comments generated: 1
• Review effort level: Balanced

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🔵 Needs a closer look

It changes concurrency, cancellation, persistence, and generated API machinery, while full CI and platform validation remain pending.

Review details

• Files reviewed: 52/53 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Pranav Pandit (vangarp)]

should we import only ResponseObject instead of all?

[Pranav Pandit (vangarp)]

same here. only import necessary ones instead of all?