[yaythomas]

Problem

Plugin hooks for a checkpoint response (on_operation_end, on_operation_change) ran on the checkpoint thread. That is the one thread that sends every checkpoint of an invocation and applies every response, for every branch. So a slow hook, such as a network export, delayed every checkpoint in the invocation.

Simply detaching the hooks is not an option. The OpenTelemetry plugins derive span order from hook call order, and the OpenTelemetry conformance suite asserts that order: a wait's span must end before the span of the step that follows it starts. So an operation's end hook must run before that operation returns to user code.

What changed

The checkpoint thread runs no plugin code. For each response it:

1. records the end hook of every terminal operation the response reports, before it publishes the response's operations;
2. queues the response's change notification for a dispatcher thread;
3. wakes the response's synchronous waiters.

End hooks run on the thread that observes the operation's terminal result, before that operation returns to user code:

• the operation's own thread, after its synchronous terminal checkpoint is confirmed;
• the thread that reads a result another response completed, such as a branch resumed after a refresh or a callback result read;
• the thread that replays an operation completed before this invocation (unchanged).

A thread that observes a result while another thread is running its end hook waits for the hook. A read made from inside any plugin hook, on the hook's own thread, is the exception: it neither runs nor waits for an end hook. So a result read cannot make the hook that makes it wait for another end hook, and a lock a plugin holds across a hook (the contract tells plugins to guard shared state with one) cannot be needed by an end hook that read starts. Before this PR, result reads dispatched nothing anywhere; this keeps that true inside hooks. A terminal operation no thread reads during the invocation gets its end hook on the handler thread before on_invocation_end; a completed descendant nobody read (a callback created and never awaited) runs with, and before, its ancestor's end hook. An operation's end hook runs at most once per invocation, even when a response lists an operation the invocation input already carried.

Change notifications run on a dispatcher thread, in response order. When a plugin is still in the hook as responses arrive, they merge into one pending notification: updated_operations carries the newest state of every operation changed since the previous delivery, operations the newest full map. Intermediate states and the number of calls are not guaranteed. The pending notification is delivered before on_invocation_end. The single pending slot bounds the backlog whatever a plugin's speed. If the process cannot start the thread, the merged notification is delivered once on the handler thread before on_invocation_end.

Shutdown order is unchanged: branch pools are joined, the checkpoint thread is stopped and joined, unclaimed end hooks run, the dispatcher is joined, then on_invocation_end runs.

Failures: ordinary hook exceptions are logged and ignored, as before. Thread-control exceptions (KeyboardInterrupt, SystemExit, GeneratorExit) propagate on operation threads and the handler thread; on the dispatcher thread they are logged, the other plugins still receive the notification, and delivery continues.

The DurableInstrumentationPlugin class docstring states this contract: where each hook runs, what order a plugin may rely on, and the merge rule.

Behavior changes for plugin authors

• on_operation_change may be coalesced; a plugin must not assume one call per checkpoint response.
• on_operation_end for an operation completed outside the invocation runs on the thread that reads the result, not on the checkpoint thread.
• Within one invocation an operation's end hook runs at most once. On main, a response that listed an already-reported operation produced a second end hook.

Tests

• tests/plugin_hook_registry_test.py (48 tests): record-before-publish and record-before-signal observed at the loop's publish and signal points; claim on the synchronous wait (success and error paths, including a failed later pagination page); claim on read; a second observer waits for a running hook; descendants before ancestors; once per invocation across the registry and the replay path; every claim decision under one lock, in one acquisition; the replay-boundary reads take no hook; two end hooks reading each other's operations concurrently both return; a change hook that holds a lock and reads a callback result does not deadlock with the callback's end hook; a propagation-metadata provider that reads a result runs no hook; a read from inside a hook runs no hook and the consuming read later does; the child-context end hook reads without waiting; a control exception in one drained hook does not stop the others; the dispatcher's order, merge rule, join before the end hook, fallback without a thread, and control-exception handling; drain and shutdown order.
• tests/e2e/plugin_hook_contract_int_test.py (8 tests) through the local test runner: end hook before the operation returns; slow hook delays only its own branch; child before parent; map items before the map; early completion exception (the quick branch returns only once the slow branch is running, since a branch the pool has not started is cancelled on early completion and has no end hook; the previous revision assumed the start and failed on the slower CI runners); a wait in a branch resumed in-process by a refresh; a callback result read.
• tests/e2e/handler_context_propagation_int_test.py: contextvar visibility on the dispatcher thread and the operation thread.
• Conformance, against the service in the test account on this commit: all 11 core suites; OpenTelemetry invocation view 20/20, execution view 20/20, long-running invocation view 4/4. The OpenTelemetry views are the jobs that failed on the previous revision of this PR.

Notes

• The OpenTelemetry plugins are unchanged. The comments in them and in the telemetry exporter that described hooks arriving after the invocation end were updated.
• The local test runner lists, in the first checkpoint response of an invocation, operations the invocation input already carried, and lists in the next invocation's UpdatedOperationIds operations already delivered in a response. The service does neither. The once-per-invocation rule makes the SDK's behavior the same under both. A separate change to the runner may follow.

[zhongkechen]

The updated assertion in test_invocation_wait_resume_int.py allows the resumed wait span to end after the following step starts. That breaks the existing otel-long-running-1 requirement, and the Invocation - Short CI check is already failing on exactly that sequence. I also reproduced it locally by delaying the queued wait-end hook until the next step had started. Could we include the OTel timestamp fix in this PR and keep the original ordering assertion? Moving the hooks off the checkpoint thread should still let the trace reflect when the operations actually happened.

[yaythomas]

@zhongkechen Thanks, and agreed. The current revision keeps the original assertion: test_invocation_wait_resume_int.py is unchanged from main. The wait's end hook now runs on the thread that reads the resumed result, before that thread starts the next operation, so the wait span ends before the following step's span starts. all the tests now pass.

[github-actions]

Codex AI review

No confirmed actionable findings. Review was limited to read-only inspection; tests were not executed, leaving residual runtime concurrency risk.

Reviewed commit cfaae2bdd77bc0bae9fafea2ec1483e652402a61. Workflow run