fix(plugin): order operation starts before attempts

[zhongkechen]

Summary

• dispatch on_operation_start synchronously before a START checkpoint is queued
• guarantee on_operation_start strictly precedes on_user_function_start for both synchronous and asynchronous checkpoints
• remove operation-start dispatch from the background checkpoint loop so each START is reported exactly once
• preserve replay detection by passing the pre-checkpoint operation snapshot to the plugin executor
• update the plugin interface documentation to describe the ordering contract

Address an issue found in aws/aws-durable-execution-conformance-tests#23

Behavior change

on_operation_start now reports operation initiation before checkpoint persistence. Plugins therefore observe START even if the checkpoint later fails; terminal/update hooks still depend on checkpoint responses.

Root cause

The default AT_LEAST_ONCE_PER_RETRY STEP semantics checkpoint START asynchronously. The background checkpoint callback could therefore arrive after on_user_function_start, causing instrumentation plugins to create an attempt before its logical operation.

Verification

• hatch run dev-core:test (1434 passed)
• hatch run dev-core:python -m pytest packages/aws-durable-execution-sdk-python/tests/state_test.py -q (132 passed)
• hatch run dev-otel:test (55 passed; OTel package unchanged)
• hatch run dev-core:typecheck
• hatch fmt --check from the core package
• hatch run python .github/scripts/lintcommit.py

[SilanHe]

how is the following guarenteed?

guarantee on_operation_start strictly precedes on_user_function_start for both synchronous and asynchronous checkpoints

[zhongkechen]

how is the following guarenteed?

guarantee on_operation_start strictly precedes on_user_function_start for both synchronous and asynchronous checkpoints

Calling sequence:

check_result_status()
  create_checkpoint(START, is_sync=...)
    on_operation_action()
      on_operation_start()       # sync=True; completes here
    checkpoint_queue.put()
    if is_sync: wait()
execute()
  wrapped_user_function()
    on_user_function_start()

[SilanHe]

I apologize if I'm misremembering. For Python, we are following a slightly different implementation where the on_operation_start and on_operation_end are called based on checkpoint calls to the backend. Say, a STEP was started in a previous invocation and failed an attempt in the previous invocation. Now, we have a second attempt of the STEP. How is the STEP operation span guaranteed to be open prior to the running of the on_user_function_start() in this case?

The attempt span should always resolve to a parent operation span but I feel like what I mentioned is another case that's (potentially) not covered by this PR.

[zhongkechen]

How is the STEP operation span guaranteed to be open prior to the running of the on_user_function_start() in this case?

START will be checkpointed again when a Step is attempted for the 2nd time, and then user function will be executed.

[SilanHe]

ah sorry about the mix up, just checked against the JS code and we checkpoint in both the AT_LEAST_ONCE_PER_RETRY and AT_MOST_ONCE_PER_RETRY semantics: https://github.com/aws/aws-durable-execution-sdk-js/blob/44a17838111bef922bc7c401a82fc29015800435/packages/aws-durable-execution-sdk-js/src/handlers/step-handler/step-handler.ts#L280