ResumableJobMixin's crash-recovery decision logic (reconnect / already-succeeded / terminal-resubmit) lives entirely inside execute_resumable, inlined together with the synchronous polling that follows it. There's no way for a caller to get just the decision without also getting blocked on poll_until_complete.
This matters for any operator that supports both deferrable=True and durable execution, GlueJobOperator included. On the deferrable path, the operator calls submit_job directly from execute(), bypassing the mixin's task_state_store read entirely — nothing is ever persisted there for a deferred run. So a deferrable retry has no cheap way to check "is there already a run I can reconnect to" and instead always falls through to the operator's own bootstrap fallback (for Glue, a full paginated get_job_runs scan), even though the run id could have been persisted to task_state_store before defer() and read back cheaply on retry.
Fixing this requires the mixin to expose its reconnect-decision step (read stored id, call get_job_status, apply is_job_active/is_job_succeeded) as something callable on its own, separate from the polling loop in execute_resumable. Once that exists, a deferrable operator could:
- Call the decision step before
defer().
- If nothing to reconnect to, submit fresh and persist the id to
task_state_store.
- On a deferrable retry, call the decision step again before deciding whether to resubmit.
This would let deferrable retries reconnect via the cheap store lookup instead of an operator-specific bootstrap scan, for every current and future ResumableJobMixin port that supports deferrable=True.
Raised during review of #71211 (kaxil, #71211 (comment) — see comment thread for full context).
ResumableJobMixin's crash-recovery decision logic (reconnect / already-succeeded / terminal-resubmit) lives entirely insideexecute_resumable, inlined together with the synchronous polling that follows it. There's no way for a caller to get just the decision without also getting blocked onpoll_until_complete.This matters for any operator that supports both
deferrable=Trueand durable execution,GlueJobOperatorincluded. On the deferrable path, the operator callssubmit_jobdirectly fromexecute(), bypassing the mixin'stask_state_storeread entirely — nothing is ever persisted there for a deferred run. So a deferrable retry has no cheap way to check "is there already a run I can reconnect to" and instead always falls through to the operator's own bootstrap fallback (for Glue, a full paginatedget_job_runsscan), even though the run id could have been persisted totask_state_storebeforedefer()and read back cheaply on retry.Fixing this requires the mixin to expose its reconnect-decision step (read stored id, call
get_job_status, applyis_job_active/is_job_succeeded) as something callable on its own, separate from the polling loop inexecute_resumable. Once that exists, a deferrable operator could:defer().task_state_store.This would let deferrable retries reconnect via the cheap store lookup instead of an operator-specific bootstrap scan, for every current and future
ResumableJobMixinport that supportsdeferrable=True.Raised during review of #71211 (kaxil, #71211 (comment) — see comment thread for full context).