Skip to content

Expose reconnect-decision separately from polling, for deferrable operators #71485

Description

@amoghrajesh

ResumableJobMixin's crash-recovery decision logic (reconnect / already-succeeded / terminal-resubmit) lives entirely inside execute_resumable, inlined together with the synchronous polling that follows it. There's no way for a caller to get just the decision without also getting blocked on poll_until_complete.

This matters for any operator that supports both deferrable=True and durable execution, GlueJobOperator included. On the deferrable path, the operator calls submit_job directly from execute(), bypassing the mixin's task_state_store read entirely — nothing is ever persisted there for a deferred run. So a deferrable retry has no cheap way to check "is there already a run I can reconnect to" and instead always falls through to the operator's own bootstrap fallback (for Glue, a full paginated get_job_runs scan), even though the run id could have been persisted to task_state_store before defer() and read back cheaply on retry.

Fixing this requires the mixin to expose its reconnect-decision step (read stored id, call get_job_status, apply is_job_active/is_job_succeeded) as something callable on its own, separate from the polling loop in execute_resumable. Once that exists, a deferrable operator could:

  1. Call the decision step before defer().
  2. If nothing to reconnect to, submit fresh and persist the id to task_state_store.
  3. On a deferrable retry, call the decision step again before deciding whether to resubmit.

This would let deferrable retries reconnect via the cheap store lookup instead of an operator-specific bootstrap scan, for every current and future ResumableJobMixin port that supports deferrable=True.

Raised during review of #71211 (kaxil, #71211 (comment) — see comment thread for full context).

Metadata

Metadata

Assignees

Type

No type

Projects

Status
In progress

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions