GlueJobOperator._find_job_run_id_by_task_uuid (the task-UUID scan fallback, used when neither task_state_store nor a cached XCom id has a prior run) has a few pre-existing quality issues, carried over verbatim from main, that are worth cleaning up:
- Unbounded pagination. It's a
while True over get_job_runs(MaxResults=50) with no page cap and no age cutoff. The no-match case — the prior attempt died before it ever called StartJobRun, which is the common retry shape — is exactly the one that walks the job's entire run history before giving up.
- Failure is silently swallowed. The two
except Exception blocks around this scan (and the XCom lookup before it) log at warning level and return None, after which the operator submits fresh — straight into ConcurrentRunsExceededException against a run that's still alive. There's no error-level signal telling anyone why.
- The
except Exception is broader than it needs to be. It should be narrowed to except ClientError, so a real bug in the surrounding code doesn't get silently absorbed the same way as a transient AWS-side issue.
- Missing IAM documentation. Anyone relying on this fallback (via
resume_glue_job_on_retry=True today, or explicit durable=True on Airflow <3.3 going forward) needs glue:GetJobRuns in their task policy, on top of StartJobRun/GetJobRun. The docs don't currently call that out.
None of this is new — it's present on main today, gated behind the opt-in resume_glue_job_on_retry flag. Raised during review of #71211, initially flagged as more urgent because that PR was going to make the scan reachable by default; the default-flip was scoped back to explicit opt-in for Airflow <3.3 (and to a narrow crash-recovery window on 3.3+) in the same PR, so the original urgency no longer applies — but the underlying issues are still real for anyone who does hit this path.
Fix direction: bound the walk (a page cap, or stop once a run's StartedOn predates the DAG run), narrow the exception handling, log the fallthrough at error level with the actual reason, and document the IAM requirement.
See #71211 (comment) for the original discussion.
GlueJobOperator._find_job_run_id_by_task_uuid(the task-UUID scan fallback, used when neithertask_state_storenor a cached XCom id has a prior run) has a few pre-existing quality issues, carried over verbatim frommain, that are worth cleaning up:while Trueoverget_job_runs(MaxResults=50)with no page cap and no age cutoff. The no-match case — the prior attempt died before it ever calledStartJobRun, which is the common retry shape — is exactly the one that walks the job's entire run history before giving up.except Exceptionblocks around this scan (and the XCom lookup before it) log atwarninglevel and returnNone, after which the operator submits fresh — straight intoConcurrentRunsExceededExceptionagainst a run that's still alive. There's no error-level signal telling anyone why.except Exceptionis broader than it needs to be. It should be narrowed toexcept ClientError, so a real bug in the surrounding code doesn't get silently absorbed the same way as a transient AWS-side issue.resume_glue_job_on_retry=Truetoday, or explicitdurable=Trueon Airflow <3.3 going forward) needsglue:GetJobRunsin their task policy, on top ofStartJobRun/GetJobRun. The docs don't currently call that out.None of this is new — it's present on
maintoday, gated behind the opt-inresume_glue_job_on_retryflag. Raised during review of #71211, initially flagged as more urgent because that PR was going to make the scan reachable by default; the default-flip was scoped back to explicit opt-in for Airflow <3.3 (and to a narrow crash-recovery window on 3.3+) in the same PR, so the original urgency no longer applies — but the underlying issues are still real for anyone who does hit this path.Fix direction: bound the walk (a page cap, or stop once a run's
StartedOnpredates the DAG run), narrow the exception handling, log the fallthrough aterrorlevel with the actual reason, and document the IAM requirement.See #71211 (comment) for the original discussion.