Repository navigation
ThrottlingRequestManager does not stop a 429 from retiring the session #2143
Description
_raise_for_session_blocked_status_code records the domain delay and then falls through to
Session.is_blocked_status_code, and 429 is in _DEFAULT_BLOCKED_STATUS_CODES. So on a domain the
throttler covers, a 429 still raises SessionError, still retires the session and still spends a
max_session_rotations slot — the churn #1437 was filed about. Only ignore_http_error_status_codes
avoids it, and nothing says so.
This wants fixing together with a stall timeout, in one PR:
- Nothing bounds a permanently rate-limiting domain except the retry accounting the fix removes.
_should_retry_requestcapsSessionErroratmax_session_rotationsand refuses
HttpClientStatusCodeErroroutright, so today such a domain drains and the crawl ends. Make a
throttled 429 free and it never terminates —run()hangs while the statistics report nothing,
which is what the TS port walked into. - So the fix also needs a per-domain stall timeout that gives up once a domain with queued requests
has been rate-limiting for that long, and akeep_aliveexemption.max_domain_stall_secsis a
constructor argument, so landing it separately ships a release where the knob does nothing.
Both halves are in apify/crawlee#3741. From #1762.
Activity
chenhz01 commentedon Sep 22, 2026
One decision is split across two layers:
_raise_for_session_blocked_status_coderecords the domain delay, then falls through to Session.is_blocked_status_code, where 429 sits in _DEFAULT_BLOCKED_STATUS_CODES — so the throttler's mitigation is instantly undone by a retirement that burns a max_session_rotations slot, the exact churn #1437 complains about. The clean fix is a three-state verdict per response: retry-in-place (throttler owns it: record delay, retry same session), rotate-session, or fail. Only statuses the throttler cannot absorb should ever reach the session layer — and because a domain that rate-limits forever would then retry in place indefinitely, this needs the per-domain stall timeout the report calls for (max_domain_stall_secs plus the keep_alive exemption), in the same PR:if throttler.handles(status): await throttler.backoff(); return RETRY_SAME_SESSION
This decision split is a core case in my published resilient-agent-toolkit. Want me to take a crack at it? Happy to open a PR with working code.
Mantisus commentedon Sep 22, 2026
Hi @chenhz01, thanks for taking a look at this
I'm sorry, but this task involves many important components. It will be implemented by one of the core maintainers.
fix(throttling-manager): Defer throttled 429s without spending retrie…
vdusek commentedon Sep 29, 2026
resolved by #2250