You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Share one concurrency budget between multiple crawlers #2133
Each crawler builds its own AutoscaledPool and SystemStatus, so several crawlers in one process oversubscribe the host - there is no way to cap their combined concurrency. JS v4 split the scaling logic out into a ConcurrencySystem that can be constructed once and passed to several crawlers.
Scope
Port the whole ConcurrencySystem contract, not only the shared budget:
A minimal protocol the pool and crawlers depend on: desired_concurrency, current_concurrency, is_running, a capacity check, and atomic task booking (try_register_task_start / register_task_end) that takes an optional task ID, so a shared budget cannot be oversubscribed.
Ownership: a crawler builds and drives its own default system, fresh per run(). An injected system is owned by the caller, who starts and stops it, and run() raises if it was not started. Passing both concurrency_settings and a system is an error.
Runtime tuning (desired_concurrency and friends) moves from the pool to the system. The crawler exposes the system read-only, plus pause() / resume().
Activity