Skip to content

Share one concurrency budget between multiple crawlers #2133

Open

Description

@vdusek

Each crawler builds its own AutoscaledPool and SystemStatus, so several crawlers in one process oversubscribe the host - there is no way to cap their combined concurrency. JS v4 split the scaling logic out into a ConcurrencySystem that can be constructed once and passed to several crawlers.

Scope

Port the whole ConcurrencySystem contract, not only the shared budget:

  • A minimal protocol the pool and crawlers depend on: desired_concurrency, current_concurrency, is_running, a capacity check, and atomic task booking (try_register_task_start / register_task_end) that takes an optional task ID, so a shared budget cannot be oversubscribed.
  • Ownership: a crawler builds and drives its own default system, fresh per run(). An injected system is owned by the caller, who starts and stops it, and run() raises if it was not started. Passing both concurrency_settings and a system is an error.
  • Runtime tuning (desired_concurrency and friends) moves from the pool to the system. The crawler exposes the system read-only, plus pause() / resume().
  • Snapshotter, SystemStatus and the load-signal internals become private to the system (load-signal configuration is Make autoscaling load signals pluggable #2134).

Depends on the ownership rules in #2301.

JS implementation: apify/crawlee#3917, apify/crawlee#4130 (task IDs, apify/crawlee#4127), apify/crawlee#4067 (initialConcurrency)

This will be a breaking change.

✍️ Drafted by Claude Code

Activity

  1. added
    enhancementNew feature or request.
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Aug 7, 2026
  2. added this to the 2.0 milestone on Aug 7, 2026
  3. self-assigned this
    on Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions