Pooling

[KoalaGeo]

Overview

Makes the SQLAlchemy connection pool of the SQL provider configurable per provider via the existing options: block, exposing pool_size, max_overflow, pool_recycle, pool_timeout and pool_pre_ping.

Previously get_engine() called create_engine(conn_str, connect_args=connect_args, pool_pre_ping=True) with no pool sizing or recycle, so the default QueuePool held pool_size connections open for the life of each worker process and never recycled them. In multi-process deployments this produces a large number of permanently-IDLE server-side connections (we saw connections idle for days, eventually exhausting max_connections). There was no way to bound or recycle the pool from configuration.

Changes:

• store_db_parameters() now extracts the five pool keys from options, coerces them to their declared types, and stores them as a sorted, hashable tuple (self.db_pool_options). They are popped out of options so they are not forwarded to the DBAPI as connect_args.
• get_engine() takes a pool_options tuple parameter and applies **dict(pool_options) to create_engine(). It stays @functools.cache-able because the parameter is a hashable tuple, so engine sharing per process is preserved; providers with differing pool config correctly get distinct engines.
• pygeoapi/process/manager/postgresql.py also calls get_engine(); its call site is updated to pass self.db_pool_options so the manager does not lose pool_pre_ping or skip recycling.

Backward compatibility: defaults preserve current behaviour exactly — pool_size=5, max_overflow=10, pool_pre_ping=True, and pool_recycle=-1 (SQLAlchemy's default, i.e. the current effective behaviour).

This PR is therefore a pure, opt-in feature add with no behaviour change for existing users. (See the issue for discussion of whether a finite default pool_recycle should be adopted as a separate follow-up.)

New tests and documentation are included.

Related Issue / discussion

Closes #2344.

Additional information

Example configuration:

providers:
  - type: feature
    name: PostgreSQL
    data:
      host: 127.0.0.1
      port: 5432
      dbname: test
      user: postgres
      password: postgres
      search_path: [osm, public]
    options:
      pool_size: 2          # persistent connections per worker process
      max_overflow: 3       # short-lived burst capacity
      pool_recycle: 300     # recycle connections older than 5 minutes
      pool_timeout: 30
    id_field: osm_id
    table: hotosm_bdi_waterways
    geom_field: foo_geom

Note (documented): because get_engine() is @functools.cache-d on its full argument set, providers that share a database must use identical pool options to continue sharing a single engine per worker; differing options intentionally yield separate engines.

Dependency policy (RFC2)

• I have ensured that this PR meets RFC2 requirements

No new dependencies are introduced; only the standard library and the already-required SQLAlchemy are used.

Updates to public demo

• I have ensured that breaking changes to the pygeoapi master demo server have been addressed
• No changes required: defaults preserve existing behaviour, so the demo local.config.yml does not need to change.

Contributions and licensing

• I'd like to contribute a bugfix/feature (configurable SQL connection pool) to pygeoapi. I confirm that my contributions to pygeoapi will be compatible with the pygeoapi license guidelines at the time of contribution
• I have already previously agreed to the pygeoapi Contributions and Licensing Guidelines

[webb-ben]

Is there a reason to not pop the attributes from the connect_args inside of get_engine? This would consolidate a bit of the complications noted in the PR between hashing and the manager using get_engine. Maybe I am missing something

[ricardogsilva]

Just leaving my two cents here - I'm not a core committer so take these with a grain of salt.

Overall I agree with the PR, as adding these connection-related options seems relevant - thanks for your work and I look forward to having it merged!

Personally, I would simplify the implementation a bit, by relying on pygeoapi's JSON Schema document for the validation of the config.

And I would not include most of these tests, which I see as not being relevant.

[KoalaGeo]

Is there a reason to not pop the attributes from the connect_args inside of get_engine? This would consolidate a bit of the complications noted in the PR between hashing and the manager using get_engine. Maybe I am missing something

That's a good shout, I'll refactor

[tomkralidis]

@KoalaGeo any update on this PR?

[KoalaGeo]

@tomkralidis think that's addressed everything raised by @webb-ben and @ricardogsilva — let me know if I've missed anything.

Quick summary of what changed:

• get_engine() now pops the pool keys out of connect_args itself (@webb-ben's suggestion). That removes the separate db_pool_options tuple entirely, and the manager call site no longer needs any special handling — it just passes **self.db_options like the provider does. It stays functools.cache-able since the kwargs are hashable scalars.
• Added pool_size / max_overflow / pool_recycle / pool_timeout / pool_pre_ping to the config JSON Schema (@ricardogsilva), so a config can be validated before the server starts. That's now what enforces the types, so I've dropped the manual type(default)(...) coercion — which also removes the bool("False") is True gotcha you spotted.
• Removed the outdated pool_recycle comment, and trimmed the tests down to a single behavioural test covering the pool/connect_args split in get_engine().

Defaults are unchanged, so existing deployments behave exactly as before.

[tomkralidis]

@tomkralidis think that's addressed everything raised by @webb-ben and @ricardogsilva — let me know if I've missed anything.

Quick summary of what changed:

• get_engine() now pops the pool keys out of connect_args itself (@webb-ben's suggestion). That removes the separate db_pool_options tuple entirely, and the manager call site no longer needs any special handling — it just passes **self.db_options like the provider does. It stays functools.cache-able since the kwargs are hashable scalars.
• Added pool_size / max_overflow / pool_recycle / pool_timeout / pool_pre_ping to the config JSON Schema (@ricardogsilva), so a config can be validated before the server starts. That's now what enforces the types, so I've dropped the manual type(default)(...) coercion — which also removes the bool("False") is True gotcha you spotted.
• Removed the outdated pool_recycle comment, and trimmed the tests down to a single behavioural test covering the pool/connect_args split in get_engine().

Defaults are unchanged, so existing deployments behave exactly as before.

Thanks @KoalaGeo! See last minor change request on source code header for tests/provider/test_sql_pool_options.py

[KoalaGeo]

Done :-)