Skip to content

Support config_constraints in AutoML - #1670

Open
Ankit Dhandharia (ankitdhandharia) wants to merge 1 commit into
microsoft:mainfrom
ankitdhandharia:fix/issue-840
Open

Ankit Dhandharia (ankitdhandharia) wants to merge 1 commit into
microsoft:mainfrom
ankitdhandharia:fix/issue-840

Conversation

@ankitdhandharia

Copy link
Copy Markdown

Why are these changes needed?

flaml.tune has long supported config_constraints, but AutoML ignored them: both search-construction paths in flaml/automl/automl.py built the searcher's config_constraints from a single hard-coded memory-size constraint and offered no way for a user or a custom estimator to constrain which hyperparameter configurations are explored.

This adds the optional, maintainer-proposed hook BaseEstimator.config_constraints() (default []), which an estimator can override to declare constraints in the existing tune config_constraints syntax — a list of (function, operator, threshold) tuples, with operators <=, >=, <, >. AutoML now merges each estimator's constraints with the memory-size constraint:

  • Sequential per-estimator search appends learner_class.config_constraints() directly (the constraint function receives the raw single-estimator config).
  • Joint/parallel search wraps each estimator's constraints so they apply only to configs whose selected learner is that estimator (mirroring the existing size dispatch), leaving other learners unconstrained.

There is no behavior change for estimators that don't override the method (they return []). Configs that violate a constraint are skipped during search by the existing BlendSearch constraint-checking logic (_violate_config_constriants), so they are never trained.

Related issue number

Closes #840

Checks

Testing

Added a regression test test_config_constraints in test/automl/test_constraints.py. It registers a custom estimator subclassing LGBMEstimator whose config_constraints caps n_estimators <= 16, fits AutoML on synthetic make_classification data (n_jobs=1, the sequential search path), and asserts both that best_config["n_estimators"] <= 16 and that every trained/logged config satisfies the constraint (read back from the training log).

  • Before the fix: python -m pytest test/automl/test_constraints.py::test_config_constraints -q → 1 failed (AutoML never consulted the hook, so configs with n_estimators up to 41 were trained; assert 41 <= 16 failed).
  • After the fix: python -m pytest test/automl/test_constraints.py::test_config_constraints -q → 1 passed.
  • Full file: python -m pytest test/automl/test_constraints.py -q → 3 passed (the two existing metric-constraint tests plus the new one).
  • Neighbouring: python -m pytest test/automl/test_classification.py -q -k "multiclass or binary or custom_learner or lrl1" → 3 passed.

@ankitdhandharia

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support config_constraints in AutoML similar to Tune

1 participant