Repository navigation
Support config_constraints in AutoML - #1670
Open
Ankit Dhandharia (ankitdhandharia) wants to merge 1 commit into
Open
Ankit Dhandharia (ankitdhandharia) wants to merge 1 commit into
Ankit Dhandharia (ankitdhandharia) wants to merge 1 commit into
Conversation
Author
|
@microsoft-github-policy-service agree |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
flaml.tunehas long supportedconfig_constraints, but AutoML ignored them: both search-construction paths inflaml/automl/automl.pybuilt the searcher'sconfig_constraintsfrom a single hard-coded memory-size constraint and offered no way for a user or a custom estimator to constrain which hyperparameter configurations are explored.This adds the optional, maintainer-proposed hook
BaseEstimator.config_constraints()(default[]), which an estimator can override to declare constraints in the existing tuneconfig_constraintssyntax — a list of(function, operator, threshold)tuples, with operators<=,>=,<,>. AutoML now merges each estimator's constraints with the memory-size constraint:learner_class.config_constraints()directly (the constraint function receives the raw single-estimator config).sizedispatch), leaving other learners unconstrained.There is no behavior change for estimators that don't override the method (they return
[]). Configs that violate a constraint are skipped during search by the existing BlendSearch constraint-checking logic (_violate_config_constriants), so they are never trained.Related issue number
Closes #840
Checks
Testing
Added a regression test
test_config_constraintsintest/automl/test_constraints.py. It registers a custom estimator subclassingLGBMEstimatorwhoseconfig_constraintscapsn_estimators <= 16, fits AutoML on syntheticmake_classificationdata (n_jobs=1, the sequential search path), and asserts both thatbest_config["n_estimators"] <= 16and that every trained/logged config satisfies the constraint (read back from the training log).python -m pytest test/automl/test_constraints.py::test_config_constraints -q→ 1 failed (AutoML never consulted the hook, so configs withn_estimatorsup to 41 were trained;assert 41 <= 16failed).python -m pytest test/automl/test_constraints.py::test_config_constraints -q→ 1 passed.python -m pytest test/automl/test_constraints.py -q→ 3 passed (the two existing metric-constraint tests plus the new one).python -m pytest test/automl/test_classification.py -q -k "multiclass or binary or custom_learner or lrl1"→ 3 passed.