Skip to content

Check metric constraints when the constrained metric is exactly zero - #1671

Open
Mohammad Sohail Shaikh (sohailshk) wants to merge 1 commit into
microsoft:mainfrom
sohailshk:fix/metric-constraint-zero-value
Open

Mohammad Sohail Shaikh (sohailshk) wants to merge 1 commit into
microsoft:mainfrom
sohailshk:fix/metric-constraint-zero-value

Conversation

@sohailshk

Copy link
Copy Markdown

Why are these changes needed?

BlendSearch.on_trial_complete() applies the penalty only if value:, so a reported 0 / 0.0 skipped the constraint: with metric_constraints=[("recall", ">=", 0.5)] a trial reporting recall=0.0 was treated as feasible (its lagrange objective equal to the raw objective). Now the constraint is skipped only when the metric is missing (if value is not None).

Test: recall is 0.0 above x=5 with mode="max"; infeasible trials must get a lagrange-modified score below the raw score, feasible ones the raw score. It fails on main (7.71 < 7.71) and passes now.

Related issue number

Same pattern as #1603 (FLOW2 / tune.report); this check was not part of it.

Checks

Checks run: the new test, the tune/automl constraint tests and test_tune/test_lexiflow (9 passed, 1 skipped); black (line length 120) and ruff on the changed files. The pinned pre-commit hook versions were not run. No doc changes are needed.

BlendSearch.on_trial_complete() read the constrained metric with 'if value:',
so a reported value of 0 (for example recall 0.0 against recall >= 0.5) skipped
the constraint and the trial was treated as feasible. Only skip when the metric
is missing.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant