[PerryLink]

Fixes #1229.

evaluate_accuracy rescales sklearn's macro average by len(label_names) / labels with support, to undo the zeros sklearn contributes for labels that never occur in the truth. The count that produces that denominator looked only at support, so a label that never occurs in the truth but is still predicted was dropped from it too. Every prediction of such a label is a false positive, so it has a measured precision of 0 — and dropping it both hid those false positives and inflated the rescale.

Measured before the change

Running the function with the mapping {0: PAD, 1: UNKNOWN, 2: ADDRESS, 3: SSN, 4: PERSON} and the default omitted_labels, on ground truth of 100 ADDRESS characters plus 100 background characters:

case	prediction	returned f1	macro P	micro P
control: perfect	—	1.0000	1.0	1.0
100 background characters tagged SSN	—	1.0000	1.0	0.5
100 background characters split over SSN + PERSON	—	1.0000	1.0	0.5
40 of 100 ADDRESS characters tagged SSN	—	0.7500	1.0	0.6
control: background tagged ADDRESS	—	0.6667	0.5	0.5
control: everything tagged SSN	—	0.0000	0.0	0.0

A model that tags background text with an absent label scored a perfect f1, and mislabelling 40% of one label's characters did not move the precision component at all.

Change

A label counts toward the average when it occurs in the truth (support) or when anything was predicted as it, which is its column of the confusion matrix being non-empty:

label_index = label_index_by_name.get(key)
predicted_as_label = label_index is not None and bool(
    conf_mat[:, label_index].sum()
)
if values["support"] or predicted_as_label:
    num_labels_with_positive_support += 1

After it, the same cases return 0.5000, 0.3333 and 0.3750 — the average over the labels that the truth or the prediction actually used. The controls are untouched, and a label that neither side used is still excluded.

Tests

Two cases added next to test_no_support_classes:

• test_predicted_label_without_support_lowers_f1 — the false positive now costs half the returned f1, and the per-label row shows precision 0 with support 0.
• test_label_neither_predicted_nor_in_truth_is_still_excluded — the rescale still ignores a label nobody used, which is what test_no_support_classes relies on.

All six cases in TestEvaluateAccuracy pass, and the four existing ones return exactly the values they did before. This environment cannot install TensorFlow, so I ran them through unittest with labeler_utils' module-level import tensorflow stubbed; nothing in evaluate_accuracy touches it. black, isort and flake8 are clean on both files.

Credit for the analysis in #1229 — the code location, the measured table and the direction of the fix are all from that report.

[CLAassistant]

All committers have signed the CLA.