Skip to content

Avoid cross-query score matrices for local-negative training - #1631

Open
Afloat16 wants to merge 1 commit into
FlagOpen:masterfrom
Afloat16:perf/local-negative-score-matrix-20261112
Open

Afloat16 wants to merge 1 commit into
FlagOpen:masterfrom
Afloat16:perf/local-negative-score-matrix-20261112

Conversation

@Afloat16

Copy link
Copy Markdown

Problem

When dense embedder training disables in-batch negatives, the default local-scoring path still builds scores for every query against every passage and then selects each query's own passage group. For B queries and G passages per query, this materializes a B × (B × G) matrix although the objective uses only B × G entries.

Changes

The three built-in dense embedder models use their existing batched similarity operation to compute each query's own passage group directly. The abstract embedder's two-dimensional scoring contract stays unchanged. Explicit scoring callbacks and dense subclasses that override compute_score retain the original path, so the optimization does not add a new shape requirement to custom models.

This is separate from #1609, which corrects M3 local-target and ensemble layouts. This patch leaves those loss and M3 callback paths unchanged.

Validation

  • The 31 new cases produce 2 allocation failures and 29 passes on the unchanged base, and 31 passes after this patch.
  • Compare local scores and embedding gradients with independent per-query dot products in float32/float64, including non-contiguous tensors.
  • Native tiny BERT, GPT-2 and decoder ICL training preserves supervised/KL losses and parameter gradients, with and without MRL. Native forward gradient comparisons use float64; the existing MRL float32 loss accumulator is retained.
  • Custom callbacks, a custom two-dimensional abstract embedder, and custom scoring overrides of all three dense model classes keep their existing behavior.
  • CPU profiling at B=128, G=8, D=16 shows the largest operator allocation in local scoring falling from 524,288 to 4,096 bytes for float32 and from 1,048,576 to 8,192 bytes for float64. These figures cover this scoring operation, not total model memory or GPU throughput.
  • New tests plus existing Trainer/import modules: 39 passed, 1 skipped on PyTorch 2.15.0+cpu and Transformers 4.57.1. The skipped check requires Transformers 5.x. No pretrained checkpoint or CUDA benchmark was used.
  • Ruff and git diff --check pass.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant