Skip to content

Load training files in a stable order across ranks - #1627

Open
Afloat16 wants to merge 1 commit into
FlagOpen:masterfrom
Afloat16:fix/sorted-training-files-20261112
Open

Afloat16 wants to merge 1 commit into
FlagOpen:masterfrom
Afloat16:fix/sorted-training-files-20261112

Conversation

@Afloat16

Copy link
Copy Markdown

Problem

Directory-backed embedder and reranker training load JSON/JSONL files in the order returned by os.listdir. That order can differ between hosts even when every host has the same files and the training seed is identical.

This changes concatenated row indices and capped per-file sampling. For same-dataset training, ranks can therefore combine examples from different datasets in one global batch and duplicate or omit rows across the epoch.

Change

Sort directory entries in the three training loaders before loading or sampling files. Preserve the caller's train_data order and the existing JSON/JSONL filtering.

Validation

PYTHONPATH=. python -m pytest tests/test_finetune_file_order.py tests/test_finetune_trainer_compat.py -q --tb=short: 11 passed.

The seven new regression cases fail on unchanged master and pass with this patch. They exercise the real dataset constructors and local JSONL data under reversed directory listings:

  • Stable concatenated rows in the embedder, same-dataset embedder and reranker loaders, both with and without per-file sample caps.
  • Two rank-specific same-dataset configurations with opposite file enumeration orders, checking that each combined batch belongs to one dataset and every input row is covered exactly once.

The four existing trainer compatibility tests also pass. Ruff check and formatting pass for the new tests, and git diff --check passes.

Validation used Python 3.12.14, PyTorch 2.15.0+cpu, Transformers 4.57.1 and Datasets 5.0.1. The rank test exercises dataset partitioning locally; a distributed GPU training job was not run.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant