Skip to content

feat: dpo precompute ref logps - #72

Open
Neonkraft wants to merge 5 commits into
mainfrom
feat/dpo-precompute-ref-logps
Open

Neonkraft wants to merge 5 commits into
mainfrom
feat/dpo-precompute-ref-logps

Conversation

@Neonkraft

Copy link
Copy Markdown
Collaborator

Summary

This PR adds the option to precompute the DPO reference log-probs. It also fixes two problems found while running DPO with FSDP2 on MN5.

1. Precomputed reference log-probs (dpo.precompute_ref_log_probs, dpo.precompute_ref_batch_size)

Two new DPOMethodConfig fields are passed through to TRL's DPOConfig:

  • precompute_ref_log_probs: true scores the dataset with the reference model once, before training, and frees the reference model afterwards. During training, only the policy model is in GPU memory.
  • precompute_ref_batch_size sets the batch size of that pass. null = per_device_train_batch_size.

Both default to the previous behavior (false / null). configs/trl/dpo.yaml and the README document the new fields.

2. Reference model on the wrong device under FSDP (device_map: None)

TRL defaults device_map to "auto" and forces it to None only for MULTI_GPU and DEEPSPEED. Under FSDP, the DPO reference model kept "auto": FSDP2 moved its weights to the local GPU, but the accelerate hooks still sent the inputs to cuda:0. The result was Expected all tensors to be on the same device. build_model_init_kwargs now sets device_map: None, so the model loads on CPU and FSDP places the weights.

3. wandb start-up timeout (WANDB__SERVICE_WAIT)

wandb waits 30 s for its wandb-core service to start. On a busy compute node, this is sometimes not enough, and rank 0 crashes in on_train_begin with:

wandb.sdk.lib.service.service_port_file.ServicePollForTokenError: Failed to read port info after 30.0 seconds

scripts/train.py now sets WANDB__SERVICE_WAIT=300 with os.environ.setdefault, so a value set in the environment (for example, in env/*.env) still takes priority.

4. MN5 config

configs/trl/olmo3-dpo-mn5.yaml: DPO on OLMo 3 7B Instruct SFT on MN5 with FSDP2, FA3, and precomputed reference log-probs. Paths, account, and partition are placeholders.

Type of change

  • Bug fix
  • New feature
  • Refactor
  • Performance
  • Documentation
  • Maintenance

Validation

  • wandb timeout: Mocked a slow wandb-core start locally (wandb 0.25.0). A wrapper script sleeps 40 s and then starts the real binary. Without the change, wandb.init fails with the same ServicePollForTokenError after 30 s. With the change, wandb.init succeeds.
  • Small test run passed on MN5
  • uv run pre-commit run --from-ref main --to-ref HEAD passes (ruff, ruff-format, black).

@Neonkraft Neonkraft changed the title Feat/dpo precompute ref logps feat: dpo precompute ref logps Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants