Skip to content

fix(train): decode TRL 1.x DPO columns in the tokenized preview - #73

Merged
Neonkraft merged 2 commits into
mainfrom
fix/dpo-tokenized-preview
Nov 12, 2026
Merged

Neonkraft merged 2 commits into
mainfrom
fix/dpo-tokenized-preview

Conversation

@Neonkraft

Copy link
Copy Markdown
Collaborator

Problem

The tokenized preview in scripts/train.py decoded only columns whose names end in input_ids. TRL 0.x named the DPO columns prompt_input_ids / chosen_input_ids / rejected_input_ids, but TRL 1.x renamed them to prompt_ids / chosen_ids / rejected_ids. The DPO preview therefore printed [Sample 0] with nothing under it. SFT (input_ids) was not affected.

Fix

Add _is_token_id_column(), which decodes input_ids, any *_input_ids column (older TRL), and prompt_ids / chosen_ids / rejected_ids / completion_ids. An explicit list is used instead of a _ids suffix match, so columns such as token_type_ids are not decoded.

Testing

--tokenize-only on a small CPU config (SmolLM2-135M, olmo3-instruct-sft template), with TRL 1.7.0 and TRL 1.17.1:

  • DPO: the preview now decodes the prompt, chosen, and rejected columns.
  • SFT: output unchanged.

Sample DPO output (TRL 1.17.1):

  [Sample 0]
    prompt_ids  (length 147)
      <|im_start|>system
      You are a helpful function-calling AI assistant. ...
      <|im_start|>user
      Write a program that reads a s ... [truncated, 362 more chars]
    chosen_ids  (length 192)
      Here's a simple Python program that reads a sequence of integers ...
    rejected_ids  (length 215)
      def multiply_by_two(input_line): ...

@Neonkraft
Neonkraft requested review from Ali-Elganzory and a balanced review from Copilot November 12, 2026 10:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the tokenized-preview compatibility update.

No open issues 🎉

What changed in this PR

Fixes the training script’s tokenized preview to recognize TRL 1.x DPO column names while preserving support for SFT and older TRL formats.

Changes:

  • Adds _is_token_id_column() to recognize supported token-ID columns explicitly.
  • Uses the helper when decoding preview samples, avoiding unrelated columns such as token_type_ids.
File Description
scripts/​train.py Adds token-ID column detection and updates preview decoding to support TRL 1.x DPO datasets.

Review effort: Balanced

@Neonkraft
Neonkraft merged commit f573741 into main Nov 12, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants