fix(rm): deepscaler silently zeroes all rewards for non-thinking models#2219
Open
acmore wants to merge 1 commit into
Open
fix(rm): deepscaler silently zeroes all rewards for non-thinking models#2219acmore wants to merge 1 commit into
acmore wants to merge 1 commit into
Conversation
get_deepscaler_rule_based_reward returns 0 whenever the response contains neither "</think>" nor "###Response". For any non-thinking model (or a thinking model with stripped tags) every sample scores 0: GRPO groups all become zero-std, advantages vanish, and training runs with no signal — with no error or warning anywhere. The symptom is deceptive: generation looks healthy (sane lengths, ~0% truncation, correct answers visible in the text) while rewards stay flat at 0. Fall back to grading the whole response instead. Measured on DeepSeek-V3.2 (non-thinking chat template): reward 0 -> 0.676 on dapo-math-17k with this one-line change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
get_deepscaler_rule_based_rewardreturns0whenever the response contains neither</think>nor###Response. For any non-thinking model (or a thinking model whose chat template strips the tags), every sample scores 0:We hit this running DeepSeek-V3.2 GRPO (non-thinking chat template) on dapo-math-17k: several steps of flat-zero reward with otherwise perfect-looking rollouts.
Change
One-line fallback: when neither marker is present, grade the whole response instead of returning 0. Extraction/verification below this point (
extract_answer+grade_answer_mathd/grade_answer_sympy) already handles raw text fine.Verification
Same DeepSeek-V3.2 setup, only this change: reward went from
0.0(all steps) to0.676at step 0 with a normal GRPO signal thereafter. Thinking-model path (</think>present) is untouched.🤖 Generated with Claude Code