Skip to content

Load the tokenizer before it is used in save_merged_model - #1638

Open
alleyxd wants to merge 1 commit into
FlagOpen:masterfrom
alleyxd:fix-merged-model-tokenizer
Open

alleyxd wants to merge 1 commit into
FlagOpen:masterfrom
alleyxd:fix-merged-model-tokenizer

Conversation

@alleyxd

@alleyxd alleyxd commented Nov 17, 2026

Copy link
Copy Markdown

In the decoder-only embedder (finetune/embedder/decoder_only/base/load_model.py), save_merged_model uses tokenizer before it is created. Since 3bc1962 ("only modify vocab_size when necessary") the vocab_size update sits inside the emb.pth branch, above the line that loads the tokenizer, so merging a LoRA run that saved its embeddings (added tokens) fails with NameError: name 'tokenizer' is not defined at the end of training (runner.py calls it when save_merged_lora_model is set).

This loads the tokenizer just before that branch and keeps the update conditional, as 3bc1962 intended. Nothing else changes; the ICL variant already loads the tokenizer first.

Found with ruff check --select F821, which no longer reports it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant