Skip to content

Add GLM-4 MoE TransformerBridge adapter#1428

Open
zeotrix wants to merge 4 commits into
TransformerLensOrg:devfrom
zeotrix:fix/issue-1404
Open

Add GLM-4 MoE TransformerBridge adapter#1428
zeotrix wants to merge 4 commits into
TransformerLensOrg:devfrom
zeotrix:fix/issue-1404

Conversation

@zeotrix

@zeotrix zeotrix commented Jun 22, 2026

Copy link
Copy Markdown

Adds TransformerBridge support for Glm4MoeForCausalLM, covering the Z.ai GLM-4.5 / 4.6 / 4.7 MoE family.

This includes:

  • A new GLM-4 MoE architecture adapter with RMSNorm, RoPE, GQA, Q/K norm, and MoE routing support.
  • Registration in the architecture factory, supported architectures package, model registry, and report generation.
  • Tiny checkpoint registry verification for trl-internal-testing/tiny-Glm4MoeForCausalLM.
  • Unit tests for adapter config, component mapping, weight conversions, registration, and component-test setup.
  • Integration parity test against HuggingFace on the tiny checkpoint.
  • A small verify_models cleanup guard so non-MPS machines do not call MPS synchronization during CPU verification.

Closes #1404

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Screenshots

Not applicable.

Checklist:

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have not rewritten tests relating to key interfaces which would affect backward compatibility

Validation

uv run pytest tests/unit/tools/test_model_registry.py -k TestRegistrySyncedWithFactory
uv run pytest tests/unit/model_bridge/supported_architectures/test_glm4_moe_adapter.py
uv run pytest tests/integration/model_bridge/test_glm4_moe_bridge.py
HF_HUB_DISABLE_XET=1 uv run python -m transformer_lens.tools.model_registry.verify_models --model trl-internal-testing/tiny-Glm4MoeForCausalLM --device cpu
uv run mypy .

Tiny checkpoint verification result:

```text
VERIFIED: P1=100.0%, P2=100.0%, P3=100.0%, P4=65.8%, P7=None%, P8=None%
Total tested: 1
Verified: 1
Failed: 0

@jlarson4 jlarson4 changed the base branch from main to dev June 22, 2026 17:20
@jlarson4

Copy link
Copy Markdown
Collaborator

@zeotrix Great work on this! I had to rebase to dev to resolve some merge conflicts, assuming it passes CI I will merge

@zeotrix

zeotrix commented Jun 22, 2026

Copy link
Copy Markdown
Author

@zeotrix Great work on this! I had to rebase to dev to resolve some merge conflicts, assuming it passes CI I will merge

Thanks! I appreciate you handling the rebase. I'll keep an eye on the CI results. If any issues or regressions show up, feel free to tag me or assign them to me and I'll investigate.

@jlarson4

Copy link
Copy Markdown
Collaborator

@zeotrix Great work on this! I had to rebase to dev to resolve some merge conflicts, assuming it passes CI I will merge

Thanks! I appreciate you handling the rebase. I'll keep an eye on the CI results. If any issues or regressions show up, feel free to tag me or assign them to me and I'll investigate.

I try to handle any rebases that touch the supported_models doc, it can get hairy when a lot of people are adding to it at once. It is looking like everything will pass! I will rerun the Bridge_Evals_Demo once Full Coverage completes, which should clear your one failure that was caused by those pesky 429 errors

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Proposal] Add GLM-4.5 MoE adapter (Glm4MoeForCausalLM)

3 participants