Skip to content

Support Nemotron-H tree packing in AsyncGRPO - #8457

Open
AmineDiro wants to merge 2 commits into
mainfrom
asyncgrpo-nemotron-tree
Open

AmineDiro wants to merge 2 commits into
mainfrom
asyncgrpo-nemotron-tree

Conversation

@AmineDiro

@AmineDiro AmineDiro commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Resolves #8233.

AsyncGRPO refused Nemotron-H at init, for two reasons:

  • the layer-type check didn't know moe and mlp blocks, which act on each token alone and need nothing from tree packing;
  • nothing made the Mamba2 layers follow the tree, so their conv and scan would run from one branch into its sibling.

What changes

Transformers runs its recurrent layers through module-level functions, shared by name across models. patch_tree_ops (it replaces patch_tree_gdn) wraps them, by family:

family functions models
gated DeltaNet causal_conv1d_fn, torch_chunk_gated_delta_rule Qwen3.5, Qwen3-Next, OLMo-Hybrid, ...
Mamba2 causal_conv1d_fn, mamba2_chunk_scan, mamba2_split_conv1d_scan_combined Nemotron-H, Granite-4.0-H, Bamba, Mamba2

A model is accepted only if its modeling module has every function of a family, so a renamed function refuses at init instead of leaking state between samples. The wrapped function runs its tree version when the forward carries the tree, and the original otherwise.

Falcon-H1 and Zamba2 are Mamba2 too, but they have hybrid layers (attention and Mamba in one block), so they are still refused.

Kernels

  • Tree scan (trl/kernels/tree_mamba2.py): Mamba2's SSD is the gated delta rule without its delta term (queries C, keys B, values dt * x, log-gates dt * A, plus D * x), so it runs on the tree state and chunk kernels of tree_gated_delta_rule.py, which get a USE_W constexpr to drop the WY term. Its own kernels: the dt * x and D * x terms, and a dq/dk backward that sums over the heads sharing a group of B and C (split over programs of up to 8 heads, as mamba_ssm does for 1 group of 128 heads).
  • Fused tree mixer: in training, transformers' Mamba2 mixer calls the fused mamba2_split_conv1d_scan_combined, and without mamba_ssm falls back to an unfused torch path (fp32 gated RMSNorm, every intermediate saved for backward) that costs 1.5x the time and 5x the saved memory of mamba_ssm's fused kernel. The tree version is the whole mixer after in_proj: tree conv, tree scan, a Triton gated RMSNorm, out_proj. Like mamba_ssm's, it saves only zxbcdt and the scan output, and recomputes the rest in backward.
  • Tree conv (both families): a Triton depthwise conv whose window runs back along each token's root path, replacing the gather + causal_conv1d + scatter of the previous version.
  • The tree state kernels now take state sizes up to 256, as in FLA (Nemotron-H-56B has 256; the vendored kernels stopped at 128). These are constexpr branches: with Triton autotune pinned to one config, gated DeltaNet's chunk kernels give bit-identical outputs and gradients to main.

Tests

  • tiny-NemotronHForCausalLM-nano added to TestTreePackingParity (packed loss and grads match each sample forwarded alone), removed from TestTreePackingRefusesUnsupportedModels.
  • TestTreePackingParity now runs the model in train mode, the mode training uses, so the Mamba2 case goes through the fused tree mixer (and the reference through transformers' torch path).
  • Nemotron-H's strict xfail in test_train is removed: it trains end to end now.

On H100, with transformers 5.17 and 5.20.2 (the latter with causal_conv1d installed), the tree packing tests pass.

Outside the test suite:

  • the tree scan against transformers' per-sample mamba2_chunk_scan on a 3-level tree, in bf16 and fp32, at the Mamba2 shapes of real configs: tiny Nemotron-H (8 x 4, 1 group, state 16), Granite-4.0-H-350M / Small, Bamba-9B, Zamba2-2.7B, Falcon-H1-1.5B / 7B / 34B (state 256), NVIDIA-Nemotron-Nano-12B-v2 (head dim 80), Nemotron 3 Nano, Nemotron-H-56B. Output and every input gradient match within 1e-2 relative (fp32 is no tighter, tl.dot runs in TF32). A 64k-token row with more than 2^31 state elements matches the same samples run as two 32k rows;
  • the fused tree mixer against transformers' torch mixer on each sample alone, at the tiny, Nemotron 3 Nano, 1-group-of-128-heads and state-256 shapes: output and every gradient within 1.4e-2;
  • the tree conv against the previous implementation (causal_conv1d on a gathered row): identical outputs, gradients within bf16 rounding.

Real Nemotron sizes

Forward and backward through tree attention and the patched Mamba2 layers, compared with each sample forwarded alone through FlashAttention-3 and transformers' own Mamba2 path (H100, bf16, transformers 5.20.2).

  • Models: built from each Hub config with random weights and the real Mamba2 and attention dims. The layer pattern is cut to its first 8 sequence-mixing layers. MoE blocks are left out: they act on each token alone, and random-weight routing flips on bf16 noise.
  • Samples: 12 samples in a 3-level tree, 1947 raw tokens of which 1308 are unique.
  • Columns: relative error vs the per-sample reference. In parentheses is the same model run on each sample alone through the tree path (no sharing). That's the noise floor of the kernels vs transformers' fp32 torch path.
Model hidden Mamba heads x dim groups state attn heads/kv x dim layers tested output grad Mamba grad attention grad all
Nemotron 3 Nano / 3.5 Lightning 2688 64x64 8 128 32/2x128 MMM*MMM* of 23 Mamba + 6 attention 1.5e-02 (1.5e-02) 2.0e-02 (2.0e-02) 2.4e-02 (2.4e-02) 2.0e-02 (2.0e-02)
Nemotron 3 Super 4096 128x64 8 128 32/2x128 MMMM*MMM of 40 Mamba + 8 attention 1.6e-02 (1.6e-02) 2.3e-02 (2.3e-02) 2.9e-02 (2.9e-02) 2.3e-02 (2.3e-02)
Nemotron 3 Ultra 8192 256x64 8 128 64/2x128 MMMM*MMM of 48 Mamba + 12 attention 2.0e-02 (2.0e-02) 3.5e-02 (3.5e-02) 4.1e-02 (4.1e-02) 3.6e-02 (3.6e-02)

Sharing prefixes adds no error: every tree number matches its no-sharing floor.

Tree packing vs plain packing

Forward + backward on H100, bf16, transformers 5.20.2, of real-config layer stacks with random weights:

  • Nemotron 3 Nano and Super: first 8 sequence mixers (Mamba2 and attention), MoE blocks left out as above.
  • Granite-4.0-H-Small and Bamba-9B: 6 layers, each a mixer with its MLP, the last one attention (Granite's own pattern; Bamba's first attention layer is its 10th). Granite's routed experts are left out (its shared MLP stays): they act on each token alone, and at 32k tokens they don't fit on one GPU next to plain packing's activations.

Samples are 4096 tokens, in groups of 8 sharing a prompt, in rows of about 32k tokens. The packing ratio is raw tokens / tree-packed tokens in a row.

  • plain packing: samples packed end to end, FlashAttention-3 varlen and mamba_ssm's fused Mamba2 kernel (kernels-community/mamba-ssm v3, what use_kernels=True loads) with seq_idx. Its time for the samples of a tree row is (raw tokens / 32k) x its time for one row.
  • tree packing: one tree-packed row through tree attention and this PR's Mamba2 kernels.

On the same samples (2 groups of 8 sharing a 768-token prompt), tree packing matches plain packing as closely as the tree kernels do with nothing shared (each sample alone through the tree path, in parentheses):

model output gradients
Nemotron 3 Nano 1.1e-02 (1.1e-02) 1.8e-02 (1.8e-02)
Nemotron 3 Super 1.5e-02 (1.5e-02) 2.3e-02 (2.3e-02)
Granite-4.0-H-Small 2.3e-02 (2.3e-02) 4.2e-02 (4.3e-02)
Bamba-9B 1.7e-02 (1.7e-02) 2.4e-02 (2.4e-02)

Uploading nemotron_tree_vs_plain_speedup.png…

nemotron_tree_vs_plain_time nemotron_tree_vs_plain_memory
model packing ratio (raw / packed tokens) plain packing tree packing speedup
Nemotron 3 Nano 1.00 (32768 / 32768) 200 ms, 15.7 GiB 208 ms, 15.0 GiB 0.96x
Nemotron 3 Nano 1.78 (65536 / 36878) 399 ms, 15.7 GiB 242 ms, 17.0 GiB 1.65x
Nemotron 3 Nano 2.91 (98304 / 33813) 599 ms, 15.7 GiB 224 ms, 15.7 GiB 2.67x
Nemotron 3 Nano 4.26 (131072 / 30748) 799 ms, 15.7 GiB 204 ms, 14.3 GiB 3.92x
Nemotron 3 Nano 5.56 (196608 / 35370) 1198 ms, 15.7 GiB 239 ms, 16.6 GiB 5.01x
Nemotron 3 Super 1.00 (32768 / 32768) 434 ms, 29.5 GiB 448 ms, 28.5 GiB 0.97x
Nemotron 3 Super 1.78 (65536 / 36878) 869 ms, 29.5 GiB 504 ms, 32.2 GiB 1.72x
Nemotron 3 Super 2.91 (98304 / 33813) 1303 ms, 29.5 GiB 460 ms, 29.6 GiB 2.83x
Nemotron 3 Super 4.26 (131072 / 30748) 1737 ms, 29.5 GiB 419 ms, 27.0 GiB 4.14x
Nemotron 3 Super 5.56 (196608 / 35370) 2606 ms, 29.5 GiB 489 ms, 31.1 GiB 5.33x
Granite-4.0-H-Small 1.00 (32768 / 32768) 368 ms, 25.7 GiB 399 ms, 25.7 GiB 0.92x
Granite-4.0-H-Small 1.78 (65536 / 36878) 737 ms, 25.7 GiB 454 ms, 29.1 GiB 1.62x
Granite-4.0-H-Small 2.91 (98304 / 33813) 1105 ms, 25.7 GiB 420 ms, 26.9 GiB 2.63x
Granite-4.0-H-Small 4.26 (131072 / 30748) 1473 ms, 25.7 GiB 380 ms, 24.6 GiB 3.88x
Granite-4.0-H-Small 5.56 (196608 / 35370) 2210 ms, 25.7 GiB 440 ms, 28.5 GiB 5.03x
Bamba-9B 1.00 (32768 / 32768) 648 ms, 44.5 GiB 677 ms, 44.5 GiB 0.96x
Bamba-9B 1.78 (65536 / 36878) 1295 ms, 44.5 GiB 760 ms, 50.2 GiB 1.70x
Bamba-9B 2.91 (98304 / 33813) 1943 ms, 44.5 GiB 697 ms, 46.2 GiB 2.79x
Bamba-9B 4.26 (131072 / 30748) 2590 ms, 44.5 GiB 640 ms, 42.2 GiB 4.05x
Bamba-9B 5.56 (196608 / 35370) 3885 ms, 44.5 GiB 740 ms, 48.7 GiB 5.25x

With nothing shared, tree packing runs at 0.92x to 0.97x plain packing's speed with the same or less memory. From there the speedup grows with the packing ratio, up to 5.0x to 5.3x at a ratio of 5.56. Peak memory per row follows the row's packed tokens (30.7k to 36.9k here).

Granite-4.0-H and Bamba (and Mamba2) are accepted by the same check and are covered above, but CI only has a tiny Nemotron-H: the tiny Granite-4.0-H checkpoint has a head dim of 4, which FA3 rejects.


Note

High Risk
Adds custom Triton autograd for tree conv/Mamba2 and patches core modeling entrypoints—incorrect segment boundaries would leak state between packed samples during RL training.

Overview
AsyncGRPO tree packing now supports Mamba2 hybrids (e.g. Nemotron-H / Nemotron 3) alongside gated DeltaNet and LFM2. patch_tree_recurrent wraps transformers’ module-level ops by family (gated DeltaNet, Mamba2, or short conv), and init checks treat moe/mlp blocks as attention-like while still rejecting unknown recurrent layers.

Kernels: New trl/kernels/tree_mamba2.py implements tree SSD scan and a fused tree mixer (conv + scan + gated RMSNorm). Shared tree_gated_delta_rule gains a USE_W path for scan-without-delta (Mamba2), Triton tree depthwise conv with autograd (replacing gather/scatter), and larger state key dims (up to 256).

Tests & docs: tiny-NemotronHForCausalLM-nano trains and passes tree-packing parity; unsupported-model tests no longer expect Nemotron-H to fail at init.

Reviewed by Cursor Bugbot for commit 26ead3c. Bugbot is set up for automated code reviews on this repo. Configure here.

@AmineDiro
AmineDiro marked this pull request as ready for review November 19, 2026 08:39
@bot-ci-comment

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Pages changed in this PR:

@qgallouedec

Copy link
Copy Markdown
Member

Thanks, nice work!
The tree parity tests pass on my side.

Two things before it can land:

@AmineDiro

Copy link
Copy Markdown
Member Author

Thanks @qgallouedec ! Merged main:

@AmineDiro
AmineDiro force-pushed the asyncgrpo-nemotron-tree branch from a02eefc to f912131 Compare November 19, 2026 21:38

@qgallouedec qgallouedec left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM! The 78 tree packing tests pass on transformers 5.24 dev, and on 5.19 with LFM2 skipped as expected. I pushed a small doc commit: the docs listed the gated DeltaNet and Mamba2 families but never LFM2's short convolutions (since #7966).

Whichever of this and #8446 merges second needs to drop Granite-4.0-H's strict xfail.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support Nemotron-H in AsyncGRPO

2 participants