Repository navigation
Conversation
AmineDiro
marked this pull request as ready for review
November 19, 2026 08:39
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. Pages changed in this PR: |
Member
|
Thanks, nice work! Two things before it can land:
|
Member
Author
|
Thanks @qgallouedec ! Merged main:
|
AmineDiro
force-pushed
the
asyncgrpo-nemotron-tree
branch
from
November 19, 2026 21:38
a02eefc to
f912131
Compare
qgallouedec
approved these changes
Nov 19, 2026
qgallouedec
left a comment
Member
There was a problem hiding this comment.
Thanks, LGTM! The 78 tree packing tests pass on transformers 5.24 dev, and on 5.19 with LFM2 skipped as expected. I pushed a small doc commit: the docs listed the gated DeltaNet and Mamba2 families but never LFM2's short convolutions (since #7966).
Whichever of this and #8446 merges second needs to drop Granite-4.0-H's strict xfail.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #8233.
AsyncGRPO refused Nemotron-H at init, for two reasons:
moeandmlpblocks, which act on each token alone and need nothing from tree packing;What changes
Transformers runs its recurrent layers through module-level functions, shared by name across models.
patch_tree_ops(it replacespatch_tree_gdn) wraps them, by family:causal_conv1d_fn,torch_chunk_gated_delta_rulecausal_conv1d_fn,mamba2_chunk_scan,mamba2_split_conv1d_scan_combinedA model is accepted only if its modeling module has every function of a family, so a renamed function refuses at init instead of leaking state between samples. The wrapped function runs its tree version when the forward carries the tree, and the original otherwise.
Falcon-H1 and Zamba2 are Mamba2 too, but they have
hybridlayers (attention and Mamba in one block), so they are still refused.Kernels
trl/kernels/tree_mamba2.py): Mamba2's SSD is the gated delta rule without its delta term (queriesC, keysB, valuesdt * x, log-gatesdt * A, plusD * x), so it runs on the tree state and chunk kernels oftree_gated_delta_rule.py, which get aUSE_Wconstexpr to drop the WY term. Its own kernels: thedt * xandD * xterms, and adq/dkbackward that sums over the heads sharing a group ofBandC(split over programs of up to 8 heads, asmamba_ssmdoes for 1 group of 128 heads).mamba2_split_conv1d_scan_combined, and withoutmamba_ssmfalls back to an unfused torch path (fp32 gated RMSNorm, every intermediate saved for backward) that costs 1.5x the time and 5x the saved memory ofmamba_ssm's fused kernel. The tree version is the whole mixer afterin_proj: tree conv, tree scan, a Triton gated RMSNorm,out_proj. Likemamba_ssm's, it saves onlyzxbcdtand the scan output, and recomputes the rest in backward.causal_conv1d+ scatter of the previous version.Tests
tiny-NemotronHForCausalLM-nanoadded toTestTreePackingParity(packed loss and grads match each sample forwarded alone), removed fromTestTreePackingRefusesUnsupportedModels.TestTreePackingParitynow runs the model in train mode, the mode training uses, so the Mamba2 case goes through the fused tree mixer (and the reference through transformers' torch path).test_trainis removed: it trains end to end now.On H100, with transformers 5.17 and 5.20.2 (the latter with
causal_conv1dinstalled), the tree packing tests pass.Outside the test suite:
mamba2_chunk_scanon a 3-level tree, in bf16 and fp32, at the Mamba2 shapes of real configs: tiny Nemotron-H (8 x 4, 1 group, state 16), Granite-4.0-H-350M / Small, Bamba-9B, Zamba2-2.7B, Falcon-H1-1.5B / 7B / 34B (state 256), NVIDIA-Nemotron-Nano-12B-v2 (head dim 80), Nemotron 3 Nano, Nemotron-H-56B. Output and every input gradient match within 1e-2 relative (fp32 is no tighter,tl.dotruns in TF32). A 64k-token row with more than 2^31 state elements matches the same samples run as two 32k rows;causal_conv1don a gathered row): identical outputs, gradients within bf16 rounding.Real Nemotron sizes
Forward and backward through tree attention and the patched Mamba2 layers, compared with each sample forwarded alone through FlashAttention-3 and transformers' own Mamba2 path (H100, bf16, transformers 5.20.2).
MMM*MMM*of 23 Mamba + 6 attentionMMMM*MMMof 40 Mamba + 8 attentionMMMM*MMMof 48 Mamba + 12 attentionSharing prefixes adds no error: every tree number matches its no-sharing floor.
Tree packing vs plain packing
Forward + backward on H100, bf16, transformers 5.20.2, of real-config layer stacks with random weights:
Samples are 4096 tokens, in groups of 8 sharing a prompt, in rows of about 32k tokens. The packing ratio is raw tokens / tree-packed tokens in a row.
mamba_ssm's fused Mamba2 kernel (kernels-community/mamba-ssmv3, whatuse_kernels=Trueloads) withseq_idx. Its time for the samples of a tree row is (raw tokens / 32k) x its time for one row.On the same samples (2 groups of 8 sharing a 768-token prompt), tree packing matches plain packing as closely as the tree kernels do with nothing shared (each sample alone through the tree path, in parentheses):
With nothing shared, tree packing runs at 0.92x to 0.97x plain packing's speed with the same or less memory. From there the speedup grows with the packing ratio, up to 5.0x to 5.3x at a ratio of 5.56. Peak memory per row follows the row's packed tokens (30.7k to 36.9k here).
Granite-4.0-H and Bamba (and Mamba2) are accepted by the same check and are covered above, but CI only has a tiny Nemotron-H: the tiny Granite-4.0-H checkpoint has a head dim of 4, which FA3 rejects.
Note
High Risk
Adds custom Triton autograd for tree conv/Mamba2 and patches core modeling entrypoints—incorrect segment boundaries would leak state between packed samples during RL training.
Overview
AsyncGRPO tree packing now supports Mamba2 hybrids (e.g. Nemotron-H / Nemotron 3) alongside gated DeltaNet and LFM2.
patch_tree_recurrentwraps transformers’ module-level ops by family (gated DeltaNet, Mamba2, or short conv), and init checks treatmoe/mlpblocks as attention-like while still rejecting unknown recurrent layers.Kernels: New
trl/kernels/tree_mamba2.pyimplements tree SSD scan and a fused tree mixer (conv + scan + gated RMSNorm). Sharedtree_gated_delta_rulegains aUSE_Wpath for scan-without-delta (Mamba2), Triton tree depthwise conv with autograd (replacing gather/scatter), and larger state key dims (up to 256).Tests & docs:
tiny-NemotronHForCausalLM-nanotrains and passes tree-packing parity; unsupported-model tests no longer expect Nemotron-H to fail at init.Reviewed by Cursor Bugbot for commit 26ead3c. Bugbot is set up for automated code reviews on this repo. Configure here.