Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality

Table 10 reports the full zero-shot evaluation row for OPT-6.7B.

The Based and ReBased kernel-approximation variants do not improve over the corresponding simple activation or normalization baselines in the reported 130M and 380M ablations. The paper notes that the negative result may reflect the difference between SSD's 1-semiseparable mask and vanilla linear-attention methods, and that the expanded-feature variants use smaller B/C projections and adjusted layer counts.

Table 5 reports the 360M, 7B-token head-structure ablation. The MIS/MVA pattern again has the lowest reported perplexity, while MCS/MQA and MES/MKA are worse despite the same stated total-state comparison.

At 2.7B scale, trained for 300B Pile tokens with 64 layers and matched parameters, adding six attention layers to Mamba-2 improves the reported Pile and downstream results over pure Mamba-2 and Transformer++. Adding MLP layers alone reduces the reported quality, while the combined SSD/MLP/attention model remains competitive. The paper reports that MLP layers may improve hardware efficiency or ease conversion to mixture-of-experts models.

Table 5 reports the 125M, 2.5B-token head-structure ablation. Multi-input/Multi-value (MIS/MVA) has the lowest reported perplexity among the listed patterns; MQA/MCS and MKA/MES are substantially worse despite equal total state size. Parameter-matched multi-head patterns fall between these choices.

Table 10 reports the full zero-shot evaluation row for Hybrid H3-2.7B.

Table 10 reports the full zero-shot evaluation row for GPT-Neo 1.3B.

Table 10 reports the full zero-shot evaluation row for Mamba-2-1.3B.

Table 10 reports the full zero-shot evaluation row for RWKV4-3B.

Table 10 reports the full zero-shot evaluation row for Mamba-2-2.7B.

Table 10 reports the full zero-shot evaluation row for Mamba-2.8B.

The kernel-activation ablation finds no consistent improvement from the listed kernel approximations over simple pointwise activations. Random Feature Attention is the lowest-perplexity entry in the table, while Positive Random Features (Performer) is the highest; the paper retains Swish as the default and notes that removing the activation may be simpler but was not extensively tested.

Table 10 reports the full zero-shot evaluation row for Pythia-6.9B.

Table 10 reports the full zero-shot evaluation row for Pythia-2.8B.

In the Chinchilla-style Pile scaling-law plots, Mamba-2 matches or exceeds Mamba and the Transformer++ recipe across approximately 125M to 1.3B parameters. Relative to the authors' Transformer baseline, the paper describes Mamba-2 as Pareto dominant in perplexity, theoretical FLOPs, and actual wall-clock time.

In the Mamba-block ablation with SSD as the inner sequence-mixing layer, parallel ABCX projections use fewer parameters and slightly lower perplexity than sequential projections, while the extra normalization used in Mamba-2 slightly improves perplexity. The paper also reports preliminary larger-scale evidence that the extra normalization improves training stability.

Table 10 reports the full zero-shot evaluation row for OPT-2.7B.

Table 10 reports the full zero-shot evaluation row for Hybrid H3-1.3B.

Appendix Table 9 reports the model sizes, architectures, optimization settings, and token budgets used for the scaling-law experiments.

In a 350M, 48-layer Pile model trained for 7B tokens with the GPT-2 tokenizer, adding a small number of attention blocks to Mamba-2 improves validation perplexity. The best reported result is around a 10% attention-layer ratio; the paper says SSD and attention are complementary and that exact spacing is not very important in these small-scale experiments.

Table 10 reports the full zero-shot evaluation row for GPT-J-6B.

Table 10 reports the full zero-shot evaluation row for RWKV4-7.4B.

On the harder multi-query associative recall (MQAR) task, Mamba-1 struggles while Mamba-2 performs well across the tested settings. Mamba-2 is reported to be significantly better than Mamba-1 even when state size is controlled at N=16, and increasing Mamba-2 state size from N=16 to N=64 and N=256 consistently improves performance. Standard multi-head softmax attention and Based are included as baselines.

The paper reports that its SSD implementation is 2–8× faster than Mamba's fused associative scan for large state expansion, is faster than FlashAttention-2 at sequence lengths of 2K and above, and is 6× faster than FlashAttention-2 at sequence length 16K. At sequence length 4K, increasing state expansion slows the optimized Mamba scan approximately linearly, whereas SSD shows little slowdown.

The paper reports that the modified Mamba-2 block is tensor-parallel friendly and reduces synchronization points per block by half. It also describes sequence/context parallelism in which devices process contiguous sequence chunks and pass recurrent states between workers, with communication linear in the number of workers.

Table 10 reports the full zero-shot evaluation row for Hybrid H3-130M.

Table 10 reports the full zero-shot evaluation row for Mamba-130M.

Table 10 reports the full zero-shot evaluation row for Pythia-160M.

Table 10 reports the full zero-shot evaluation row for Mamba-2-130M.

Table 10 reports the full zero-shot evaluation row for Hybrid H3-360M.

Table 10 reports the full zero-shot evaluation row for Pythia-410M.

Table 10 reports the full zero-shot evaluation row for Mamba-370M.

Table 10 reports the full zero-shot evaluation row for Mamba-2-370M.

Table 10 reports the full zero-shot evaluation row for Pythia-1B.

Table 10 reports the full zero-shot evaluation row for Mamba-790M.

Table 10 reports the full zero-shot evaluation row for Mamba-2-780M.

Table 10 reports the full zero-shot evaluation row for OPT-1.3B.

Table 10 reports the full zero-shot evaluation row for Pythia-1.4B.

Table 10 reports the full zero-shot evaluation row for RWKV4-1.5B.

Table 10 reports the full zero-shot evaluation row for Mamba-1.4B.

Table 10 reports the full zero-shot evaluation row for GPT-Neo 2.7B.