At 2.7B scale, trained for 300B Pile tokens with 64 layers and matched parameters, adding six attention layers to Mamba-2 improves the reported Pile and downstream results over pure Mamba-2 and Transformer++. Adding MLP layers alone reduces the reported quality, while the combined SSD/MLP/attention model remains competitive. The paper reports that MLP layers may improve hardware efficiency or ease conversion to mixture-of-experts models.

At 2.7B scale, trained for 300B Pile tokens with 64 layers and matched parameters, adding six attention layers to Mamba-2 improves the reported Pile and downstream results over pure Mamba-2 and Transformer++. Adding MLP layers alone reduces the reported quality, while the combined SSD/MLP/attention model remains competitive. The paper reports that MLP layers may improve hardware efficiency or ease conversion to mixture-of-experts models. · CiteArk

Measurement results

0/50 assessed

0 supported0 disputed or conflicting50 not assessed

The complete result group stays with its claim. Assessed rows come first; their judgments do not replace the whole-claim conclusion. Expand a row for conditions, sources, and evidence history.

Each row is a measurement with its reported value, latest available observation, and cumulative evidence state.
Reported	Latest observation	Evidence judgment
6.13 perplexity	—	Not assessed
3.99 perplexity	—	Not assessed
70.3%	—	Not assessed
66.4%	—	Not assessed
75.2%	—	Not assessed
67.7%	—	Not assessed
37.8%	—	Not assessed
63.9%	—	Not assessed
40.4%	—	Not assessed
60.2%	—	Not assessed
6.09 perplexity	—	Not assessed
4.1 perplexity	—	Not assessed

Reported and observed measurements

Assessments (0)

Runs (0)

Measurement results