In a 350M, 48-layer Pile model trained for 7B tokens with the GPT-2 tokenizer, adding a small number of attention blocks to Mamba-2 improves validation perplexity. The best reported result is around a 10% attention-layer ratio; the paper says SSD and attention are complementary and that exact spacing is not very important in these small-scale experiments.

In a 350M, 48-layer Pile model trained for 7B tokens with the GPT-2 tokenizer, adding a small number of attention blocks to Mamba-2 improves validation perplexity. The best reported result is around a 10% attention-layer ratio; the paper says SSD and attention are complementary and that exact spacing is not very important in these small-scale experiments. · CiteArk

Measurement results

0/13 assessed

0 supported0 disputed or conflicting13 not assessed

The complete result group stays with its claim. Assessed rows come first; their judgments do not replace the whole-claim conclusion. Expand a row for conditions, sources, and evidence history.

Each row is a measurement with its reported value, latest available observation, and cumulative evidence state.
Reported	Latest observation	Evidence judgment
8.6 perplexity	—	Not assessed
8.38 perplexity	—	Not assessed
8.32 perplexity	—	Not assessed
8.29 perplexity	—	Not assessed
8.29 perplexity	—	Not assessed
8.28 perplexity	—	Not assessed
8.26 perplexity	—	Not assessed
8.27 perplexity	—	Not assessed
8.28 perplexity	—	Not assessed
8.3 perplexity	—	Not assessed
8.34 perplexity	—	Not assessed
8.5 perplexity	—	Not assessed

Reported and observed measurements

Assessments (0)

Runs (0)

Measurement results