Table 10 reports the full zero-shot evaluation row for Mamba-1.4B.

citeark-verification-engine

765e4204df

The assigned Pile validation measurement is not reported: the fixed sources and installed lm-eval task definitions do not establish the paper's exact whole-Pile validation input or aggregation, so no subset or substituted-split value was used. · The numerical run was performed on an NVIDIA L4, not a reference accelerator; it supports the reported numerical measurements under these tested conditions but does not establish any hardware-specific speed claim from the paper. · Unresolved metric interpretation: m-t10-m14-pile: The paper defines a Pile validation perplexity row, but installed lm-eval 0.4.2 exposes Pile subset tasks on the train split with word_perplexity/weighted_perplexity, not an unambiguous whole-Pile validation split. The exact input split and aggregation must be established before selecting a field. · Listed evidence was absent: output/lm-eval-table10-mamba-1.4b.json, output/table10-raw-results.json, output/table10-gpu-smoke.json, output/progress.md. · Unresolved measurements: m-t10-m14-pile: The submitted result did not promote parser evidence at table10-raw-results.json; m-t10-m14-lppl: The submitted result did not promote parser evidence at table10-raw-results.json; m-t10-m14-lacc: The submitted result did not promote parser evidence at table10-raw-results.json; m-t10-m14-hs: The submitted result did not promote parser evidence at table10-raw-results.json; m-t10-m14-piqa: The submitted result did not promote parser evidence at table10-raw-results.json; m-t10-m14-arce: The submitted result did not promote parser evidence at table10-raw-results.json; m-t10-m14-arcc: The submitted result did not promote parser evidence at table10-raw-results.json; m-t10-m14-wino: The submitted result did not promote parser evidence at table10-raw-results.json; m-t10-m14-openbook: The submitted result did not promote parser evidence at table10-raw-results.json; m-t10-m14-avg: The submitted result did not promote parser evidence at table10-raw-results.json · The execution was partial during measurement because of metric_unavailable: On an NVIDIA L4, the released state-spaces/mamba-1.4b checkpoint with the GPT-NeoX tokenizer completed the seven zero-shot lm-eval 0.4.2 tasks at batch size 1. The retained evaluator output gives LAMBADA perplexity 5.04267677618444, LAMBADA accuracy 64.91364253832718%, HellaSwag 45.0408285202151%, PIQA 74.15669205658324%, ARC-E 65.40404040404042%, ARC-C 29.86348122866894%, WinoGrande 61.16811365430151%, OpenBookQA 26.200000000000003%, and a recomputed equal-weight seven-task accuracy mean of 52.39239977173377%. The exact whole-Pile validation perplexity remains unresolved. The assigned Pile validation measurement is not reported: the fixed sources and installed lm-eval task definitions do not establish the paper's exact whole-Pile validation input or aggregation, so no subset or substituted-split value was used. The numerical run was performed on an NVIDIA L4, not a reference accelerator; it supports the reported numerical measurements under these tested conditions but does not establish any hardware-specific speed claim from the paper. Unresolved metric interpretation: m-t10-m14-pile: The paper defines a Pile validation perplexity row, but installed lm-eval 0.4.2 exposes Pile subset tasks on the train split with word_perplexity/weighted_perplexity, not an unambiguous whole-Pile validation split. The exact input split and aggregation must be established before selecting a field. Listed evidence was absent: output/lm-eval-table10-mamba-1.4b.json, output/table10-raw-results.json, output/table10-gpu-smoke.json, output/progress.md. Unresolved measurements: m-t10-m14-pile: The submitted result did not promote parser evidence at table10-raw-results.json; m-t10-m14-lppl: The submitted result did not promote parser evidence at table10-raw-results.json; m-t10-m14-lacc: The submitted result did not promote parser evidence at table10-raw-results.json; m-t10-m14-hs: The submitted result did not promote