The Bennett shape is accurate near 3–4 bits but increasingly underestimates deployed-quantizer distortion at low bits; replacing it with measured per-head distortion makes exact DP improve over the shipped allocator on all three models at 1 bpd.

Reported and observed measurements

Assessments (0)

No immutable Assessment has been published for this Claim yet.

Runs (0)

No execution has been linked to this Claim yet.

Each row is a measurement with its reported value, latest available observation, and cumulative evidence state.
Metric and conditions	Reported	Latest observation	Evidence judgment
models: all three 7–8B models · bitWidth: 4 · aggregation: median across heads	1 ratio	—	Not assessed
models: all three 7–8B models · bitWidth: 4	31%	—	Not assessed
models: all three 7–8B models · bitWidth: 3 · aggregation: median across heads	1.1 ratio	—	Not assessed
models: all three 7–8B models · bitWidth: 3	33%	—	Not assessed
models: all three 7–8B models · bitWidth: 2 · aggregation: median across heads	1.7 ratio	—	Not assessed
models: all three 7–8B models · bitWidth: 2	41%	—	Not assessed
models: all three 7–8B models · bitWidth: 1 · aggregation: median across heads	6.9 ratio	—	Not assessed
models: all three 7–8B models · bitWidth: 1	36%	—	Not assessed
bpd: 16 · side: K-only · model: LLaMA-3.1-8B · configuration: FP16	5.71 perplexity	—	Not assessed
bpd: 16 · side: K-only · model: Mistral-7B-v0.3 · configuration: FP16	4.77 perplexity	—	Not assessed
bpd: 16 · side: K-only · model: Qwen2.5-7B-Instruct · configuration: FP16	6.07 perplexity	—	Not assessed
bpd: 1 · side: K-only · model: LLaMA-3.1-8B · configuration: water-filling + Bennett estimate (shipped)	16.33 perplexity	—	Not assessed

Metric and conditions

Reported

Latest observation

Evidence judgment

models: all three 7–8B models · bitWidth: 4 · aggregation: median across heads

1 ratio

—

Not assessed

models: all three 7–8B models · bitWidth: 4

31%

—

Not assessed

models: all three 7–8B models · bitWidth: 3 · aggregation: median across heads

1.1 ratio

—

Not assessed

models: all three 7–8B models · bitWidth: 3

33%

—

Not assessed

models: all three 7–8B models · bitWidth: 2 · aggregation: median across heads

1.7 ratio

—

Not assessed

models: all three 7–8B models · bitWidth: 2

41%

—

Not assessed

models: all three 7–8B models · bitWidth: 1 · aggregation: median across heads

6.9 ratio

—

Not assessed

models: all three 7–8B models · bitWidth: 1

36%

—

Not assessed

bpd: 16 · side: K-only · model: LLaMA-3.1-8B · configuration: FP16

5.71 perplexity

—

Not assessed

bpd: 16 · side: K-only · model: Mistral-7B-v0.3 · configuration: FP16

4.77 perplexity

—

Not assessed

bpd: 16 · side: K-only · model: Qwen2.5-7B-Instruct · configuration: FP16

6.07 perplexity

—

Not assessed

bpd: 1 · side: K-only · model: LLaMA-3.1-8B · configuration: water-filling + Bennett estimate (shipped)

16.33 perplexity

—

Not assessed

Reported and observed measurements

Assessments (0)

Runs (0)

Measurement results