On RULER-4k retrieval, attention-KL reordering outperformed the MSE ordering at 0.5 and 1 bpd for every 7–8B model, with its largest objective separation among the paper's benchmarks; at 2 bpd it came within 0.7–3.6 points of FP16.

bpd: 0.5 · side: K-only · model: LLaMA-3.1-8B · objective: MSE · contextLength: 4096

bpd: 0.5 · side: K-only · model: LLaMA-3.1-8B · objective: KL · contextLength: 4096

bpd: 1 · side: K-only · model: LLaMA-3.1-8B · objective: MSE · contextLength: 4096

bpd: 1 · side: K-only · model: LLaMA-3.1-8B · objective: KL · contextLength: 4096

bpd: 2 · side: K-only · model: LLaMA-3.1-8B · objective: KL · contextLength: 4096

model: LLaMA-3.1-8B · configuration: FP16 · contextLength: 4096

bpd: 0.5 · side: K-only · model: Mistral-7B-v0.3 · objective: MSE · contextLength: 4096

bpd: 0.5 · side: K-only · model: Mistral-7B-v0.3 · objective: KL · contextLength: 4096

bpd: 1 · side: K-only · model: Mistral-7B-v0.3 · objective: MSE · contextLength: 4096

bpd: 1 · side: K-only · model: Mistral-7B-v0.3 · objective: KL · contextLength: 4096

bpd: 2 · side: K-only · model: Mistral-7B-v0.3 · objective: KL · contextLength: 4096

model: Mistral-7B-v0.3 · configuration: FP16 · contextLength: 4096