FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

The paper used an H100 80GB SXM5; L4 measurements cannot verify the reported H100 335/338 TFLOPs/s values or hardware utilization.

The contract-fixed FlashAttention-2 source identifies as package 2.8.4 and postdates the paper. Its assigned CUDA kernels were unchanged, but an evaluation-only host dispatch was bounded to FP16 head dimensions 64/128 and rejected unassigned SplitKV paths.

The paper does not identify the original FlashAttention package version. The retained official v1.0.9 is only a source-grounded candidate and cannot execute d=128 backward on SM89, leaving 12 assigned comparison values unavailable.

PyTorch d=64 8k non-causal and causal backward exceeded the L4 memory capacity, leaving two assigned values unavailable; isolated fresh-process retries confirmed this was not allocator fragmentation.

The four PyTorch 16k figure positions have no numeric measurement IDs in the fixed contract and were retained as not-assigned rather than benchmarked.

Unresolved measurements: fig7a-8k-pytorch: The preserved parser evidence did not contain a trustworthy metric: JSON 指针没有指向有限数值; fig7b-512-flashattention: The preserved parser evidence did not contain a trustworthy metric: JSON 指针没有指向有限数值; fig7b-1k-flashattention: The preserved parser evidence did not contain a trustworthy metric: JSON 指针没有指向有限数值; fig7b-2k-flashattention: The preserved parser evidence did not contain a trustworthy metric: JSON 指针没有指向有限数值; fig7b-4k-flashattention: The preserved parser evidence did not contain a trustworthy metric: JSON 指针没有指向有限数值; fig7b-8k-flashattention: The preserved parser evidence did not contain a trustworthy metric: JSON 指针没有指向有限数值; fig7b-16k-flashattention: The preserved parser evidence did not contain a trustworthy metric: JSON 指针没有指向有限数值; fig7c-8k-pytorch: The preserved parser evidence did not contain a trustworthy metric: JSON 指针没有指向有限数值; fig7d-512-flashattention: The preserved parser evidence did not contain a trustworthy metric: JSON 指针没有指向有限数值; fig7d-1k-flashattention: The preserved parser evidence did not contain a trustworthy metric: JSON 指针没有指向有限数值; fig7d-2k-flashattention: The preserved parser evidence did not contain a trustworthy metric: JSON 指针没有指向有限数值; fig7d-4k-flashattention: The preserved parser evidence did not contain a trustworthy metric: JSON 指针没有指向有限数值; fig7d-8k-flashattention: The preserved parser evidence did not contain a trustworthy metric: JSON 指针没有指向有限数值; fig7d-16k-flashattention: The preserved parser evidence did not contain a trustworthy metric: JSON 指针没有指向有限数值

Execution used a roughly 23GB NVIDIA L4 instead of an H100 80GB SXM5, materially affecting throughput and memory behavior.

PyTorch at 16k—the claim's central OOM observation—was not run; it was skipped based on the paper label.

The listed measurements omit a numeric PyTorch 8k result because that L4 attempt OOMed; this does not establish H100 behavior.

The paper does not identify the exact FlashAttention baseline version; v1.0.9 was only a candidate.

The evaluated FlashAttention-2 source identifies as later package 2.8.4 and used a bounded evaluation-only host dispatch.

The inherited 16k-total-token and hidden-dimension-2048 grid remains an assumption because the H100 paragraph does not independently restate it.

The captured run freshly measured the listed Figure 7a cells with 30 forward and 30 backward CUDA-event samples, but on an NVIDIA L4 rather than the claimed H100 80GB SXM5. Since throughput is device-dependent, these absolute values are not comparable to Figure 7a. The central qualitative claim that PyTorch is OOM at 16k was not executed: the evaluator skipped PyTorch 16k and labeled it not_assigned based on the paper. The run therefore establishes only L4 portability behavior, not the H100 throughput or OOM claim. Some original claim measurements remain without evidence; available measurements are assessed individually.