For causal attention, skipping blocks wholly above the causal diagonal is reported to make the kernel around 1.7–1.8× faster than attention without a causal mask; only one square block per row needs elementwise causal masking.

citeark-verification-engine

c640198839

The actual device was an NVIDIA L4, not the paper's A100 80GB SXM4; these timings cannot establish the paper's A100-specific values or ranges. · Only non-causal head-dimension-64 configurations at sequence lengths 512 and 8192 were probed with five repetitions; the 120-cell 30-repeat full workload was not run. · The exact historical triton.ops.flash_attention.attention callable is absent from installed Triton 3.4.0 and was not replaced with a different implementation. · Unresolved metric interpretation: finding-002-backward-triton: The paper defines the operand as matched backward runtime speedup and describes it only as 'around 2x'; it does not state whether the single summary is a mean, median, extremum, or representative cell. The evaluator will retain all paired ratios and expose a geometric mean as a clearly labeled descriptive statistic, but that statistic is not uniquely identified by the paper. · Unresolved measurements: finding-001-lower: The submitted result did not promote parser evidence at evidence/a100/attention-results.json; finding-001-upper: The submitted result did not promote parser evidence at evidence/a100/attention-results.json; finding-002-fa-low: The submitted result did not promote parser evidence at evidence/a100/attention-results.json; finding-002-fa-high: The submitted result did not promote parser evidence at evidence/a100/attention-results.json; finding-002-triton-low: The submitted result did not promote parser evidence at evidence/a100/attention-results.json; finding-002-triton-high: The submitted result did not promote parser evidence at evidence/a100/attention-results.json; finding-002-pytorch-low: The submitted result did not promote parser evidence at evidence/a100/attention-results.json; finding-002-pytorch-high: The submitted result did not promote parser evidence at evidence/a100/attention-results.json; finding-002-forward-triton-low: The submitted result did not promote parser evidence at evidence/a100/attention-results.json; finding-002-forward-triton-high: The submitted result did not promote parser evidence at evidence/a100/attention-results.json; finding-002-backward-triton: The submitted result did not promote parser evidence at evidence/a100/attention-results.json; finding-003-peak: The submitted result did not promote parser evidence at evidence/a100/attention-results.json; finding-003-forward-efficiency: The submitted result did not promote parser evidence at evidence/a100/attention-results.json; finding-003-backward-efficiency: The submitted result did not promote parser evidence at evidence/a100/attention-results.json; fig4a-512-pytorch: The submitted result did not promote parser evidence at evidence/a100/attention-results.json; fig4a-512-flashattention: The submitted result did not promote parser evidence at evidence/a100/attention-results.json; fig4a-512-xformers: The submitted result did not promote parser eviden · The execution was partial during measurement because of metric_unavailable: A real two-configuration NVIDIA L4 probe ran with the exact fixed FlashAttention-2 source, two warmups, and five repetitions. FlashAttention-2 passed output/gradient checks and measured 49.927/30.172/34.018 TFLOPs/s forward/backward/combined at sequence 512 and 57.543/57.982/57.856 at sequence 8192. Both xFormers baselines ran; the historical Triton callable was unavailable, and PyTorch OOMed at 8192. The cumulative runner budget did not leave enough time for the full grid. The actual device was an NVIDIA L4, not the paper's A100 80GB SXM4; these timings cannot establish the paper's A100-specific values or ranges. Only non-causal head-dimension-64 configurations at sequence lengths 512 and 8192 were probed with five repetitions; the 120-cell 30-repeat full workload was not run. The exact historical triton.ops.flash_attention.attention callable is absent from installed Triton 3.4.0 and was not replaced with a different implementation. Unresolved metric interpretation: finding-002-backward-triton: The paper defines the operand as matched backward runtime speedup and describes it only as 'around 2x'; it does not state whether the single summary is a mean, median, extremum, or representative cell. The evaluator will retain all paired ratios and expose a geometric mean as a clearly labeled descriptive statistic, but that statistic is not uniquely identified by the paper. Unresolved measurements: finding-001-lower: The submitted result did not promote parser evidence at evidence/a100/attention-results.json; finding-001-upper: The submitted result did not promote parser evidence at evidence/a100/attention-results.json; finding-002-fa-low: The submitted result did not promote parser evidence at evidence/a100/attention-results.json; finding-002-fa-high: The submitted result did not promote parser evidence at evidence/a100/attention-results.json; finding-002-triton-low: The submitted result did not pr