The paper reports that its query-sparse Triton kernel reaches a 1.99x forward-plus-backward speedup over the dense Triton baseline at 64k sequence length and a 0.5 attention budget.

direction: higher is faster · batch_size: 1 · comparator: dense FlashAttention Triton baseline · head_dimension: 128 · implementation: LoGo query-sparse Triton kernel · attention_heads: 8 · sequence_length: 65536 · attention_budget: 0.5