Loading page…
The paper introduces LoGo, a decoder-only Transformer attention mechanism that gives every token a local causal-attention branch and selectively activates a full-context branch through a learned token-level gate. An adaptive threshold controls the global activation ratio without an auxiliary balancing loss, progressive masking stabilizes training, and query-sparse Triton kernels avoid computing global attention for unselected queries. Experiments compare LoGo with full-attention and static local-global hybrids across model scales, long-context extension stages, recall benchmarks, kernel runtimes, ablations, and routing analyses.
If its reported conditions can be reproduced, LoGo could lower the cost of long-context language modeling by reserving full-context attention for tokens that appear to need distant information rather than applying it uniformly. This may help model builders serving long documents or retrieval-heavy prompts, while preserving ordinary local processing for predictable tokens. The value is conditional on the learned router, sparse kernels, training recipe, checkpoints, and evaluation data matching the paper; the fixed snapshot supplies none of the author implementation or in-house pretraining assets, so the claims remain unverified here.
The paper’s claims are available in Research claims.