KITE: KV-Invariant Transformer Expansion for Efficient Agentic LLM Scaling

At the same final checkpoints used for the matched-compute comparison, 67B SST scores higher than both classic baselines on all seven reported downstream tasks and has the lowest held-out ARXIV token NLL: 1.4421 versus 1.4424 for 63B classic and 1.4521 for 47B classic.

At final checkpoints with approximately matched theoretical cumulative training FLOPs, 67B SST has the lowest reported EMA-200 training LM loss: 1.5900 versus 1.6006 for 47B classic and 1.5921 for 63B classic, improvements of 0.0106 and 0.0021 respectively.

For each of six selected OpenRouter models, total input tokens and uncached input tokens exceed output tokens; estimated uncached-input charges account for 67.4% to 76.7% of uncached-input-plus-output charges.

SST trades lower bulk-prefill active-body count for higher decode active-body count relative to 47B classic. Under the paper's common analytical proxy, SST becomes cheaper than 47B once prefill weight exceeds 65.5% and cheaper than 63B above 13.4%; at a 75:25 prefill:decode mix it is estimated 6.7% below 47B and 31.6% below 63B.

The matched-compute result is based on an effective-architecture theoretical FLOP estimator whose reported forward coefficients differ by model and whose training estimate multiplies forward computation by three to approximate forward plus backward work.

The evaluated SST route expands a 33.819B-parameter, 18-layer source into a 66.959B two-tower model by copying source layers into an 18-layer Prefiller and an 18-layer Decoder; the Prefiller produces aligned layer-wise KV, the Decoder reads it, and both towers are jointly trained after conversion.

SST's training loss rises sharply at source-to-SST conversion and then falls during joint continuation; the paper reports the direction from Figure 4 but does not assign an exact conversion-jump magnitude.

Increasing the classic baseline from 47B to 63B does not improve every task: the 63B classic score is lower on GSM8K (55.04 versus 56.56) and HumanEval few-shot (32.93 versus 35.98), although it is higher on the other reported evaluation columns.

Appendix A reports the token-count and estimated-charge operands underlying the six OpenRouter ratios and uncached-input charge shares, with model-specific standard rates and half-price Batch rates.

The OpenRouter sample is selected, has incomplete and unequal day coverage, and is not separately labeled as agentic traffic; its charge estimates use a fixed price snapshot and omit several pricing effects, so it motivates rather than measures an agentic GPU-cost distribution.

The compared models use matched sparse-expert and hybrid-attention patterns but different widths, heads, and body sizes; in SST only the Prefiller contains K/V projections and K normalization, while both towers have independent Transformer-block parameters.

The evaluated SST checkpoint includes 167.98B source-stage tokens followed by 222.55B continuation tokens, for 390.54B cumulative tokens; continuation jointly trains both towers in BF16 and retains optimizer state for Prefiller/shared parameters while initializing fresh Decoder state.