Speculative decoding[1] strikes a simple bargain. A small drafter guesses a block of tokens, the target checks all of them in one forward pass, and every accepted guess is a token the target never had to produce on its own. The bargain pays exactly as long as guessing stays cheap.
On long contexts, guessing quietly stops being cheap. Today’s strongest drafters read the entire prefix every round. At 32K tokens, DSpark[2] spends 6.3 ms drafting each block; at 128K it spends 13.8 ms, while its drafter state swells from 640 MiB to 2.5 GiB per request. The drafter inherits the very O(t) scaling it was meant to hide.
LongSpark removes that scaling entirely. It reads the prefix through three fixed-size views of the target’s own verification pass, so its drafting time sits near 3.5 ms from 1K to 128K tokens and its context state stays at 6.3 MiB, a 406× reduction over DSpark at 128K. End to end, it delivers 1.88×–2.13× throughput speedups on Qwen3-4B, 8B, and 14B, and the lowest time-per-output-token on every long-context workload we tested.
Drafter state: 406× smallerDrafting: 3.8× faster
Fig. 1Drag the prefix length. DSpark’s drafting time (proposal plus sampling) and state grow with the prefix; LongSpark’s stay flat. Measured on Qwen3-8B, target on four GPUs (TP4).
A standalone language model has to represent its prefix faithfully, because it alone answers for every token it emits. Anything it forgets becomes an error in the output.
A drafter signs a gentler contract. The target verifies every proposal before a token is committed, and rejection sampling repairs any mistake exactly[1]. A wrong guess costs one extra round and never a wrong answer. So the drafter can afford a compressed, lossy view of the prefix, as long as that view keeps its guesses good enough to be accepted.
That turns drafter design into a budgeting problem. With mean drafting and verification latencies Tdraft and Tverify, and τ tokens gained per round, the time per token is
L = Tdraft + Tverifyτ(1)
Recent drafters have pushed hard on τ. LongSpark goes after Tdraft and fixes it to a constant. We call this principle fixed-cost drafting: a drafter keeps just enough context to propose well, at a cost that no longer depends on the prefix length.
Every round has the same shape. The target verifies the proposed block in one forward pass, and that same pass refreshes three fixed-size views of the prefix. The drafter reads them, drafts the next block in one parallel pass, and hands it back. The only full-prefix operation in the loop is the verification the target runs anyway.
Fig. 2One LongSpark round. The target’s KV cache grows every round; the boundary state, the recent window and the global summary keep their size. Click a stage to jump to it.
LongSpark taps a few evenly spaced target layers and reads the prefix at three time scales:
Boundary state. The target’s hidden states at the decoding boundary, fused into one conditioning vector ct. It anchors where generation is right now.
Recent KV window. The last W KV rows of each tapped layer, read straight out of the target cache. It supplies token-level detail about the neighborhood being continued.
Global context summary. The whole prefix compressed into a fixed number of entries per layer and head. It keeps the drafter oriented toward everything beyond the window.
Each drafter layer is paired with one tapped target layer, and it attends over a single concatenated context:
K = [ Kb ; Kw ; G ], V = [ Vb ; Vw ; Mt ](2)
The drafter’s attention context therefore has the same size at 1K tokens and at 128K.
The summary has to compress an ever-growing prefix into a fixed number of entries, and keep doing so cheaply as tokens are committed. LongSpark gives each tapped layer and head a bank of R learned queries G ∈ ℝR×d and lets them attend over the target’s keys and values:
Mt = Attn( G, K1:t, V1:t )(3)
The queries are trained with the drafter and frozen at inference, so past attention contributions never go stale. Each query stores one value vector and one log-normalizer; after verification it merges in only the rows just committed. The same queries then act as the drafter’s summary keys, reading back what they wrote.
λ′ = log( eλ + eλΔ ), M′ = eλ−λ′M + eλΔ−λ′MΔ(4)
The update never looks at the old prefix, and it gives exactly what recomputing attention from scratch would.
The backbone sees every slot of the block at once, so the backbone and the vocabulary projection run in a single parallel pass. Tokens sampled independently from that pass can disagree with one another, so LongSpark adds a DSpark-style low-rank correction that lets each position see the token sampled just before it:
Only this correction and the sampling step run sequentially, and both are tiny. These corrected distributions are exactly what the target verifies against, so the output distribution stays the target’s own. Training freezes the whole target and optimizes the drafter, the fusion and input maps, the summary queries, and the correction jointly on target-regenerated responses from Open-PerfectBlend.
Break one round down and target verification dominates, at nearly the same cost whatever the drafter. What separates the drafters is everything around it, and there LongSpark is cheapest: its proposal takes 2.03 ms against DSpark’s 4.59 ms, and a full round 31.0 ms against 34.5 ms. A long answer takes thousands of rounds.
VerifyProposalSamplingUpdate
Fig. 3One decoding round at 32K context on Qwen3-8B, target on a single GPU (TP1). Verification costs about the same for every drafter; LongSpark’s proposal is 2.3× faster than DSpark’s.
Across eight benchmarks in math, code, and chat, LongSpark posts the highest average throughput at every target size, and the lead grows with the target: 1.88× on Qwen3-4B, 1.99× on 8B, 2.13× on 14B. DSpark accepts slightly longer blocks at every scale. LongSpark still wins, because it pays less for each one.
Show all eight benchmarks
Fig. 4Left: mean accepted length τ. Right: throughput speedup over autoregressive decoding (dashed line = 1×). DSpark proposes more accepted tokens per round; LongSpark turns fewer of them into more throughput.
Serving systems rarely run at concurrency 1. As we raise concurrency from 8 to 128 on Alpaca, MBPP, and MATH-500, the drafter’s work multiplies with the batch, and a cheaper drafter matters more. LongSpark starts level with DSpark at C8, overtakes it by C16, and pulls ahead steadily at every scale. By C128 it leads by +12.0% on 4B, +7.8% on 8B, and +6.0% on 14B.
Fig. 5LongSpark’s throughput gain over DSpark as concurrency grows. Hover a point for per-benchmark gains.
From 32K to 128K tokens, LongSpark delivers the highest throughput and the lowest time-per-output-token on all three long-context workloads. On CodeSpan at 64K, it beats DSpark, the strongest competitor, by 29.1% in throughput and cuts TPOT by 23.1%. At 128K on LongSWE-Bench, where every other drafter sits within a few milliseconds of plain decoding, LongSpark brings TPOT from 97.2 ms down to 74.3 ms.
Table 1Long-context decoding on Qwen3-8B. TPS in tokens/s (↑), TPOT in ms/token (↓).
Take away any one view, retrain, and acceptance drops in every domain. The recent window carries the most: without it, τ falls from 4.69 to 3.71. The summary and the boundary state each still add 0.3 to 0.4 tokens per round.
Table 2Mean accepted length τ on Qwen3-8B with one view removed.
A drafter and a target have different jobs. The target needs the full, exact state of the prefix to guarantee correctness. The drafter needs only enough orientation to guess well. LongSpark builds on that asymmetry: a boundary state for position, a recent window for detail, a global summary for everything else, all fixed in size and all read from work the target has already done.
The result is a drafter whose cost is a flat fee per round. At short contexts, that fee already buys the best throughput we measured. At long contexts, where every other drafter’s fee keeps rising, the gap only grows.
Code and weights for Qwen3-4B, 8B, and 14B are on GitHub and on Hugging Face. We would love to hear what you build with them.
@misc{he2026longspark,
title = {LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter},
author = {He, Hao-Yuan and Liu, Peng-Fei and Shen, Si and Li, Ming},
year = {2026},
eprint = {2609.37029},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2609.37029}
}
All speculative methods run in a target–drafter disaggregated setup: one dedicated drafter GPU is added to the target GPUs used by autoregressive decoding. Seven draft tokens per round, thinking mode disabled.
Baselines: EAGLE-3[3] (autoregressive), DFlash[4] and DSpark[2] (parallel block drafters), each trained for the matching Qwen3 target. Long-context runs use fixed NTK scaling[5] with α = 4.
The drafter’s persistent state is the global summary, O(|𝓛| H R d). The recent window is read in place from the target cache, and block KV is discarded after every round.
References
Y. Leviathan, M. Kalman, Y. Matias. Fast inference from transformers via speculative decoding. ICML 2023.