Quantization already won. So why speculate?

I, like everyone else, have become increasingly demanding with inference. I want to run powerful models, and I want them to be snappy. My performance-greed has a specific cause: there's an expanding number of software packages doing their best to bury me in maintenance, and my current-best tool against them (repo-review) is slow on my local hardware, a Ryzen AI Max+ 395 (Radeon 8060S iGPU).

My model of choice for defense against these repos is Qwen3.6-35B-A3B, a well-regarded mixture-of-experts (MoE) that performs well for its size: 35B parameters in total, but only about 3B fire on any given token (the “A3B”). But it's a bit sluggish on my computer, slowing how many reviews I do. The most obvious optimizations are quantization and batching, neither of which solves my problem. Quantization helps my bandwidth-limited machine by reducing the size of the weights, but this comes at the cost of weakening the model. A review by a lobotomized agent is arguably worse than no review at all since it could give a false sense of security. Batching avoids these quality concerns and is seemingly well suited to the task (run multiple reviews at once... duh), but the reviews are actually fairly memory intensive (they involve installs, profiling code, writing/running tests, etc.), so batching is out.

This leaves speculative decoding: a small draft model proposes tokens for the big one to check in bulk. Fortunately, it dodges both issues from batching and quantization: it speeds up a single inference thread and is lossless. For the drafter I use DFlash, specifically the checkpoint z-lab released for Qwen3.6-35B-A3B: it's reported to perform well, it's small, and it drafts the whole block in parallel. It's a 6-layer, 2048-wide diffusion-style network conditioned on hidden states from 8 of the target's 40 layers, where the paper's default is 5 draft layers reading 5 target layers. Before changing anything, a baseline: I measured the model in my quickest quantized variant, and unquantized both with and without DFlash. The idea will be to merge these fronts, but one thing at a time:

Both bf16 rows use the off-the-shelf fused-MoE kernel, so the 2.1× is speculation alone.

A quantized llama.cpp build already wins on speed: 47.8 tok/s with no speculation, well past the 17.4 the full speculative setup reaches. But that is the lossy lever, an approximation of the weights. Speculation is the lossless one: in the same engine, weights, and precision, the released drafter turns 8.1 into 17.4 tok/s, a 2.1× speedup that reproduced the target's tokens exactly on every prompt I checked (greedy decode). That needs context, though: 8.1 tok/s comes from an unoptimized bf16 research stack, so it's a speedup over a slow baseline, and speculative decoding helps most when a decode step leaves the hardware underused. The faster quantized engine uses the hardware more fully, so I'd expect a smaller multiple there. DFlash is in llama.cpp mainline now, so I can go measure it. So what I want is both at once: lossless speculation inside a fast engine.

Spoilers: what I found

• A 16-token block verify costs only ~1.6× a single decode step: the block wakes ~47 of 256 experts, but attention, embeddings, and head are read once for the whole block.
• The draft is cheap next to the target (T_d/T_t ≈ 0.12), so within the current loop, acceptance length τ is the lever. The bigger win is probably changing the loop: syncing less often.
• Across two seeds (early, not a ranking): KL with mixed attention led consistently, the attention pattern barely moved CE, and the uniform-corruption arm trailed.
• 32k draft vocabulary is a coverage/memory call that barely touches speed.
• The target mixes full and linear attention, so keeping speculation lossless took a patch to the transformers cache path (below) 🫠.

The rest is how I got there. First, what actually sets the speedup?

The ratio behind it

The drafter predicts γ = 15 tokens per block (16 positions, counting the current token). The speedup η grows linearly in τ, the mean number of tokens accepted per cycle (the accepted draft prefix plus the target's one bonus token):

Here, T_t is the time in seconds of one target (verify) pass and T_d one draft pass. This formula assumes you follow the paper's recipe: the verification pass runs in parallel over the draft, checking whether each drafted token matches what the model would actually output (this is why it's lossless!), and that same pass primes the hidden states for the next draft, so the target model runs only once per cycle (not twice). These times are measured below on my hardware:

The draft is cheap compared to the target (T_d/T_t ≈ 0.12), so I can hope for at most a ~4.4× speedup (using τ ≈ 4.9). The catch is that a 16-token block verify costs about 1.6× a single decode step (205 vs 125 ms), because the target checks all 16 positions in one parallel pass but each routes independently, so the block wakes ~47 of 256 experts instead of 8 for a single token (those 8, plus an always-on shared expert, are the ~3B active). It's only 1.6×, not 6×, because the attention, embeddings, and output head are read once for the whole block; only the expert share grows, which a dense model wouldn't have. Put the real per-token baseline back in, τ·L_target/(T_d+T_t) = 4.9·125/230, and the ceiling is about 2.7×; the board's 17.4/8.1 is 2.1×, about 80% of the ceiling. The rest is per-cycle overhead I haven't broken down.

So η gives me two real handles: push τ up, or push the draft's cost T_d down.

Resolving the differences

The first thing to sort out was whose recipe to follow. I used DFlash's paper, their repo, and Speculators all as references for learning how to do this “right,” but they differed, and some of what they agreed on came with no reason attached. For instance, the paper trains with a position-weighted cross-entropy (Eq. 4: weight w_k = exp(−(k−1)/λ) at block position k, so early tokens count most), while Speculators defaults to KL divergence, decayed the same way (reasonable, since the draft is basically a distillation of the target). (I write λ for the paper's decay rate, which they call γ, to keep γ free for the block's draft count.) The paper and its code, meanwhile, both attend bidirectionally within the block (each draft position sees the others, MLM-style). Causal seemed worth a shot: it's how the target factorizes, and acceptance is a prefix anyway (first mismatch and the rest of the block is gone), so I'd rather a position lean on what came before it than on siblings that might get rejected too. And there's a diffusion angle (the D in DFlash): standard diffusion corrupts its input on a graded schedule, but DFlash masks the whole block at once, in training as much as when it drafts. I was curious whether a graded schedule would help, so I also tried a uniform one (kl_mixed_uniform). Three knobs, then: loss, attention, and corruption. All are cheap in the η sense: they raise τ without changing what the draft costs, so they're the first thing to nail down.

To test them, I built a two-box homelab, patch cable between them: one trains the draft model on a 16 GB card, the other runs the 35B-A3B MoE target and serves its hidden states. (Getting to train the thing on such a small box was half the fun. Inference work usually feels a long way from hardware you can poke at.)

So I trained several recipes over the loss and mask choices and tracked acceptance length as the training data grew:

Across both seeds, KL with mixed attention (kl_mixed) led; the two cross-entropy runs sat in the middle, almost on top of each other, so the attention pattern barely mattered for CE; and swapping kl_mixed's corruption to a uniform schedule (kl_mixed_uniform) dropped it to the bottom, under CE. That last one probes the masking choice the paper argues for but never isolates. The uniform arm losing fits that choice, since at inference the drafter only ever sees a fully-masked block, so partial masking would train it on inputs it never meets. The recipes (arms) don't isolate one factor at a time (kl_mixed changes both the loss and the attention versus the CE runs), so I'd read the loss and corruption schedule as the bigger levers here and the attention as a minor one.

Draft vocabulary: a coverage call

One way to decrease the draft's cost T_d is to shrink the vocabulary that the drafter can predict. Unlike loss and mask, draft vocabulary isn't free: it trades coverage against the head's per-step cost. The drafter predicts over the top-K most frequent tokens (which Speculators supports) instead of the full vocab the paper uses. That head is read on every decode step and grows with K, so a smaller vocab makes a cheaper step; shrink too far, though, and the drafter can't propose the rarer tokens the target wants. So I picked the smallest K that still covers most held-out tokens. One caveat: inference today still routes through the target's full head, so the per-step speed win only lands once decoding goes through the reduced head (I just haven't wired that path yet).

That per-step head cost comes down to memory bandwidth: every decode step, the draft streams the head's [2048×V] weight (V the full vocab) from memory; on a bandwidth-limited machine like mine, that read is what costs time. A bigger vocab is more bytes to stream, so latency climbs roughly linearly with vocab size.

Back through η: since T_d/T_t ≈ 0.12, even the full-vocab head is a small slice of an already-cheap draft, and τ looks prediction-limited anyway. So vocab size probably moves speedup by a couple percent at most (an estimate from the ratio; I didn't isolate it end to end). It's really a training-memory knob.

Lossless, and fast

My goal (so far...) has been to follow speculative decoding's promise in staying lossless. This took a fix: DFlash's decode loop crops the KV cache back to the accepted prefix every cycle; a full-attention layer crops cleanly, but a linear-attention layer's recurrent state (fixed-size, not a growing KV cache) got advanced through the whole block and can't be (transformers still has no rollback for it), so I patched the cache path so that, under greedy decode, the accepted tokens match what the target would produce alone.

The verify's own cost

One last variable, then I'll stop: T_t. Cutting it is a real inference win in its own right, just a semi-orthogonal one: the same speedup hits the baseline decode step too, so don't expect much movement of η. There's one cheap win here: on ROCm my experts get walked one at a time, which is silly since they're independent. That's not transformers' fault (it asks for grouped matmuls); PyTorch's ROCm grouped-GEMM only covers CDNA cards, so my gfx1151 falls back to a per-group loop. To really get inference running, I dropped in an off-the-shelf fused-MoE grouped-GEMM kernel (tokens sorted by expert, then one grouped launch per projection instead of a loop), patched into the target's expert forward. It's token-identical on every prompt I've tried, about 3× on the expert matmuls and ~1.65× on DFlash decode overall.

What this isn't

These are early results:

• The recipe comparison is two seeds: suggestive, but not enough to rank the recipes; cleaner runs are next.
• The τ ≈ 4.9 and 2.1× are the released drafter; my own arms' τ is lower so far.
• The vocabulary and latency calls lean on the η formula plus isolated component timings, not a full end-to-end sweep.
• I only spot-checked losslessness (greedy decode, my own prompts): the fused-MoE kernel changes the arithmetic order, so decode matches that kernel's target, and I haven't verified bit-exactness against a reference.

Where it's going

After all that drafter tuning, the real win probably isn't a better draft. It's syncing less often. Today's decoders keep the draft on a short leash, rechecking every few tokens to stay lossless; I'd rather let it run long, using confidence-guided refinement to fix its mistakes, and spend losslessness for far fewer syncs. I mean that's likely how anyone reading this really uses AI: have a model build scaffolding, then redirect and iterate on the weak bits. The target pass dominates each cycle, so the sync savings should dwarf the lossless win I give up. It costs quality in a different way than quantization, though: quantization approximates the weights, while this commits some tokens the target wouldn't have. At least that's the bet.

Testing that fairly needs a production-grade engine, not a research stack. The board up top makes the point: plain quantized llama.cpp/Vulkan already beats my PyTorch/ROCm DFlash setup, so a speedup that only shows up in the research loop doesn't count for much yet. Getting my arm in there is its own project: the DFlash path has no draft-side head or vocab map at all (EAGLE3 has both), so a 32k head is a non-starter, and Vulkan still gets my target wrong on some settings. The MoE penalty I'd chase anyway: it shows up on CUDA too, worst at long drafts, so I read it as drafter overhead against a cheap decode rather than a backend gap. The hard part is already done, though: for Qwen3.5/3.6 targets llama.cpp keeps a full copy of the recurrent state per draft position, so a rejection rewinds by flipping an index instead of the recompute I needed above. Then I can run that code-review tool locally as much as I want, and so can anyone else running models at home.