Skip to main content

emit_qwen38_mtp_probe

Function emit_qwen38_mtp_probe 

Source
pub fn emit_qwen38_mtp_probe(
    cfg: &Qwen38Cfg,
    wdir: &str,
    prompt: &[usize],
    golden: &[usize],
) -> String
Expand description

Emit the MTP ACCEPT-RATE PROBE.

Multi-token prediction only pays if the draft head is usually right: the speedup is (1+alpha) * full / (full + draft), and with the draft costing 7% of a forward that is 1.24x at alpha=0.33 and 1.59x at alpha=0.70. The expensive half of MTP is the two-token VERIFY step — GEMVs batched over two positions while the recurrence still advances sequentially — so it is worth measuring alpha BEFORE building it, not after.

This probe does exactly that and nothing more. It walks the known golden continuation, and at each step runs the draft head on (last hidden state, token just produced) and asks whether it predicted the NEXT golden token. No verification path, no speculation machinery — just the accept rate, which is the number that decides whether the rest is worth writing.

The draft must not disturb the decode it speculates on: it has its own KV cache and uses an argmax that does not advance the position.