Skip to main content

compile_to_cuda_forward_q8_moe_exact

Function compile_to_cuda_forward_q8_moe_exact 

Source
pub fn compile_to_cuda_forward_q8_moe_exact(
    source: &str,
    weights: &HashMap<String, (String, usize)>,
    q8: &HashMap<String, Q8Ref>,
    moe: &HashMap<String, MoeWeights>,
) -> Result<String, String>
Expand description

compile_to_cuda_forward_q8_moe_dp4a’s LOGIT-EXACT sibling: the same sparse top-K deeply-fused kernel shape, but the activation is quantized TWO-PASS (hi byte + lo residual byte ⇒ ~14-bit) and every block runs two __dp4a (hi and lo) — ~2× the activation math for logits within ~1e-3 of the f64 reference, so a near-tie greedy token does NOT flip (single int8 activation flips it — llama.cpp included). Weights are still read once, so the memory-bound cost is unchanged; the measured price of exactness is ~6% tok/s.