Skip to main content

emit_moe_forward

Function emit_moe_forward 

Source
pub fn emit_moe_forward(x: &[f64], w: &MoeWeights) -> String
Expand description

Emit a self-contained CUDA program for ONE Mixture-of-Experts FFN block over real int8 weights: router GEMV → top-K route (softmax, keep K, renormalize) → each expert’s SwiGLU with an EXPERT-INDEXED weight base → weighted combine. The weights stream from files (int8 quants + f32 block scales), never a 4×-larger f32 buffer, so the 30B block (~30 GB int8) fits a 48 GB card. Dequant-in-kernel GEMV (f32 activation × int8 weight, f64 accumulate) matches the LOGOS moe op on the VM exactly; the whole block is captured as one CUDA graph. x is the block input (d floats, inlined). The experts are UNROLLED in the host launch list (not the kernels), so E scales without changing a kernel — the same emitter serves 8B-scale and 30B-scale.