Skip to main content

compile_to_cuda_forward_q8

Function compile_to_cuda_forward_q8 

Source
pub fn compile_to_cuda_forward_q8(
    source: &str,
    weights: &HashMap<String, (String, usize)>,
    q8: &HashMap<String, Q8Ref>,
) -> Result<String, String>
Expand description

compile_to_cuda_forward_weighted with an INT8 (Q8_0) weight manifest: any tensor variable named in q8 is loaded resident as int8 quants + f32 block scales, and a matmul against it becomes a dequant-in-kernel GEMV. This is what fits real 8B weights on a 12 GB card (8 GB int8, not 32 GB f32) and the DP4A path that beats llama.cpp — see crate::codegen_cuda::qwen_emit::Q8Ref.