pub fn compile_to_cuda_forward_q8(
source: &str,
weights: &HashMap<String, (String, usize)>,
q8: &HashMap<String, Q8Ref>,
) -> Result<String, String>Expand description
compile_to_cuda_forward_weighted with an INT8 (Q8_0) weight manifest: any tensor variable named in q8
is loaded resident as int8 quants + f32 block scales, and a matmul against it becomes a dequant-in-kernel
GEMV. This is what fits real 8B weights on a 12 GB card (8 GB int8, not 32 GB f32) and the DP4A path that
beats llama.cpp — see crate::codegen_cuda::qwen_emit::Q8Ref.