Skip to main content

compile_to_cuda_forward_q8_dp4a

Function compile_to_cuda_forward_q8_dp4a 

Source
pub fn compile_to_cuda_forward_q8_dp4a(
    source: &str,
    weights: &HashMap<String, (String, usize)>,
    q8: &HashMap<String, Q8Ref>,
) -> Result<String, String>
Expand description

compile_to_cuda_forward_q8 emitting the INT8-activation DP4A MMQ kernel for each Q8 matmul: the activation is quantized to int8 per 32-block and __dp4a’d against the int8 weight (int32 accumulate) — the same kernel shape llama.cpp’s fast path uses, and the shipped batch-1 default. Numerically the int8-activation approximation of the dequant GEMV (small quantization error), not bit-identical; the TwoPassInt8 mode recovers ~14-bit activation exactness at two __dp4a per block.