Skip to main content

q8_gemv

Function q8_gemv 

Source
pub fn q8_gemv(
    spec: &MatmulSpec,
    names: &MatmulNames,
    fills: (&Fill, &Fill),
    view: &str,
    target: &SpatialTarget,
) -> Result<Option<Vec<SpatialProgram>>, SpatialReject>
Expand description

y = x · W for an m-row x and a Q8_0 W: the reshape of x into its blocks, then the fold — or None when the fold would move more than reading the weight dense.

★ A BATCH READS ITS VIEW ONCE PER COLUMN GROUP. Every item folds all m rows against its own columns of the weight, so the view’s m tiles per K block are read by every item: m · 2 bytes per K element per group of 32·wb columns, against the weight’s 1.125 · 32·wb. A site holds wb of 2 at a batch of 8 — the view adds ~22% — and fewer at a larger batch, where the view would outweigh what the int8 values save. So the fold is taken only while its bytes stay below the bf16 weight’s own, which bounds every dense lowering from below.