Enum Q8MatKind 

pub enum Q8MatKind {
    Dequant,
    Int8Act,
    TwoPassInt8,
}

How a Q8_0 matmul lowers to CUDA. At batch-1 decode a matmul is bound on the int8 WEIGHT bytes, not the math — so these three modes trade accuracy/latency, NOT weight bandwidth.

Variants

Dequant

f64 dequant, thread-per-output — the numeric reference (exact, but uncoalesced and slow).

Int8Act

SINGLE-PASS int8-activation MMQ (__dp4a, int32 accumulate), coalesced — THE DEFAULT fast path: llama.cpp-matched activation precision (identical greedy token) with the shortest per-iteration dependency chain, which wins the latency-bound batch-1 regime (the A100 head-to-head beat). A near-tie logit can flip exactly as llama.cpp’s does.

TwoPassInt8

TWO-PASS int8 activation (hi byte + lo residual byte ⇒ ~14-bit) — the logit-exact mode: the same short __dp4a dependency chain, ~2× the activation math (two dp4a per block), logits within ~1e-3 of the f64 reference so a near-tie token does NOT flip. The correctness mode llama.cpp lacks.

Trait Implementations

impl Clone for Q8MatKind

fn clone(&self) -> Q8MatKind

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Q8MatKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Q8MatKind

fn eq(&self, other: &Q8MatKind) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for Q8MatKind

impl Eq for Q8MatKind

impl StructuralPartialEq for Q8MatKind