Skip to main content

from_lut_attention

Function from_lut_attention 

Source
pub fn from_lut_attention(
    spec: &LutAttentionSpec,
    names: &StageNames,
    target: &SpatialTarget,
) -> Result<Vec<SpatialProgram>, SpatialReject>
Expand description

C = lut attention(Q, K, V) — an int8 model’s single-query attention, integer throughout, in THREE sections: the scores s = q·k_j over the cache; per head, the weights w_j = E[clamp(((max − s_j)·M) >> S, 0, n − 1)] for the positions attended and 0 past them; and the context ⌊Σ_j w_j·v_j / Σ_j w_j⌋. A masked position is fenced off only where it could change an answer — the maximum and the weights — and the sums need no mask: a weight of 0 adds nothing. Every value the masked step computes is one the unmasked step could: a masked position folds into the maximum as the head’s first score, which every P ≥ 1 attends, so no sentinel widens what the maximum holds.