Skip to main content

from_attention

Function from_attention 

Source
pub fn from_attention(
    spec: &AttentionSpec,
    names: &StageNames,
    target: &SpatialTarget,
) -> Result<Vec<SpatialProgram>, SpatialReject>
Expand description

GQA attention of a block of query rows over a cache of l rows: score[j] = (q·k_j)/√hd, row-softmax, then the weighted V sum — THREE sections, because a softmax cannot be one Generic (its maximum and its denominator are reductions whose results the next stage broadcasts back). One row is decode’s single query; a causal block of T rows is a prefill chunk’s queries, row r attending one position past row r − 1.