Skip to main content

attn_heads_per_block

Function attn_heads_per_block 

Source
pub fn attn_heads_per_block(
    group: usize,
    hd: usize,
    vd: usize,
    span: usize,
    sms: usize,
    smem: usize,
    acc: usize,
) -> Option<usize>
Expand description

The query heads an attention block takes together: the largest divisor of group, at most 16, whose keys and values a position stay within ATTN_BLOCK_VALUES (one head whatever its width), whose accumulators (g heads’ value pairs a thread, twice over in a block that takes every chunk) stay within 32 registers, and whose queries, scores and chunk sums fit smem bytes — None when not even one head fits.