Skip to main content

ATTN_HELD_BYTES

logicaffeine_compile::codegen_cuda::lane::kernels

Constant ATTN_HELD_BYTES 

Source 
pub const ATTN_HELD_BYTES: usize = _; // 32_768usize
Expand description

The most shared memory an attention block (lk_attention) keeps its chunks’ partials in, so a row it takes whole (a batch’s sequence, a chunk’s row) weighs each chunk once: a block of 256 threads keeps three to a streaming multiprocessor.