Skip to main content

lower_decode_streamed

Function lower_decode_streamed 

Source
pub fn lower_decode_streamed(
    spec: &ModelSpec,
    storage: Storage<'_>,
    ctx: usize,
    target: &SpatialTarget,
    fill: impl Fn(&str) -> String,
) -> Result<StreamedModel, StreamedReject>
Expand description

Lower spec as an SN50 DECODE deployment: the embedding lookup, one program per layer kind in which every attention layer appends to and attends over its KV cache of ctx positions, and a head that emits the next token. tok and pos are runtime inputs — the programs read them as data — so the bundle serves every step; the values the programs are lowered with (token 1, position 1) only seed their declarations.