Skip to main content

lower_batched_decode_streamed

Function lower_batched_decode_streamed 

Source
pub fn lower_batched_decode_streamed(
    spec: &ModelSpec,
    storage: Storage<'_>,
    ctx: usize,
    batch: usize,
    target: &SpatialTarget,
    fill: impl Fn(&str) -> String,
) -> Result<StreamedModel, StreamedReject>
Expand description

Lower spec as an SN50 BATCHED decode deployment: batch sequences per step, each with its own token, position and carried state, every weight read once for all of them (the model’s Cut::Batch). The same three kinds of program as lower_decode_streamed — the embedding lookups, one program per layer kind, the head — over the stacked batch × d hidden states; the token ids toks and each sequence’s pos_s{b} are runtime inputs.