Skip to main content

lower_batched_tp_decode_streamed

Function lower_batched_tp_decode_streamed 

Source
pub fn lower_batched_tp_decode_streamed(
    spec: &ModelSpec,
    storage: Storage<'_>,
    ctx: usize,
    batch: usize,
    tp: usize,
    target: &SpatialTarget,
    fill: impl Fn(&str) -> String,
) -> Result<StreamedModel, StreamedReject>
Expand description

Lower spec as an SN50 BATCHED, TENSOR-PARALLEL decode deployment — batch sequences per step over tp chips, the serving shape: lower_batched_decode_streamed’s stacked programs with every layer kind lower_tp_decode_streamed’s program over the chips.