Skip to main content

demonstrated_stage_latency

Function demonstrated_stage_latency 

Source
pub fn demonstrated_stage_latency(
    step: &DecodeStep,
    machine: &Machine,
    published_tok_s: f64,
    precision: Precision,
) -> Result<f64, DecodeRefusal>
Expand description

The per-stage latency a published per-user rate implies: everything a token took beyond its memory floor, spread over its dependent stages. A rate this machine reached is a latency it has shown.