Struct Deployment 

pub struct Deployment {
    pub ctx: u64,
    pub weights: WeightBytes,
    pub kv_bytes: f64,
    pub state_bytes: f64,
    pub activation_bytes: f64,
    pub speculative: bool,
}

What a deployment of a checkpoint holds per token.

Fields

ctx: u64

The context the KV cache has reached.

weights: WeightBytes

kv_bytes: f64

Bytes per KV-cache element.

state_bytes: f64

Bytes per recurrent-state element.

activation_bytes: f64

Bytes per activation element a reduction across chips carries.

speculative: bool

Whether the speculative (multi-token-prediction) blocks run as part of the step.

Implementations

impl Deployment

pub fn as_stored(ctx: u64) -> Deployment

The checkpoint as stored, with llama.cpp’s defaults: an f16 KV cache and an f32 recurrent state.

Trait Implementations

impl Clone for Deployment

fn clone(&self) -> Deployment

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Deployment

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Deployment

fn eq(&self, other: &Deployment) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for Deployment

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl Copy for Deployment

impl StructuralPartialEq for Deployment