Struct Serving 

pub struct Serving {
    pub batch: u32,
    pub contexts: ContextMix,
    pub kv_page: Option<u64>,
    pub attention: AttentionMath,
    pub kv_bytes: f64,
    pub state_bytes: f64,
    pub activation_words: u32,
    pub scaling: ActScaling,
    pub kv_rotated: bool,
    pub draft: Option<Draft>,
    pub flush: StateFlush,
    pub host_memory: bool,
}

One serving configuration of a checkpoint on a card.

Fields

batch: u32

Sequences that ride a pass.

contexts: ContextMix

kv_page: Option<u64>

Positions per KV page when the caches are paged: each sequence then holds its own length and half a page, on average, instead of its whole limit.

attention: AttentionMath

kv_bytes: f64

Bytes per KV-cache element.

state_bytes: f64

Bytes per recurrent-state element.

activation_words: u32

Words each activation crosses the array in: one 8-bit word, or two (a word and its residual).

scaling: ActScaling

How the activations a matrix reads are scaled: per 32 elements, or per the matrix format’s block.

kv_rotated: bool

Whether each cached key and value head row is rotated by the Walsh–Hadamard transform before its int8 codes, the queries rotated alike and every head’s weighted sum rotated back.

draft: Option<Draft>

flush: StateFlush

host_memory: bool

Whether sequences the card’s memory cannot hold, or whose traffic it is better without, may keep their state and caches in host memory, streamed over the host link each pass.

Trait Implementations

impl Clone for Serving

fn clone(&self) -> Serving

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Serving

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Serving

fn eq(&self, other: &Serving) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for Serving

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for Serving