Enum CachePrecision 

pub enum CachePrecision {
    F32,
    F16,
}

The precision an attention’s cache is stored in (CudaKnobs::cache_precision).

Variants

F32

Four bytes a value, as the step computes it.

F16

Two bytes a value: binary16, rounded to the nearest, ties to even, as llama.cpp’s default cache keeps it.

Trait Implementations

impl Clone for CachePrecision

fn clone(&self) -> CachePrecision

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CachePrecision

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for CachePrecision

fn default() -> CachePrecision

Returns the “default value” for a type.

impl PartialEq for CachePrecision

fn eq(&self, other: &CachePrecision) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for CachePrecision

impl Eq for CachePrecision

impl StructuralPartialEq for CachePrecision