Skip to main content

computed_tables

Function computed_tables 

Source
pub fn computed_tables(spec: &ModelSpec) -> Vec<HostTable>
Expand description

The host tables beyond the rotary ones that a decode step of spec reads, each once: a row of ones as wide as each head that a layer norms without a weight, a row of ones as wide as the model that spreads a shared expert’s gate across its row, and the cosines and sines of the attention temperature for each head width that takes one.