Module hf 

A Hugging Face checkpoint directory read in place: config.json (its text_config merged over it as a multimodal checkpoint nests it), the safetensors shards its index names, and the tokenizer files. It is presented as the GGUF llama.cpp’s converter (convert_hf_to_gguf.py) writes from it — the same architecture keys, tensor names, ggml dim order and element order — so ModelSpec::from_source, the binder and the reference read it unchanged.

Nothing is read whole: opening reads the shards’ headers, and a tensor’s rows are read when they are asked for. Most tensors are their bytes as stored, dims reversed into ggml’s order; a dtype other than f32, f16 and bf16 (the 8-bit floats, the integers) is widened to f32, as the converter widens it. The converter’s rearrangements are views over the stored bytes, kept in the checkpoint’s own dtype:

• a row permutation — Llama’s query and key rows into interleaved rotation pairs; the value heads of a Gated DeltaNet block from grouped-by-key-head to tiled order; • a column permutation or pick — the same reorder over an output projection’s inputs, gpt-oss’s interleaved gate and up halves; • a transpose of the last two axes — gpt-oss’s expert matrices, latent attention’s key half; • a stack — a mixture’s experts, one tensor per projection.

Two converter steps change values: a norm stored as an offset from one (Gemma 3, Qwen3.5: the forward multiplies by 1 + w) and Gated DeltaNet’s decay stored as its logarithm (A = -exp(A_log)). The converter computes them in the dtype it holds the tensor in — f16 and f32 as stored, every other dtype widened to f32 first — and the checkpoint’s own forward computes them in f32 from the stored value. The two differ only for an f16 checkpoint, whose converter rounds each result to f16; Recompute chooses which the reader presents: the converter’s by default, so a directory reads as its converted GGUF, or the forward’s. Either way 1 + w is exact and the exponential correctly rounded; the converter’s own exponential is torch’s on the CPU (MKL’s on x86 builds, chosen by the processor), which lands one unit in the last place off the correctly rounded value now and then.

A configuration key is either read, known to leave the computation unchanged, or refused by name: a checkpoint is never read as something other than what it says it is.

The tokenizer files are read as the converter reads them (HfCheckpoint::tokenizer_meta): the vocabulary, merges, special tokens and chat template it writes, the byte-pair pre-tokenizer named by the digest the converter’s own probe gives under the directory’s tokenizer.json, run as the tokenizers library runs it (HfTokenizer).

Structs

HfCheckpoint

A Hugging Face checkpoint directory, presented as its converted GGUF.

HfPreTokenizer

A tokenizer.json pre-tokenizer alone, as tokenizers’ pre_tokenize_str runs one.

HfTokenizer

A tokenizer.json byte-pair tokenizer.

Tiktoken

A tiktoken tokenizer read from its directory.

Enums

Form

How a presented tensor’s elements are found in the checkpoint.

Op

An elementwise step the converter applies to a tensor’s values.

Recompute

How the values a converter step computes (a norm’s offset from one, a decay’s logarithm) are presented.

Constants

NORMAL_FORM_VERSION

The Unicode version of the tables the normal forms read (unicode-normalization-alignments’).

PATTERN_VERSION

The Unicode version of the tables the patterns read (Oniguruma’s).

PRE_TOKENIZER_DIGESTS

The converter’s get_vocab_base_pre table: the SHA-256 of the probe’s ids, written as a Python list, and the pre-tokenizer name it writes for them, in the converter’s order (a digest listed twice names what its last entry says).

PRE_TOKENIZER_PROBE

The text whose tokenization names a byte-pair pre-tokenizer (the converter’s chktxt).

Functions

nfc

NFC: the canonical decomposition, recomposed by the canonical composition algorithm.

pre_tokenizer_named

The pre-tokenizer the converter’s table names for a probe digest: its last entry for the digest, as the converter’s chain of ifs leaves it when a digest is listed twice.

tiktoken_split

The pieces tiktoken’s pattern pattern finds in text, in order.