Format · Established · Advanced
GGUF format
A file format for distributing quantised models for local inference, carrying weights and metadata in one file.
What GGUF format is
GGUF became the standard container for locally run models, with a range of quantisation levels trading size against quality.
How it works
A model is converted from its original weights to GGUF at a chosen quantisation, then loaded by compatible runtimes on CPU or GPU.
Why it matters
It is why a multi-billion-parameter model can be downloaded as a single file and run on a laptop.
Common uses
- →Local model distribution
- →Offline deployments
- →Consumer hardware inference
Strengths
- ✓Single portable file
- ✓Wide runtime support
Watch for
- ✓Quality loss at low bit widths
- ✓Conversion step required
Continue exploring
More in this collection
Browse all AI Technology