quantization edge-inference gguf llama-cpp distillation

LFM2.5 Q4_0 GGUFs recover 97% BF16 accuracy

Quantization-Aware Distillation trades post-training quantization for teacher-student distillation, recovering ~97% accuracy at native Q4_0 speed/memory on edge hardware.

Summary

Eliminates the accuracy cliff of standard 4-bit quantization, letting developers deploy smaller models to phones and Raspberry Pi without retraining. Direct swap into llama.cpp pipelines.

Why it matters

Eliminates the accuracy cliff of standard 4-bit quantization, letting developers deploy smaller models to phones and Raspberry Pi without retraining. Direct swap into llama.cpp pipelines.

Implementation verdict

Replaces post-training Q4_0 checkpoints. Requires no code changes—drop QAD GGUFs into any GGUF-compatible runtime. Ready now on HuggingFace. Worth testing if you're shipping on constrained edge hardware and currently accepting PTQ quality loss.

Sources

1.Recovery: 97% of their BF16 average accuracy lost to quantization is recovered
2.The QAD checkpoints retain 97.1%, 96.5%, 97.4%, and 96.6% of their respective BF16 baseline performance
3.The 230M and 350M QAD Q4_0 checkpoints match Q5_K_M quality within evaluation variance at a 4-33% higher decode throughput
4.Use the files with llama.cpp or any runtime that supports GGUF Q4_0 artifacts

Dev Signal

Get briefs like this in your inbox — free, every weekday.

100+ sources compressed into one 4-minute read. Ranked, cited, implementation-ready.

Read the full issue →All briefs