3.1B vision-language model hits 228 tokens/sec on M5 Max, 3GB footprint, with native llama.cpp/MLX/vLLM support—skip the API roundtrip for document/UI understanding.
Summary
Eliminates inference latency and API costs for real-time screen parsing, document OCR, and object grounding in edge applications. Direct transformer loading cuts deployment friction versus proprietary APIs.
Why it matters
Eliminates inference latency and API costs for real-time screen parsing, document OCR, and object grounding in edge applications. Direct transformer loading cuts deployment friction versus proprietary APIs.
Implementation verdict
Replaces heavier 8B models (Gemma-4-E4B, Qwen3.5-4B) for on-device use; requires transformers>=5.10.1 and bfloat16 support. Ready now—benchmark parity on RefCOCO grounding (87.9) and ScreenSpot GUI tasks, but tool-use lags (ToolSandbox 59.5 vs Qwen3.5-2B 47.7). Start with document/screen workloads where it leads.
Sources
Dev Signal
Get briefs like this in your inbox — free, every weekday.
100+ sources compressed into one 4-minute read. Ranked, cited, implementation-ready.