multimodal-llm mixture-of-experts open-model audio-vision-text inference-optimization

Inkling open model handles image text audio natively

1T-param decoder-only MoE with 1M context, day-0 transformers/SGLang support, requires 600GB VRAM (NVFP4) or serverless inference.

Summary

Replaces separate vision/audio/text encoders with unified multimodal decoder for reasoning tasks. Enables domain adaptation via fine-tuning without rebuilding modal towers.

Why it matters

Replaces separate vision/audio/text encoders with unified multimodal decoder for reasoning tasks. Enables domain adaptation via fine-tuning without rebuilding modal towers.

Implementation verdict

Ready now via transformers 5.14.0+ or llama.cpp quants. Replaces fragmented multimodal stacks (separate encoders). Requires Hopper/Blackwell GPU or serverless router. Start with NVFP4 (600GB) over BF16 (2TB) for hardware constraints.

Sources

1.1T params and 1M context window to natively receive image, text, and audio inputs
2.975B total and 41B active parameters
3.trained on 45 trillion tokens of text, images, audio and video
4.the bf16 checkpoint requires 2 TB of VRAM, while the nvfp4 version requires 600 GB of VRAM
5.day-0 transformers support and is supported in major inference engines like SGLang and vLLM
6.The multimodal towers are relatively simple modules, unlike other models that employ separate encoders for each modality

Dev Signal

Get briefs like this in your inbox — free, every weekday.

100+ sources compressed into one 4-minute read. Ranked, cited, implementation-ready.

Read the full issue →All briefs