NVIDIA open-sourced NeMo Switchyard to route each agent step independently to different models, replacing single-model-per-request patterns with per-step selection that cuts costs to roughly one-third while maintaining frontier accuracy.
Summary
Agent workflows fan out into dozens of heterogeneous subtasks (planning, coding, review, testing). Per-request routing wastes compute on cheap steps and degrades output on hard ones. Per-step routing lets you route lightweight tasks to fast cheap models and complex reasoning to frontiers, directly reducing your inference bill while maintaining quality.
Why it matters
Agent workflows fan out into dozens of heterogeneous subtasks (planning, coding, review, testing). Per-request routing wastes compute on cheap steps and degrades output on hard ones. Per-step routing lets you route lightweight tasks to fast cheap models and complex reasoning to frontiers, directly reducing your inference bill while maintaining quality.
Implementation verdict
Replaces gateway-only routing products like Google Cloud API Gateway and Databricks Unity AI Gateway. Requires self-hosting a router (Switchyard or custom logic) plus consolidating vendor credentials into a unified supply layer (the article suggests wrouter.ai as one option). Ready now: weights ship on Hugging Face, Switchyard is open-source, and VS Code extension support exists. Start with per-step routing tables and a single base_url entry point; this is worth trying immediately if you run agentic workflows.
Sources
Dev Signal
Get briefs like this in your inbox — free, every weekday.
100+ sources compressed into one 4-minute read. Ranked, cited, implementation-ready.