Forge lifts 8B models to agent-class reliability
Drop-in guardrails middleware + proxy server that rescues malformed tool calls, enforces step ordering, and manages VRAM context for self-hosted agentic workflows — no model retraining required.
May 29, 2026
Summary
Local inference teams hit a wall with multi-step tool use — models fail at parsing, skip steps, or blow context. Forge's composable middleware (validator, step enforcer, retry nudges) plugs directly into existing orchestration or works as a transparent OpenAI-compatible proxy, letting developers upgrade reliability without refactoring agents.
Why it matters
Local inference teams hit a wall with multi-step tool use — models fail at parsing, skip steps, or blow context. Forge's composable middleware (validator, step enforcer, retry nudges) plugs directly into existing orchestration or works as a transparent OpenAI-compatible proxy, letting developers upgrade reliability without refactoring agents.
Implementation verdict
Replaces manual response validation + retry logic in your agentic loop. Requires Python 3.12+, a running llama.cpp/Ollama/Anthropic backend, and either direct integration (WorkflowRunner) or proxy interception (minimal code). Ready now — 26-scenario eval suite validates real workflows; top config (Ministral-3 8B Q8) scores 86.5% baseline, 76% on hard tier. Proxy path has zero integration cost if you already use OpenAI-compatible clients (Continue, aider, opencode).
Sources
- 1.Ministral-3 8B Instruct Q8 on llama-server) scores 86.5% across forge's 26-scenario eval suite — and 76% on the hardest tier
- 2.guardrails (rescue parsing, retry nudges, step enforcement) and context management (VRAM-aware budgets, tiered compaction)
- 3.Drop-in OpenAI-compatible proxy (python -m forge.proxy) that sits between any client (opencode, Continue, aider, etc.) and a local model server
- 4.Supports Ollama, llama-server (llama.cpp), Llamafile, and Anthropic as backends
Dev Signal
Get briefs like this in your inbox — free, 3x a week.
100+ sources compressed into one 4-minute read. Ranked, cited, implementation-ready.