vision-language-models spatial-reasoning benchmark evaluation multimodal

Vision models fail spatial reasoning despite format validity

StateSight benchmark exposes that GPT-4.5 and Claude Sonnet 5 produce syntactically correct but spatially incoherent outputs on cube-net and tower-counting tasks, masking fundamental reconstruction failures.

Summary

If you're building multimodal QA systems that depend on spatial understanding from single images, format-valid API responses hide real failures in image-state reconstruction—you need task-specific benchmarks to catch this gap before production.

Why it matters

Implementation verdict

Doesn't replace existing vision pipelines yet; reveals that broad evals mask spatial reasoning brittleness. Requires custom procedural benchmarks for your domain. Worth running StateSight on your target model before shipping spatial-inference features—human baselines (80.8% on cubes) far exceed current model performance (59.3% GPT-4.5, 53.3% Claude).

Sources

1.Format-valid responses can mask failures to recover the spatial structure required for verifiable visual inference
2.OpenAI GPT-5.5, using the API model identifier gpt-5.5, achieved 59.3%, 33.3%, and 28.3% accuracy across the three tasks
3.A 30-participant human baseline on 60 items exceeded both models on every task, with mean accuracies of 80.8%, 68.8%, and 64.3%

Dev Signal

Get briefs like this in your inbox — free, every weekday.

100+ sources compressed into one 4-minute read. Ranked, cited, implementation-ready.

Read the full issue →All briefs