Gemini 3.5 Flash annotates robot video subtasks for $2.64/hour—19x cheaper than humans—with open-source pipeline in Refiner ready to run on your own videos.
Summary
Robotics teams scaling data collection need automated annotation pipelines that work across egocentric and robot-camera footage. This benchmark lets you validate VLM performance before committing annotation budget, and the cost baseline ($2.64/hour) quantifies when human annotation becomes wasteful.
Why it matters
Robotics teams scaling data collection need automated annotation pipelines that work across egocentric and robot-camera footage. This benchmark lets you validate VLM performance before committing annotation budget, and the cost baseline ($2.64/hour) quantifies when human annotation becomes wasteful.
Implementation verdict
Replaces manual video annotation workflows for robotics datasets. Requires access to Gemini 3.5 Flash (batch pricing) and the Refiner framework; best end-to-end F1 is 0.168, so ground-truth validation still needed. Worth prototyping now if you're processing >10 hours of robot video monthly; production viability depends on your F1 tolerance for downstream policy training.
Sources
Dev Signal
Get briefs like this in your inbox — free, every weekday.
100+ sources compressed into one 4-minute read. Ranked, cited, implementation-ready.