Streaming execution and spill-to-disk now default; SQL becomes first-class with TPC-H/TPC-DS benchmarks showing Polars faster than DuckDB 1.5.6 and DataFusion 54.0.0.
Summary
Calling collect() on LazyFrame now uses streaming by default, slashing memory overhead on most queries. Out-of-core spilling at 80% RAM threshold means datasets larger than available memory no longer crash—critical for production workloads. Stricter type enforcement enables agents to validate schemas without materializing data, shortening AI-driven iteration cycles.
Why it matters
Calling collect() on LazyFrame now uses streaming by default, slashing memory overhead on most queries. Out-of-core spilling at 80% RAM threshold means datasets larger than available memory no longer crash—critical for production workloads. Stricter type enforcement enables agents to validate schemas without materializing data, shortening AI-driven iteration cycles.
Implementation verdict
Replaces pandas/DuckDB for medium-scale ETL if you accept non-deterministic row order in joins/group_by without maintain_order=True. Requires code review for order-sensitive operations. Worth upgrading now if you hit memory limits or run SQL workloads; migration guide exists. Spill-to-disk for joins/group_by still pending, so very large aggregations may still need tuning.
Sources
Dev Signal
Get briefs like this in your inbox — free, every weekday.
100+ sources compressed into one 4-minute read. Ranked, cited, implementation-ready.