DuckDB v2.0 Introduces Async I/O for Parquet and CSV
Original: Asynchronous I/O in DuckDB: Work, Thread, Work
Why This Matters
Async I/O is essential for DuckDB's competitiveness in cloud-native data lake query workloads at scale.
DuckDB announced that v2.0, scheduled for fall 2026, will support asynchronous reads of Parquet and CSV files. This enhancement aims to significantly improve query performance in remote storage setups like EC2/S3, where synchronous I/O often fails to saturate available network bandwidth.
DuckDB has detailed its upcoming asynchronous I/O implementation in a technical blog post, confirming the feature will ship with v2.0 in fall 2026. Preview builds under v2.0.0-dev are already available for testing.
Historically, DuckDB focused on local SSD workloads where synchronous I/O was sufficient, relying on filter and projection pushdown to minimize data reads. However, growing adoption for remote data lake scenarios — such as querying Parquet files stored in Amazon S3 from EC2 instances — has exposed a critical bottleneck: with synchronous I/O, worker threads block and wait for remote reads instead of performing computation like decoding or aggregation.
The new asynchronous I/O pipeline decouples I/O requests from worker thread execution, allowing threads to continue processing while awaiting data. Initial support covers Parquet files and uncompressed, seekable UTF-8 CSV files. Support for DuckDB's native format and JSON is described as forthcoming.
DuckDB's architecture has increasingly targeted remote storage use cases, including the DuckLake data lake integration and a server mode via the Quack protocol launched in May 2026. These developments make efficient remote I/O critical. The blog post includes benchmarks demonstrating meaningful performance gains in EC2/S3 configurations where network bandwidth was previously underutilized due to insufficient concurrent requests.