DuckDB 1.4 will support producing Polars Lazy Dataframes

4 months ago 11

This PR, extends the .pl() function to produce lazy dataframes. We added a lazy argument to the function, that by default is set to False, meaning that .pl() will produce an eager dataframe, while .pl(lazy=True) will produce a lazy one.

Additionally, this PR adds projection and filter pushdown. The filter pushdown is on par with our PyArrow filter pushdown implementation.

We also fall back to not performing filter pushdown if it isn't possible. For example, when a type not currently supported by the pushdown mechanism is encountered, the filter will be executed in the Polars layer.

Special thanks to @AndreasAlbertQC for providing a gist demonstrating how to use the Polars IO plugin functionality.

Fix: #8017

Read Entire Article