pandas read_csv is ~9.82× slower than polars read_csv for reading a CSV
Polars' reader infers in parallel and skips pandas' object-dtype probing.
Result
| Run | pandas read_csv | polars read_csv | Ratio |
|---|---|---|---|
| Run 1 | 418.79 | 42.66 | 9.82× |
| Run 2 | 430.56 | 42.97 | 10.02× |
| Run 3 | 413.26 | 42.97 | 9.62× |
Spread over 3 runs: pandas read_csv CV 2.1% / polars read_csv CV 0.4% / ratio CV 2.0%
The two calls
df = pd.read_csv(path)df = pl.read_csv(path, infer_schema_length=10000)Conditions
Warm page cache on both sides; the file is 400,000 rows with five columns.
Where this number does not apply
The gap narrows when the machine reads from a cold disk, and pandas with `engine='pyarrow'` is a different comparison.
Reproduce
cd bench && python run.py --run 1 --out ../data/run1.json