pandas merge is ~11.24× slower than polars join for joining a 1,000-row lookup
Left join against a 1,000-row key table.
Result
| Run | pandas merge | polars join | Ratio |
|---|---|---|---|
| Run 1 | 31.11 | 2.73 | 11.40× |
| Run 2 | 38.45 | 3.42 | 11.24× |
| Run 3 | 30.99 | 2.81 | 11.03× |
Spread over 3 runs: pandas merge CV 12.7% / polars join CV 12.7% / ratio CV 1.6%
The two calls
df.merge(lookup, on='g', how='left')df.join(lookup, on='g', how='left')Conditions
Both sides build the same hash table on an integer key.
Where this number does not apply
A join against a large table has different memory behaviour and is not covered by the small-join number.
Reproduce
cd bench && python run.py --run 1 --out ../data/run1.json