pandas merge is ~11.93× slower than polars join for joining a 10,000-row lookup
Same shape as the small join, with a ten times larger right side.
Result
| Run | pandas merge | polars join | Ratio |
|---|---|---|---|
| Run 1 | 32.79 | 2.75 | 11.93× |
| Run 2 | 35.19 | 2.79 | 12.61× |
| Run 3 | 35.42 | 3.41 | 10.39× |
Spread over 3 runs: pandas merge CV 4.2% / polars join CV 12.4% / ratio CV 9.7%
The two calls
df.merge(lookup, on='g', how='left')df.join(lookup, on='g', how='left')Conditions
10,000 distinct keys on the right, 400,000 rows on the left.
Where this number does not apply
Duplicated keys on the right would multiply rows and change the cost profile.
Reproduce
cd bench && python run.py --run 1 --out ../data/run1.json