pandas drop_duplicates is ~1.77× slower than polars unique for removing duplicate key pairs
Two integer columns, 400,000 rows, many duplicates.
Result
| Run | pandas drop_duplicates | polars unique | Ratio |
|---|---|---|---|
| Run 1 | 32.92 | 18.58 | 1.77× |
| Run 2 | 31.86 | 19.04 | 1.67× |
| Run 3 | 46.34 | 25.76 | 1.80× |
Spread over 3 runs: pandas drop_duplicates CV 21.8% / polars unique CV 19.0% / ratio CV 3.8%
The two calls
df[['g','k']].drop_duplicates()df.select(['g','k']).unique()Conditions
Only the two key columns are materialised on both sides.
Where this number does not apply
Keeping the first/last row (`keep=`) adds work that is not timed here.
Reproduce
cd bench && python run.py --run 1 --out ../data/run1.json