Pandas Drop Duplicates – pd.df.drop_duplicates()

Do you ever have repeat rows in your data when you don’t want to? Pandas Drop duplicates will remove these for you.

Pandas DataFrame.drop_duplicates() will remove any duplicate rows (or duplicate subset of rows) from your DataFrame. It is super helpful when you want to make sure you data has a unique key or unique rows.

1. YourDataFrame.drop_duplicates()

This function is a combination of DataFrame.drop() and DataFrame.duplicated().

I use this function most when I have a column that represents a unique id of an object. I’ll run .drop_duplicates() specifying my unique column as the subset.

Pseudo code: Look at all the rows (or subset of columns with your rows) and see if there are duplicates. If so, drop ’em.

Pandas Drop Duplicates

.drop_duplicates() is pretty straight forward, the two decisions you’ll have to make are 1) What subset of your data do you want pandas to evaluate for duplicates? and 2) Do you want to keep the first, or last or none of your duplicates?

Duplicate Parameters

• subset: By default, Pandas will look at your entire row to see if it is a duplicate of any other entire row. However, you can tell pandas to only look at a subset of columns to look for duplicates vs the whole. The subset parameter specifies what subset of columns you would like pandas to evaluate.
• keep (Default: ‘first’): If you have two duplicate rows, you can also tell pandas which one(s) to drop. keep=’first’ will keep the first duplicate and drop the rest. Keep=’last’ will keep the last duplicate and drop the last. None will drop all of them.
• inplace (Default: False): If true, you would like to do your operation in place (write over your current DataFrame). If false, then your DataFrame will be returned to you.
• ignore_index (Default: False): If True, then the axis returned to you will be labeled cleaning 0, 1, 2, …, n-1. If not, then the prior index will be used with the index labels of the drop rows dropped as well.

Here’s a Jupyter notebook showing how to set index in Pandas

Link to code

Official Documentation