How to remove duplicate rows from a CSV file

Duplicate rows creep into contact lists, order exports and sign-up sheets all the time. Here are five ways to remove them, starting with the ones you already have. Before you start, decide what counts as a duplicate: the whole row, or one column such as Email. Two rows for the same customer usually differ somewhere (a sign-up date, a spelling), so whole-row matching often misses them.

1. Excel

1. Open the CSV in Excel and click anywhere in the data.
2. Go to Data › Remove Duplicates.
3. Tick only the columns that define a duplicate (for example Email), then click OK.
4. Save as CSV again (File › Save As › CSV UTF-8).

Watch out: Excel keeps the first occurrence and does not tell you which rows it removed. A stray space ("ann@example.com " with a trailing space) makes otherwise identical values count as different, so trim the column first (the TRIM function). Excel may also change long numbers, leading zeros and dates when it opens a CSV.

2. Google Sheets

1. File › Import, and upload the CSV. Note that this uploads the file to Google.
2. Select the data, then Data › Data cleanup › Remove duplicates.
3. Choose the columns, then click Remove duplicates. Then File › Download › CSV.

Run Data cleanup › Trim whitespace first, so stray spaces do not hide duplicates. Like Excel, it does not list the rows it removed.

3. LibreOffice Calc (free, offline)

1. Open the CSV and select the data, including the header row.
2. Go to Data › More Filters › Standard Filter, set the condition to a column "= Not empty", open Options, and tick No duplications. To keep the result separately, tick Copy results to as well.
3. Save the filtered rows as a new CSV.

4. Command line (macOS, Linux, Windows with WSL)

Remove exact duplicate lines and keep the original order:

awk '!seen[$0]++' contacts.csv > contacts-deduped.csv

sort -u contacts.csv also removes exact duplicates, but it re-orders the rows and can move your header line. Both work on whole lines only. A quoted field that contains a comma or a line break will confuse anything that splits columns by commas.

5. Python (standard library only)

Remove duplicates by email, ignoring capitalisation and spaces, and keep a list of what was removed. For a fuller version that also records the reason for each removed row, see removing duplicates by one column.

import csv

with open("contacts.csv", newline="", encoding="utf-8-sig") as f:
    reader = csv.DictReader(f)
    fields = reader.fieldnames
    rows = [r for r in reader if r["Email"] is not None]    # skips the short row

seen, kept, removed = set(), [], []
for r in rows:
    key = r["Email"].strip().lower()
    (removed if key in seen else kept).append(r)
    seen.add(key)

for name, data in (("contacts-deduped.csv", kept), ("contacts-removed.csv", removed)):
    with open(name, "w", newline="", encoding="utf-8-sig") as f:
        w = csv.DictWriter(f, fieldnames=fields)
        w.writeheader()
        w.writerows(data)
print(len(kept), "kept,", len(removed), "removed")

5 kept, 2 removed

Notice the count: the row with an empty email is "kept" as one row (all empty emails look alike to this simple script, so only the first survives), and the row that was missing a column was skipped without a trace. The longer script linked above reports both instead of hiding them.

When a dedicated tool is worth it

The methods above are free and fine for occasional use. They fall short in three situations:

• The file is private (customers, patients, members), and uploading it to an online converter is not acceptable.
• You need to know exactly what was removed, for a client or an audit.
• You are not comfortable with code, but the file is too big or messy for a spreadsheet.

Last updated 2026-10-27. Code on this page was run on that date and the output shown is the real output. Spreadsheet menu steps are written from Microsoft's and Google's documented behaviour; where we have not run something ourselves, the page says so.