Is it case-sensitive?
Yes: 'Apple' and 'apple' are treated as different lines. Use the Smart Case Converter first if you need case-insensitive dedup.
Key Features
- Removes exact duplicate lines (case-sensitive)
- Preserves first-seen order
- Handles lists of any size: thousands of lines in milliseconds
- No trimming: blank lines are kept unless duplicated
Benefits
- Clean up email lists, CSV exports, or log dumps
- Dedupe URL lists before batch processing
- Speed up import-to-database workflows
- Sanity-check that your data is unique
How to use this tool
Paste your list, one item per line
Click Process
Copy the deduplicated list
Deduplicating lines, and the three things that silently defeat it
Removing duplicate lines is conceptually trivial and practically full of edge cases, because equality of two lines is less obvious than it looks. Three things defeat a naive comparison, and they account for nearly every case where a list still has duplicates after deduplication.
The first is whitespace. A line with a trailing space is not equal to the same line without one, and neither is visible. A list copied from a spreadsheet, a web page or a PDF is frequently full of them, which is why an export that clearly contains the same email twice comes back with both entries intact. Trimming before comparing is nearly always what you meant.
The second is case. Whether two addresses differing only in capitalisation are duplicates depends entirely on what the list is. Email addresses are effectively case-insensitive in practice, so two spellings are the same person. Identifiers, passwords, hashes and file paths on case-sensitive systems are not, and folding case there merges two distinct records into one. This is a decision to make per list rather than a setting to leave on.
The third is invisible characters. A zero-width space or a non-breaking space inside one copy of a line makes it unequal to its twin, and nothing on screen distinguishes them. A list that refuses to deduplicate no matter what you try usually has one of these in it, and cleaning invisible characters first turns an intractable problem into a trivial one.
Order is the other decision. Preserving the original order and keeping the first occurrence is right when the sequence carries meaning, such as a log or a ranked list. Sorting first is right when it does not and you want the result comparable to something else. Sorting changes the output in ways that matter if anyone downstream assumed the order, so it is worth being deliberate rather than accepting a default.
Everything is processed in your browser. Lists of this kind are usually addresses, customer records or identifiers, which is exactly the category that should not be pasted into a service to be tidied.