· 5 min read
Deduplicating a list sounds like one operation. It is actually four decisions, and a tool that makes them silently will sometimes give you an answer that is confidently wrong.
For email addresses, the local part is technically case sensitive but in practice never is, so treating them case-insensitively is right. For identifiers, tokens and anything used as a key, case matters and folding it merges two different things. Getting this backwards either leaves duplicates behind or silently deletes real entries.
Lines copied from a document or a spreadsheet routinely carry trailing spaces, and the invisible ones are worse. Two lines that look identical can differ by a trailing space or a non-breaking space, and an exact comparison keeps both. Trimming before comparing is almost always what you want.
If the lines are just values, it makes no difference. If they are records where later entries are updates, it makes all the difference. Keeping the first is the common default, which is the wrong one for a log where the last entry is the current state.
Sorting makes duplicates adjacent and is how most command-line pipelines do it, but it destroys the original order. If the order carries meaning, such as a sequence of steps or a chronological list, deduplicate while preserving order instead.
The combination that quietly loses data is: exact comparison, keep first, sort. It leaves whitespace-variant duplicates in place, discards the newest version of a record, and destroys the ordering that would have let you notice.
Sorting lines, removing blank lines and trimming whitespace are usually wanted together, and doing them in the right order matters: trim first, then deduplicate, then sort. Trimming after deduplicating leaves the whitespace-variant duplicates behind.
These run in the browser here, so a list of customer addresses stays on your machine while you clean it.
Related: How to extract email addresses from a block of text
Almost always trailing or invisible whitespace, or a case difference. Trim and normalise the lines first, then compare.
Keep the last when later lines are updates, such as in a log or an export with revisions. Keep the first when the lines are plain values and order of appearance matters.
No. Sorting is one way to bring duplicates together, but order-preserving deduplication keeps the original sequence, which matters when the order carries meaning.