Remove Duplicate Lines & Sort Text

Deduplicate, trim, drop blank lines and sort a pasted list five ways, each stage switchable on its own. Everything happens in your browser.

🌐 Español

🔒 Private by design: your text is processed locally in your browser and never uploaded to any server.

Written and maintained by Last updated

Five switches that compose

This is a pipeline rather than a single operation, and every stage of it is independently switchable. That sounds like a small design detail and it is the reason the tool is useful for more than one job.

Lines are split, then optionally trimmed, then optionally emptied of blanks, then optionally deduplicated, then optionally sorted. Turn the duplicate removal off and you have a sorter. Turn the sorting off and you have a deduplicator that preserves order. Turn everything off except the blank line filter and you have a compactor.

The order is fixed and chosen deliberately. Trimming has to come before deduplication or two lines differing only in indentation will never match. Sorting has to come last or it will be sorting a set of lines that is about to change.

Cleaning up a pasted list

  1. Paste your list into the box. The action button stays disabled while it is empty.
  2. Choose a Sort mode, or leave it on No sorting (keep order).
  3. Remove duplicate lines is ticked by default. Untick it if you only want to sort.
  4. Tick Treat “Apple” and “apple” as duplicates if capitalisation should not distinguish entries.
  5. Trim leading/trailing whitespace from each line is ticked by default and is almost always what you want.
  6. Tick Remove empty/blank lines to drop the gaps.
  7. Click Remove Duplicate Lines & Sort Text, then Copy to clipboard. Process another clears both boxes.

Line endings are normalised on the way in, so a list copied out of a Windows text editor and one copied out of a terminal behave identically. That single detail removes the most common reason a deduplicator appears to do nothing at all.

Whitespace is why your duplicates survived

The overwhelmingly common complaint about deduplication tools is that they miss obvious duplicates. The cause is nearly always invisible.

A line ending with a space is a different string from the same line without one. So is a line indented with a tab. Copy a column out of a spreadsheet or a web page and you will very often get trailing spaces on some rows and not others, and no amount of staring at the screen will show you which.

That is why trimming defaults to on. It removes the leading and trailing whitespace from each line before anything else looks at them, so the comparison happens on the visible content. If duplicates still survive after trimming, the next suspect is a non-breaking space, which looks exactly like a space and is a completely different character.

The two case settings are not the same setting

There are two places case comes up here and they are independent.

Case insensitive duplicate matching decides whether two lines differing only in capitalisation count as the same entry. It defaults to off, because for a list of identifiers, codes or filenames, case usually carries meaning. Turn it on for a list of names or words where it does not. The surviving line keeps its own capitalisation, so turning this on never silently lowercases your data.

Alphabetical sorting, on the other hand, is always case insensitive and also ignores accent differences. That is not configurable, because a case sensitive sort produces the classic unwanted result where every capitalised word comes before every lowercase one, splitting a list into two blocks nobody asked for.

The numeric sort, and where non-numbers go

Numeric sorting reads a number from the front of each line and sorts on that, which is what makes it useful for lists like version numbers with prefixes, or measurements with trailing units.

Lines with no leading number are treated as larger than any real value, so they collect at the end. Because the sort is stable, they arrive there in the order you pasted them rather than shuffled, which keeps a header row or a note at the bottom in one recognisable piece.

Sorting by length is the least obvious mode and genuinely handy for spotting outliers: a mistyped entry in a list of postcodes or product codes will usually be the wrong length, and length sorting puts it at one end.

Where a list goes next

For counting rather than cleaning, Word Counter gives you words, characters, sentences and paragraphs on the same pasted text. To change the capitalisation of every line rather than compare it, Case Converter handles the transformations.

To find what actually differs between two lists rather than deduplicate one, Diff Text is the right tool. If your list is buried in a block of prose, Email Extractor pulls addresses out of it. And once the list is final, TXT to PDF makes it printable. The rest is on the text tools hub.

See it in action

Screenshot of the Remove Duplicate Lines & Sort Text tool with the sample input “png-to-jpg compress-pdf png-to-jpg srt-to-vtt co…”, Sort set to No sorting (keep order), Remove duplicate lines set to on
Remove Duplicate Lines & Sort Text mid-process: the sample input “png-to-jpg compress-pdf png-to-jpg srt-to-vtt co…”, Sort set to No sorting (keep order), Remove duplicate lines set to on.
Screenshot of the Remove Duplicate Lines & Sort Text result screen showing the generated output “png-to-jpg compress-pdf srt-to-vtt base64-encode”
The finished result: the generated output “png-to-jpg compress-pdf srt-to-vtt base64-encode”. The download link is a local blob URL — the file never leaves your device.

Frequently asked questions

Which copy of a duplicate is kept?

The first one, in the order it appeared. Deduplication is a stable first wins pass rather than a sort, so if you do not also ask for sorting, the surviving lines stay in exactly the order you pasted them. When case insensitive matching is on, the kept line keeps its own original capitalisation rather than being lowercased.

Why do lines that look identical survive as separate entries?

Almost always trailing whitespace. A line ending in a space is a different string from one that does not, and the difference is invisible on screen. The trimming option is on by default for exactly this reason, and it runs before deduplication so trimmed lines can then match each other. If duplicates still survive, check for a non-breaking space or a stray tab.

In what order do the stages run?

Trim first, then drop blank lines, then deduplicate, then sort. That order matters. Trimming before deduplication is what lets differently indented copies of the same value collapse together, and sorting last means the sort sees the final set of lines rather than the original one.

Does the blank line filter also catch lines containing only spaces?

Yes, always. That check is made against the trimmed value regardless of whether you have the trimming option switched on, so a line holding nothing but tabs or spaces counts as blank either way. It is one of the few places where a setting is applied for a check without being applied to the output.

Can I sort without deduplicating?

Yes. Every stage is an independent switch, so untick the duplicate option and pick a sort mode and you get your full list reordered with nothing removed. The reverse is also true, and deduplicating without sorting is the default combination.

How does the numeric sort handle lines that do not start with a number?

They all go to the end, keeping their original order relative to each other. The sort key is the number parsed from the front of each line, and lines without one are treated as larger than any real value. Sorting is stable, so a group of non-numeric lines arrives at the bottom in the sequence you pasted them.

Is alphabetical sorting case sensitive?

No. Sorting compares at a level that ignores both case and accent differences, which is how people expect a plain list to sort, so apple and Apple land next to each other rather than in two separate blocks. That is a separate setting from the case insensitive duplicate matching, which is off by default.

Related tools