fileformer
search⌘KtranslateENexpand_moreupload_fileConvert
Converters8 file families · 44 direct conversions
What changes in a conversionarrow_forward
home/document-tools/text-cleaner

Paste a messy list. Tidy it in one pass.

Trim spaces, drop blank lines, remove duplicates and sort, in the order you choose. Each step shows how many lines it changed, so you can see exactly what happened to your data.

input17 lines · 235 chars
steps · run top to bottom
Fix invisible characters3 lines changedignore caseexpand_lessexpand_more
Trim spaces3 lines changedignore caseexpand_lessexpand_more
Collapse inner spacesno changeignore caseexpand_lessexpand_more
Remove blank lines3 lines removedignore caseexpand_lessexpand_more
Remove duplicates4 lines removedcheck_boxignore caseexpand_lessexpand_more
Sort A→Z (natural)9 lines movedignore caseexpand_lessexpand_more
Reverse orderoffignore caseexpand_lessexpand_more
output10 lines · 7 fewer
"Priority" customerÉmile Laurentliam.brown@example.comNoah PatelOrder file1.pdfOrder file2.pdfOrder file10.pdfPriya Shahpriyashah@example.comZoë Ahmed
content_copyCopy resultinputUse as input
~/a-to-z-is-ambiguous

Why "file10" sorts before "file2".

A plain sort compares character codes: capitals before lower case, digits one at a time, accented letters at the very end. A natural sort reads numbers as numbers and ignores case, which is what people expect. The same eight lines, three ways.

Plaincharacter codes
AppleBananaReport2.pdfapplereport1.pdfreport10.pdfzebraÉmile
Capitals first, then lower case; “report10” before “report2”; Émile last.
Ignore casestill text digits
appleAppleBananareport1.pdfreport10.pdfReport2.pdfzebraÉmile
Apple and apple sit together, but numbers are still compared digit by digit.
Naturalwhat we use
appleAppleBananaÉmilereport1.pdfReport2.pdfreport10.pdfzebra
Numbers are read as numbers and accents sort with their letter.
~/why-duplicates-survive

Two lines that look identical often aren't.

Text copied from web pages, PDFs and Word carries characters you can't see. They stop duplicates matching and break formulas. This scan shows what's hiding in the input above; the "Fix invisible characters" step replaces them.

Non-breaking spaces1Noah·PatelLook like spaces, but a search for “Noah Patel” won’t find them. Common from web pages.
Zero-width characters1priya|shahTake up no space at all. Break emails, passwords and matching without any visible sign.
Curly quotes2“Priority”Word and phones swap straight quotes for curly ones, which break code, CSV files and formulas.
Leading or trailing spaces3␣Zoë AhmedThe most common reason two “identical” rows don’t count as duplicates in a spreadsheet.
~/same-letters-different-bytes

Two strings can look identical and still not match.

Unicode often has more than one way to write the same thing. “é” can be one code point, or an “e” followed by a combining accent. “fi” can be a single ligature character. Normalisation picks one form, so text compares and searches correctly. Type or paste anything to see every code point, and what each of the four Unicode normal forms does to it. All of it is computed by your browser's String.prototype.normalize.

café vs caféfileZero​widthHangulÅ ngströmEmoji familyRight-to-left
cU+006363aU+006161fU+006666éU+00E9C3 A9U+002020cU+006363aU+006161fU+006666eU+006565◌́U+0301CC 81
■ ordinary■ combining mark■ invisible or control■ compatibility characterbottom line: UTF-8 bytes
formresultcode pointsUTF-8 byteswhat it does
inputcafé café1012As typed or pasted.
NFCcafé café911Canonical composition. Combines letters and accents where a single character exists. The form the web (W3C recommends it), most databases and most keyboards produce.
NFDcafé café1113Canonical decomposition. Splits characters into base letters plus combining marks, the form older macOS file systems (HFS+) stored.
NFKCcafé café911Compatibility composition. Also flattens ligatures, circled digits, full-width letters and similar into plain ones. Good for search keys and user names; it loses formatting meaning.
NFKDcafé café1113Compatibility decomposition. Flattens as NFKC does, then splits accents off.
lightbulbTwo words here look the same but are different code points. They only become equal after normalising, so compare strings with a.normalize('NFC') === b.normalize('NFC'), and normalise before storing.
~/reference

Characters you can't see, and where they come from.

The usual suspects when two lines look the same but won't match, or text behaves oddly after a copy and paste. The last column checks your browser's \s regex and trim(), which don't catch all of them.

code pointnameUTF-8usually comes from\s · trim()
U+00A0No-break spaceC2 A0Web pages, Word,   in HTMLyes · removed
U+200BZero width spaceE2 80 8BWeb editors, Slack and copied URLsno · kept
U+200CZero width non-joinerE2 80 8CPersian and Indic text, where it’s meaningfulno · kept
U+200DZero width joinerE2 80 8DJoins emoji into one, like families and flagsno · kept
U+2060Word joinerE2 81 A0Prevents line breaks; replaced the old BOM useno · kept
U+FEFFByte order markEF BB BFThe start of files saved by Notepad and Excelyes · removed
U+00ADSoft hyphenC2 ADHyphenation hints from Word and InDesignno · kept
U+202FNarrow no-break spaceE2 80 AFFrench punctuation, and macOS time formats since 2023yes · removed
U+2028Line separatorE2 80 A8Copied from PDFs; broke JavaScript strings before ES2019yes · removed
U+202ERight-to-left overrideE2 80 AESpoofed file names: invoice[U+202E]fdp.exe displays as invoiceexe.pdfno · kept
U+3000Ideographic spaceE3 80 80Chinese and Japanese input methodsyes · removed
U+0009Character tabulation09Spreadsheets and code; renders as variable widthyes · removed