Text encoding, Unicode normalization and case mapping: decisions that survive counting, sorting and diffing

Before you transform text: encoding, normalization form, and line endings

Three questions decide whether a transformation is safe, and they must be answered in this order: are these bytes decoded yet, which charset decodes them, and what shape is the text in once it exists as a string. Uppercasing, trimming or truncating a string that a decoder already filled with U+FFFD does not fail; it succeeds quietly and publishes the damage, which is why ordering is the first gate.

Decode strictly. TextDecoder replaces each malformed sequence with U+FFFD and returns a string, so a corrupted decode looks like success; { fatal: true } raises a TypeError on the same input. Choose fatal mode whenever the output will be stored, and replacement only when showing best-effort text to a human reader. The charset is a declaration, not a guess: UTF-8 covers U+0000 to U+10FFFF in one to four bytes, and reading valid UTF-8 through a legacy single-byte charset splits 中 (E4 B8 AD) into three characters that no later pass can reunite.

Use UTF-8 for interchange and storage. Choose UTF-16 only where a platform demands it, such as Windows wide-character APIs or the internal representation of JavaScript strings, and remember that a code point above U+FFFF costs two UTF-16 units there. Normalize to NFC on entry, and treat NFKC as a display-only concession described in section three.

Line endings are a fourth decision that teams defer until a diff explodes. Adopt LF as the internal form and convert CRLF at protocol boundaries: RFC 5322 messages and HTTP header fields require CRLF, and CSV written by RFC 4180 tooling often arrives with it. Record the original style before converting, because a file with mixed endings cannot be reconstructed afterwards.

Case conversion is locale-dependent: Turkish i, German sharp s, Greek final sigma

Case mapping is a function from one string to another, not a swap of two letters, and it is not injective. Three locales break the naive assumption, and all three failures reach production because the feature only looks wrong to speakers of the language it affects.

German: U+00DF (sharp s) uppercases to SS under the default full mapping recorded in the Unicode SpecialCasing data, so three characters become two. Capital sharp s U+1E9E exists and has since Unicode 5.1, but uppercase followed by lowercase still does not restore the original: STRASSE does not return to Straße. Any slug, filename or identifier built by uppercasing German text is permanently lossy, and collisions are real, because Masse and Maße both map to MASSE.

Turkish: the pairs are I with dotless ı (U+0131) and i with dotted İ (U+0130). Locale-insensitive toUpperCase of i yields I; toLocaleUpperCase('tr') yields İ. Lowercasing İ without the Turkish locale produces i followed by U+0307 combining dot above, two code points, which then fails an equality test against a stored i and cannot be repaired by comparing only the first character.

Greek: sigma lowercases by position. Word-final Σ becomes ς (U+03C2), elsewhere σ (U+03C3). Uppercasing both returns Σ, so the distinction is destroyed in one direction only, and a lowercase-then-compare cycle quietly merges two spellings that a reader distinguishes.

Choose by purpose. For display, use the reader locale. For comparison, use case folding from CaseFolding.txt instead of lowercase: full folding maps into ss and the Kelvin sign U+212A to k. For stored keys, fold and store the folded value, keeping the original column for display, since folding is not reversible.

Unicode normalization: NFC, NFD and the equality trap

The equality trap in practice: café typed on macOS arrives as c a f e plus U+0301, while the same word from a Linux form arrives as c a f é with U+00E9. They render identically, they are canonically equivalent under Unicode UAX #15, and they are different strings. A unique index, a Set, a Map key or a password check compares code units, so both spellings survive and duplicates are created by the first insert.

Pick the form by what the text is for. NFC is the interchange and storage default: the W3C Character Model requires NFC in scope, and it keeps one code point per accented letter, cheaper against byte-limited columns. NFD is a manipulation form: decompose, then drop characters in general category Mn to strip accents for search, which is lossy by design and must not overwrite the input. NFKC and NFKD apply compatibility mappings and are unsafe for identifiers: U+FB01 fi becomes fi, U+00B2 ² becomes 2, U+FF21 Ａ becomes A, and U+337F, the corporation ligature, expands to four characters. Those substitutions cannot be undone, and that is the irreversible point of this section.

Order matters more than form. Fold first, then normalize, and apply the identical order to both sides: İ lowercased becomes i plus U+0307, which needs recomposition before comparison. Normalizing one side only is the most common defect in email lookups, deduplication and cache keys, and its symptom is a false mismatch rather than an exception.

NFC is idempotent, so normalizing twice changes nothing, but the input form is not recorded. Store the normalized value and keep the raw string whenever a downstream contract, such as a signature over exact bytes, depends on it.

Counting text: code points, grapheme clusters, and display width

Three different numbers all get called length. UTF-16 code units is what .length returns in JavaScript and Java. Code points is what a Unicode-aware database column and a Unicode-aware validator count. Grapheme clusters is what a person calls a character, defined by UAX #29 extended grapheme clusters. They diverge exactly where the defects live: e plus U+0301 is two units, two code points and one cluster; U+1F600 is two units and one code point; a family emoji joined by three zero-width joiners is eleven units, seven code points, one cluster and twenty-five UTF-8 bytes.

Choose by the boundary being enforced. A limit stated by a protocol or schema is usually in code points or bytes, so measure with the encoder and never with string length. A limit stated by a design is in visible characters, and truncation must therefore break on grapheme clusters: cutting mid-surrogate produces a lone surrogate that cannot be encoded as UTF-8 at all, and cutting between a zero-width joiner and the next person renders two glyphs where the author wrote one.

Column alignment needs a third measure, display width from UAX #11. East Asian Wide and Fullwidth characters occupy two terminal columns, combining marks occupy zero, and Ambiguous characters depend on context, so any hard-coded width table is wrong for some audience.

Word counts are a separate segmentation problem. Splitting on whitespace undercounts Chinese and Japanese by roughly a factor of three, since CJK prose has no spaces between words, so use UAX #29 word boundaries or a dictionary segmenter when the number is contractual. Segmenting costs time and Intl.Segmenter must exist in the runtime, so cache counts for long documents.

Splitting, sorting and deduplicating lines: what "case-insensitive" hides

Splitting looks unambiguous until real data arrives. Splitting on LF alone leaves a trailing CR on every line of a CRLF file, so keys stop matching and regex anchors stop working, while the text still looks right in a viewer, and the defect survives review. ECMAScript also treats U+2028 and U+2029 as line terminators, and legacy data carries NEL U+0085 and lone CR, so convert CRLF and lone CR to LF first and split afterwards. Decide explicitly whether a final newline yields an empty last element: POSIX text files end with one, so most inputs produce a trailing empty string that then becomes a phantom record.

Deduplication hides the same problem. A Set compares code units, so an NFC and an NFD spelling of one word both survive, as do a and A. Case-insensitive deduplication is a pipeline rather than a flag: normalize, fold, then compare, and decide which copy is kept, the first occurrence or the normalized and folded form. Trim before deduplicating only if leading space means nothing; for indentation-sensitive or fixed-width data it does not.

Sorting repeats the trap. The default comparator orders by UTF-16 unit, so Z (U+005A) precedes a (U+0061) and Zebra sorts before apple, with accented letters after all ASCII. Locale collation from UTS #10 fixes the order for human readers and costs a comparison per pair, with results that can shift between ICU versions, so never persist an order computed that way. Sorting is stable since ES2019, which makes a secondary fold key safe.

Fallback order that works: normalize, then fold, then compare; sort by a fold key and break ties by the original string; deduplicate after sorting, so first-occurrence semantics stay explicit.

Diffing text: line, word or character granularity

Granularity decides whether a diff can be read. Line diffs answer which lines changed, which is the right question for source code, configuration and CSV where structure is per line, and the only granularity that stays legible on files of thousands of lines. Its failure mode is reformatting: one run of a formatter changes every line and the diff becomes noise, so either diff a normalized copy or drop to word granularity.

Word diffs answer which sentences changed and suit prose, specifications and translation files. Character diffs suit short strings only: identifiers, hashes, version numbers, single spreadsheet cells. Run a character diff across two paragraphs and the result is a wall of single-letter edits that hides the one substantive change; run a line diff across two long paragraphs and a changed clause marks the entire line.

Cost follows the algorithm. The straightforward longest-common-subsequence diff is quadratic in time and memory, while the Myers algorithm runs in time proportional to the edit distance, which is why two near-identical large files diff instantly and two unrelated ones crawl. Character granularity multiplies the number of compared items by the average line length, so it turns a fast diff into a frozen tab.

Normalization before diffing is itself a judgement. CRLF against LF, a trailing space, NFC against NFD all produce differences that mean nothing to a reviewer, so normalize first and also run a raw diff: when the raw diff is non-empty and the normalized one is empty, the change is whitespace or encoding only, which is a finding worth reporting rather than hiding. Character granularity must be grapheme-aware, or the output splits surrogate pairs and prints broken glyphs to the reader.

A checklist before you trust a text transformation

Run the gates in order, because each one protects the next and a later gate cannot repair an earlier failure. The order is not stylistic: folding before decoding operates on mojibake, and truncating before normalizing can split the mark that normalization would have recomposed.

The last gate catches what inspection misses. Round-trip the result and compare against the input: uppercase then lowercase, fold then unfold, NFC then NFD, encode then decode, sort then restore original order. Where the round trip fails, the loss is permanent, and the operation must not be applied to data you cannot re-derive. In practice four operations fail this test almost always, which makes them the ones to review hardest: NFKC on identifiers, case mapping on stored keys, trimming on indentation-sensitive text, and truncation by code unit.

Two habits pay for themselves. Keep the original alongside every derived value, so a wrong choice is a recomputation rather than a data loss incident. And test with the datasets that break the defaults: a Turkish city name, a German street name ending in ss, a decomposed accent from a macOS form, a flag emoji, a CRLF file with a final newline. An ASCII-only test suite will pass every transformation in this guide, including the ones that destroy data.

When a transformation fails that test, the fault is usually a wrong default accepted silently, not a missing feature. Recording the decision, the form and the locale is what makes the pipeline auditable a year later.

• Keep the original bytes and a hash of them before the first transformation.
• Decode with a declared charset in fatal mode; treat U+FFFD as a stop signal.
• Convert CRLF and lone CR to LF once, and record the original style.
• Normalize to NFC before storing, comparing or diffing; reserve NFKC for display.
• Fold with an explicit locale for comparison, and keep the unfolded value for display.
• Count grapheme clusters for interface limits, code points or bytes for specification limits.
• Round-trip the result and diff it against the input before trusting the output.

Try it now