Character Encoding, Demystified: ASCII, Unicode, and UTF-8 Without the Confusion

Every developer eventually meets mojibake: 中文 where Chinese should be, � where a character should be, or a string that looks identical but refuses to match. Encoding bugs feel mystical only because the three concepts involved — characters, code points, and bytes — are usually explained jumbled together. Separated, they are simple.

Three different things we call “text”

1. A character set (ASCII, Unicode) is a numbered catalog of characters. ASCII assigns 0–127 to Latin letters, digits, and controls. Unicode assigns code points — written U+4E2D for 中 — to over 150,000 characters today.
2. An encoding (UTF-8, UTF-16, GBK) is the rule for turning those code points into bytes.
3. Bytes are what files, network sockets, and memory actually hold.

Mojibake is almost always step 2 and step 3 disagreeing: bytes written under one encoding read under another. 中文 is literally the UTF-8 bytes of 中文 misread as Latin-1.

Why UTF-8 won

UTF-8’s design is the reason it became the universal encoding:

• ASCII compatibility — code points 0–127 encode as exactly one byte with the same value. Any valid ASCII file is valid UTF-8, byte for byte. This made migration incremental.
• Self-synchronization — the first byte of every sequence announces its length (1110xxxx means a 3-byte sequence follows), so a reader can resync after corruption and no character’s bytes can be mistaken for another’s.
• No byte-order ambiguity — unlike UTF-16, UTF-8 has no endianness, so there is no BOM dilemma on streams.

The cost: code points above U+007F take 2–4 bytes, which is why a 2-character Chinese string is 6 bytes in UTF-8 — and why strlen() answers surprised a generation of C programmers.

The surrogate pair wrinkle (and why emojis are two “\u” escapes)

Many programming languages (JavaScript, Java, C#) represent strings as UTF-16 code units. Code points above U+FFFF — most emoji — do not fit in one unit, so they are stored as a surrogate pair: 😀 (U+1F600) is \uD83D\uDE00. Consequences you have likely met: "😀".length === 2 in JavaScript, and reversing a string splits emojis in half. Our Unicode escape tool shows the code point, the UTF-16 escapes, and the UTF-8 bytes side by side, so the three layers stop blurring together.

The debugging method that always works

Encoding bugs surrender to one discipline: stop looking at rendered text and look at bytes.

1. Hex-dump the suspicious string (our hex tool shows each byte).
2. Compare against the expected bytes for the content. 中 in UTF-8 must be E4 B8 AD — if you see D6 D0, the bytes are GBK.
3. Fix at the boundary where the wrong encoding entered: file save dialog, Content-Type charset header, database connection charset, or a decoder call.

The � replacement character specifically means “these bytes are not valid UTF-8” — it is your decoder reporting damage, and the hex dump tells you where.

A field guide to prevention

• Declare charset=utf-8 in HTTP headers and HTML meta, and make database connections UTF-8 explicitly (utf8mb4 in MySQL — the older utf8 is only 3-byte and drops emoji).
• Normalize once: Unicode has multiple encodings for visually identical text (é as one code point vs e + combining accent), so normalize inputs (NFC is the usual choice) before comparing or hashing.
• Treat encoding as a property of bytes, never of “strings in memory” — every I/O boundary (file read, socket, HTTP) names its encoding explicitly.

With the model in place, the tools turn debugging into verification: ASCII table for the 0–127 layer, Unicode escapes for what a JS engine really holds, and hex for what the bytes really are.