DEVTOOLBOX / GUIDES this guide
Character encodings and transfer encodings: a decision tree from charset to Base64
Most encoding bugs are layer confusions, not hard problems. This guide separates the character repertoire, the byte encoding and the transfer encoding, then pins down the boundaries you can verify: UTF-8 at one to four bytes, Base64 at four thirds of the payload, JSON escapes under RFC 8259, URL rules under WHATWG. Each section ends with a criterion you can apply and the point past which the damage cannot be undone.
Three layers people confuse: repertoire, character encoding, transfer encoding
The three layers behind almost every encoding failure all get called "the encoding". The repertoire is the set of characters a system handles: Unicode assigns code points from U+0000 to U+10FFFF, and a repertoire question is whether a character exists at all. The character encoding is the byte format for those code points: UTF-8, UTF-16LE, GBK. The transfer encoding is a reversible wrapper that carries bytes through a channel tolerating only ASCII or only syntax-safe characters: Base64 under RFC 4648, percent-encoding under WHATWG URL, HTML entities, JSON \uXXXX escapes.
Locate the failure by what changed. If the bytes are identical and only glyphs differ, the bug is at layer two: a decoder was handed the wrong charset. If characters already read as ? or U+FFFD, layer one failed earlier, because the target repertoire could not represent them. If the payload is byte-identical and a parser still rejects it, a wrapper was applied twice, not at all, or in the wrong alphabet.
Two layers are reversible and one is not. Base64 decode, percent-decode and entity decode round-trip exactly: RFC 4648 and WHATWG URL define them as bijections over a fixed alphabet. Charset conversion is not, because encoding into a repertoire that lacks the character emits a substitute, U+FFFD or a plain question mark in many legacy encoders, and the code point is gone with no inverse. That is the irreversible point, crossed the moment damaged text reaches a file or a database column. Cost runs the other way: UTF-8 spends one to four bytes per code point, UTF-32 four always, and Base64 adds thirty-three percent.
| Layer | Question it answers | Typical symptom | Fix direction |
|---|---|---|---|
| Repertoire | Does this character exist at all? | Question marks or U+FFFD in stored text | Recover source bytes; substitution is already lossy |
| Character encoding | Which bytes represent the code point? | Same bytes, wrong glyphs | Re-decode with the correct charset |
| Transfer encoding | How do bytes survive a text-only channel? | Parser rejects payload, or layers stack up | Decode exactly once, with the right alphabet |
Diagnose top-down: a layer-three fix applied to a layer-one failure leaves the damage untouched.
Text to bytes: the actual UTF-8 boundaries
UTF-8, specified in RFC 3629 and required by RFC 8259 for JSON exchanged between systems, spends one byte from U+0000 to U+007F, two from U+0080 to U+07FF, three from U+0800 to U+FFFF and four from U+10000 to U+10FFFF. The three-byte band holds most non-Latin text: U+4E2D encodes as E4 B8 AD, so a CJK character costs three bytes. Emoji sit in the four-byte band, U+1F600 as F0 9F 98 80. Two numbers prevent most bad size estimates, and both are worth measuring.
JavaScript adds a boundary of its own. Strings are sequences of UTF-16 code units, so a code point above U+FFFF is stored as a surrogate pair and a single emoji has length 2, not 1. Slicing at the wrong index produces a lone surrogate, which is not a scalar value and cannot be encoded as UTF-8 at all. Grapheme clusters are a third boundary: a family emoji joined with zero-width joiners is one glyph and eighteen bytes. Measure bytes with a real encoder, never with string length.
The decisive judgement is how validation failure gets reported. TextDecoder in its default mode never throws: it substitutes U+FFFD for each malformed sequence and returns a string, so a corrupted decode looks like success. Passing { fatal: true } turns the same input into a TypeError. Choose fatal whenever decoded text will be persisted, replacement only when showing best-effort text to a human. Truncation is the usual cut: a three-byte character split across a buffer boundary is no longer valid UTF-8, and ext4 caps a filename at 255 bytes, which is 85 CJK characters.
| Encoding | A | CJK character | Emoji U+1F600 |
|---|---|---|---|
| UTF-8 | 1 byte | 3 bytes | 4 bytes |
| UTF-16LE | 2 bytes | 2 bytes | 4 bytes |
| UTF-32LE | 4 bytes | 4 bytes | 4 bytes |
| GBK | 1 byte | 2 bytes | Not representable |
Byte cost per code point. GBK cannot represent U+1F600, so a transcode substitutes and does not round-trip.
Bytes to safe text: Base64, Base32, Hex and Base64URL
RFC 4648 defines the family in one document: Base64 in section 4 over A to Z, a to z, 0 to 9, plus and slash; Base64URL in section 5 replacing those two with minus and underscore; Base32 in section 6 over A to Z and 2 to 7; base16 in section 8. Sizes follow from radix and block size. Base64 emits four characters per three input bytes, so 3000 bytes become 4000 characters, one third larger. Hex emits two characters per byte, doubling the payload. Base32 emits eight per five bytes, a sixty percent expansion.
Pick by channel, not by habit. When the consumer is a URL component or a file name, Base64URL wins, because plus and slash are percent-encoded in queries and slash separates path segments; the URL-safe alphabet removes both, and dropping padding removes the equals sign that query parsers split on. When a human has to read or diff bytes, hex wins despite doubling the size, because each byte stays a fixed two-character window. When the channel folds case or a person reads the value aloud, Base32 wins: RFC 4648 chose A to Z and 2 to 7 to avoid 0 and O, 1 and I.
Two properties get forgotten. Base64 carries no integrity check: one altered character yields different bytes with no error raised, which is why payloads that matter travel with a hash beside them. RFC 4648 section 3.2 requires decoders to reject characters outside the alphabet, and section 3.3 requires padding unless the referring specification says otherwise. Memory is the hidden cost: a 100 MB upload becomes 133 MB on the wire and usually another 133 MB as a string before decoding.
| Scheme | Alphabet tail | Size for n bytes | Reach for it when |
|---|---|---|---|
| Base64 | Plus and slash, equals padding | ceil(n/3) x 4 | Binary inside JSON or a data URL |
| Base64URL | Minus and underscore, padding optional | ceil(n/3) x 4 | Binary inside a query or file name |
| Base32 | A to Z and 2 to 7 | ceil(n/5) x 8 | Case-insensitive channel, human transcription |
| Hex | 0 to 9 and a to f | 2n | Byte boundaries must stay visible |
Expansion is a property of the radix, so it is settled before you look at the payload.
URLs: reserved characters, plus versus percent-20, path versus query
WHATWG URL percent-encodes anything outside its ASCII safe set by taking the UTF-8 bytes of the character and writing each as a percent sign plus two hex digits, so a CJK character appears as three escapes. The reserved set carries syntax: colon, slash, question mark, hash, brackets, at sign, plus sub-delimiters including ampersand, equals, plus, semicolon and comma. Encoding a reserved character demotes it from syntax to data; failing to encode one promotes data into syntax.
Space is the trap, and it differs by component. In a query string the form-urlencoded parser reads plus as a space, and that is the parser WHATWG URL applies to the search string, so a=1+2 yields 1 2. In a path, plus stays a literal plus. %20 is the only spelling that means space in every component, which makes it the safe default. Building with URLSearchParams rather than concatenation also settles which alphabet the encoder chose, which is where hand-built strings go wrong.
Path and query diverge after decoding. WHATWG URL keeps %2F encoded inside pathname, but a framework that decodes the path before routing sees an extra separator and matches a different route, the mechanism behind path-traversal bypasses. The irreversible point is decode order: decoding twice turns the escape for a literal percent sign into a space, so a safely escaped value becomes syntax. Decode once, where the framework hands you the raw string. Hosts follow another rule set: non-ASCII labels travel as punycode under RFC 3492, so a Chinese domain is xn--fiq228c.com on the wire.
| Component | Space encoding | Plus means | Watch out for |
|---|---|---|---|
| Path | %20 | A literal plus | %2F stays encoded, but routers may decode it |
| Query, form-urlencoded | %20 or plus | A space | Silent corruption in both directions |
| Query, generic RFC 3986 | %20 | A literal plus, unless form parsing applies | Ambiguous in practice, so avoid plus |
| Fragment | %20 | A literal plus | Never reaches the server |
| Form body | Plus, or %20 with multipart | A space | Shares the parser with the query string |
One physical string, two parsers. Decide per component which one will read it.
Escaping inside HTML, JSON, and logs
HTML gives five characters meaning: ampersand, less-than, greater-than, and the two quote characters. In a text node only ampersand and less-than can begin markup; inside a quoted attribute the quote you chose ends the value early. Entity decoding is not contextual: the escaped form of a less-than sign decodes to ampersand plus l-t, and decoding that output again yields a live tag. Escape once, at the point of output, and treat any second pass as a bug.
Raw-text elements are a separate context. Inside a script or style element the parser does not decode entities at all, and the closing script sequence terminates the block even inside a JavaScript string, so JSON embedded in a page needs HTML-context escaping too. U+2028 and U+2029 are legal unescaped in JSON strings under RFC 8259 and ECMA-404, but engines before ES2019 treated them as line terminators, breaking the surrounding script rather than the JSON.
JSON escaping is narrower. RFC 8259 section 7 requires escaping only the quote, the reverse solidus and U+0000 to U+001F. Serialization adds a lossy transform that escaping cannot repair: JSON.stringify drops properties holding undefined, a function or a symbol, turns array holes into null, converts NaN and Infinity to null, writes -0 as 0, throws on a BigInt, and loses integer precision past 2^53.
Logs are line-oriented, so a newline inside a user-supplied field forges a second record: log injection, not a display bug. NDJSON avoids that structurally, because JSON.stringify already escapes newlines and each line parses alone with one JSON.parse call, so one bad record does not destroy the file. A pretty-printed document cannot be read that way.
| Context | Must escape | Leave alone | Failure when wrong |
|---|---|---|---|
| HTML text node | Ampersand and less-than | Greater-than and quotes | Live tag built from user text |
| Quoted HTML attribute | Ampersand, less-than, the quote used | The other quote character | Attribute break-out |
| Script or style block | Closing script sequence, line separators | HTML entities, which are not decoded | Script break-out |
| JSON string | Quote, reverse solidus, U+0000 to U+001F | Slash and U+2028 | Parse error, or a changed value |
| NDJSON line | Newline, already handled by stringify | Record separators added by hand | Two records from one write |
| Log line | Newline and control characters | Nothing | Forged log entry |
Each context has its own grammar. Escaping for the wrong one produces output that looks correct and parses wrong.
Diagnosing mojibake in three steps
Step one is to read bytes, not glyphs. Open the payload in a hex view. A leading EF BB BF is a UTF-8 byte order mark; FF FE and FE FF mark UTF-16 little and big endian, and RFC 8259 section 8.1 forbids JSON producers from adding a BOM, so finding one is already a producer bug. Bytes from C2 to F4 followed by 80 to BF continuation bytes are UTF-8. A run of 3F bytes means the loss happened at encode time: question marks were substituted, and no decoder reverses that.
Step two is to test a hypothesis by decoding with { fatal: true }. If it throws, the bytes are not UTF-8, so the candidates are GB18030, Big5, Shift_JIS or windows-1252. If it decodes but the text is nonsense, suspect a UTF-8 payload read as ISO-8859-1: three-byte CJK sequences turn into two visibly Latin characters, so a run of accented letters is usually UTF-8 in disguise. Re-encoding to ISO-8859-1 and decoding as UTF-8 recovers the original exactly, because ISO-8859-1 maps all 256 byte values invertibly; windows-1252 does not, since bytes 81, 8D, 8F, 90 and 9D are undefined there.
Step three is to audit what survived. After recovery, count U+FFFD: each occurrence marks a sequence some decoder already replaced, and the code point behind it cannot be reconstructed, because replacement is many-to-one with no inverse. A repair that lowers the count while keeping the text readable is usually right; one that keeps the count and shifts characters is a normalization side effect in disguise. NFKC is not a repair: it rewrites U+2460 to the digit 1 and the fi ligature to two letters, irreversibly.
| Bytes seen | Likely encoding | Verdict |
|---|---|---|
| EF BB BF at offset 0 | UTF-8 with BOM | Strip once at ingest, never write it back |
| C3 A4, or E4 B8 AD | UTF-8 | Bytes are valid; the display decoder is wrong |
| FF FE or FE FF | UTF-16LE or UTF-16BE | Check byte pairing before decoding |
| 3F repeated | Already substituted | Unrecoverable, restore from source |
| Two-byte pairs in the A1 to FE range | GBK, GB18030 or Big5 | Try GB18030 first, it is a superset |
Read the first sixteen bytes and most diagnoses end there.
Tool selection at a glance
Choose the layer before the tool. A question about bytes starts with hex, a question about a text-safe channel starts with Base64 or URL encoding, and a question about a document starts with the syntax escaper for that document. Most wasted effort comes from applying a layer-three tool to a layer-two problem: Base64-encoding a mojibake string preserves the damage and hides it, and percent-encoding invalid JSON does not make it valid.
Encoding is not encryption: Base64 is a keyless bijection from RFC 4648, a decoder is a few lines in any language, and secrecy needs real cryptography. Escaping is not sanitization either: HTML-entity encoding does nothing for a SQL statement, a shell command or a regular expression, because each has its own grammar. Validate on input, then escape for the grammar you are entering.
RFC 6901 JSON Pointer and RFC 9535 JSONPath address a location inside a document rather than encode its bytes, and both escape differently from JSON strings: JSON Pointer writes a tilde as ~0 and a slash as ~1, a namespace that misleads precisely because it looks like no escaping at all. Use pointer semantics for one deterministic path in configuration or patch operations, JSONPath for a query over many nodes. Confusing them yields selectors that address the wrong node without failing.
Verify conversions instead of trusting them. Round-trip the payload and compare byte lengths; if Base64 output length is not a multiple of four, or a decoded string contains U+FFFD, one end used the wrong alphabet. Spot checks are cheap in Node: a CJK character is three bytes and 5Lit in Base64, an emoji four bytes and eight characters.
| Situation | Use | Avoid | Why |
|---|---|---|---|
| Binary inside JSON or a data URL | base64 | percent-encoding | JSON strings accept the alphabet unchanged |
| Binary inside a query or file name | base64url | base64 with padding | Plus, slash and equals break parsers |
| Opaque identifier read by a human | hex | base64 | Two characters per byte, boundaries visible |
| Case-insensitive storage | base32 | base64 | Alphabet has no case pairs |
| Non-ASCII hostname | punycode | percent-encoding the host | RFC 3492 defines the wire form |
| JSON embedded in an HTML page | json-escape plus HTML escaping | raw JSON | Script termination and entity contexts differ |
| Addressing a node inside JSON | json-pointer | splitting a path string by hand | Tilde and slash have their own escapes |
The left column is a question about a channel, and the channel decides the alphabet.
Frequently asked questions
- Is Base64 a form of encryption?
- No. RFC 4648 defines Base64 as a reversible mapping between byte triples and four-character blocks, with no key and no secret. Anyone holding the string can decode it, and the padding character even hints at the original length. Use it only to move bytes through a text-only channel. If the requirement is confidentiality, that is a cryptographic primitive, and the encoding layer neither provides nor weakens it.
- Why does JSON.parse fail with a position number that points at the wrong character?
- The position in a V8 error message is a zero-based offset in UTF-16 code units, not a column count and not a byte count. Multi-byte characters before the error shift it away from what an editor shows, and a BOM at offset 0 makes the very first token fail. Re-open the file as UTF-8 without a BOM, then look at the reported offset rather than at the visible column.
- Is UTF-8 always smaller than UTF-16?
- For Latin text, yes: one byte per character against two. For CJK it is the opposite, three bytes against two, and for emoji both cost four. UTF-16 also needs a byte order mark or an out-of-band agreement, which UTF-8 does not. Size is therefore a weak argument; choose UTF-8 because RFC 8259 requires it for JSON interchange and because every parser accepts it without configuration.
- Why do I see plus signs in one URL and percent-20 in another?
- The query string is parsed with the form-urlencoded rules, where plus means space, so encoders that target forms emit plus. Percent-20 means space in every component, including the path, where plus stays literal. If one string has to be valid in both places, use percent-20. If you control neither end, encode the space once and check what the receiver actually logged.
- Can mojibake always be reversed?
- Only while the original bytes are intact. If a decoder replaced bytes with U+FFFD, or a legacy encoder wrote question marks, that mapping is many-to-one and no inverse exists. Recovery works when the wrong step was a lossless one such as ISO-8859-1, because all 256 byte values map back uniquely. Count replacement characters before you attempt a repair and treat any that remain as permanent loss.