JSON: the decision path from validation and formatting to conversion

Four kinds of JSON problems, and why the fix differs

The diagnosis picks the tool, and the tool decides which evidence survives. Four classes of problem look identical in a code review, need different evidence, and only one of them is repaired by re-indenting text. Name the class before opening a formatter, because the first operation you choose is irreversible in three of the four cases.

Syntax or encoding. The text never becomes a value. JSON.parse throws, or a decoder quietly substitutes U+FFFD and the parser then throws at a position unrelated to the damage.

Semantics. Parsing succeeds and the values are still wrong. An identifier above 2^53 lands on a neighbouring integer, two members with the same name collapse into the last one, and a null written by one system means missing while the same token from another system means empty.

Scale. The document is valid and the process still exhausts memory, because one array of ten million records has to exist as text and as an object tree before the first record is read.

Representation. The JSON is correct and the consumer wants XML, YAML, CSV or HTML form fields. The question becomes which information survives the crossing and which is silently dropped.

The classes differ in cost and in reversibility. Formatting answers only the first class, and it destroys the evidence for the other three: it rewrites duplicate keys, turns -0 into 0, converts NaN into null and drops members whose value is undefined. Capture the original bytes first, and keep a hash of them; that is the only way to prove later whether the formatter or the producer changed the data.

Validation: why the reported error position misleads you

A position in a V8 JSON error is a zero-based offset in UTF-16 code units, not a byte offset and not a line and column. Parse {"中文": 1, } and the message reports position 10, while the character it refers to begins at byte 13: each Chinese character is three UTF-8 bytes but a single UTF-16 unit, so the counts diverge as soon as the document is not pure ASCII. An emoji counts as two units and four bytes. Slicing the input at that offset lands in the wrong place, hence the blame on the parser.

The grammar being enforced is small and closed. RFC 8259 admits no comments, no trailing commas, no unquoted member names, no NaN or Infinity and no hexadecimal numbers. Encoding is UTF-8; section 8.1 says a byte order mark must not be added and parsers may ignore one. V8 does not, so a BOM yields an error containing an invisible U+FEFF.

Two failures are reported as syntax errors and are nothing of the kind. A body truncated by an interrupted stream fails at the end of the buffer, so the offset points past the last byte received, not at the producer bug. A body decoded with a non-fatal TextDecoder has each invalid sequence replaced by U+FFFD, so the parser complains about a character that never existed in the source; new TextDecoder("utf-8", { fatal: true }) raises a TypeError at decode time instead.

Duplicate names pass silently: RFC 8259 section 4 says names SHOULD be unique, and JSON.parse keeps the last occurrence. Validation has layers people conflate: syntax accepts a document, a schema checks its shape, and range rules are a third layer no parser checks.

Formatting: indentation, key order, and number representation

Indentation is the least interesting part of formatting and the only reversible one. JSON.stringify(value, null, n) clamps the indent to ten characters, so asking for twelve and asking for twenty both produce ten; a string argument is truncated to its first ten characters, and a tab counts as one. Whitespace between tokens carries no meaning in RFC 8259, which makes two spaces versus four a style decision and never a data decision.

Re-serialising is where data is lost, and every loss is silent. Members whose value is undefined, a function or a symbol are omitted from objects and become null inside arrays; -0 prints as 0; NaN and both infinities print as null; a BigInt throws instead of printing. Integers above 2^53 have already been rounded before serialisation, so 9007199254740993 arrives as 9007199254740992 and nothing in the output records that any of this happened.

Key order does not behave the way most people assume. JSON.stringify emits array-index-like keys in ascending numeric order first, then the remaining string keys in insertion order, so an object written as {b:1,"2":2,a:3,"1":4} serialises with 1, 2, b, a in that sequence. A replacer array fixes an order but applies at one level only, and RFC 8259 defines an object as an unordered collection, so a consumer that depends on member order depends on an implementation detail.

Numbers print through the ECMAScript shortest round-trip algorithm: 0.1+0.2 becomes 0.30000000000000004, 1e20 stays decimal while 1e21 becomes 1e+21, and magnitudes below 1e-6 go exponential. If a diff must be limited to whitespace, re-indent the source text with a tokeniser instead of parsing and printing: only that route preserves the spelling of every number and every duplicate key.

Querying structure: JSON Pointer, JSONPath, or jq

Three query mechanisms answer three different questions, and mixing them up produces code that reimplements a selector badly, with a bug on the payload shape nobody tested.

JSON Pointer (RFC 6901) names exactly one location: tokens separated by /, where ~0 stands for a tilde and ~1 for a slash. Decoding order is not optional, because the input ~01 must become the literal key ~1, which means expanding ~1 before ~0. Choose a pointer when the path is fixed in code, when it arrives from user input and can be checked against a schema, or when a patch needs an address: JSON Patch (RFC 6902) is built on it.

JSONPath (RFC 9535, 2024) selects a set rather than a location: wildcards, descendant segments, slices, filters. It is a standard with a defined result model, so a query written against RFC 9535 behaves the same in more than one language, which was never true of the pre-standard dialects. It is read-only, its filter grammar is closed, with no arithmetic or user functions.

jq is a programming language, not a query standard. It earns its place when the task is shaping rather than selecting: joining documents, aggregating, emitting another structure. The costs are portability, since a filter lives in a shell script and cannot move to a browser, and precision, since numbers jq touches pass through an IEEE 754 double.

Decide by the shape of the answer. One location with a known path is a pointer; a set selected by predicates is JSONPath; a new document is jq or code. To change a document in place, none of the three fits, since a query returns values, not references.

Large documents: arrays versus JSON Lines

The choice between one array and JSON Lines decides whether the file can be read at all inside a memory budget. A single array of N records must arrive as one string before parsing starts, and the parsed tree is larger than the string, since every object has its own header. V8 caps a string at 536,870,888 characters, so a larger document cannot exist as a JavaScript string; that ceiling is hard, not a tuning problem.

JSON Lines, described by jsonlines.org, writes one complete JSON value per line, terminated by a newline, encoded in UTF-8 without a byte order mark. Each line is an independent JSON.parse call, so a reader can consume a record and release it, and a producer can append without touching existing bytes. Append cost stops being linear in file length, and a malformed line is isolatable while every other record still parses.

The costs are structural and they are the reason arrays still exist. There is no document-level wrapper, so totals, schema version and checksums must travel in a sidecar file or repeat in every record. Nesting is limited by the line, since a record cannot be split across lines, and pretty-printing it with newlines destroys the format: a log pipeline must never pass JSON Lines through a formatter that re-indents.

Choose the array when the document is one published object, when the consumer needs the collection atomically, or when nesting goes deeper than a record. Choose JSON Lines when records arrive over time, when the file exceeds memory, when processing splits by line across workers, or when appends matter. An array converts to JSON Lines as a streaming read; converting back is a full rewrite.

Converting representation: where JSON, XML, YAML and CSV lose information

A representation change is a lossy projection, and the target model decides what is lost. JSON has six types: null, boolean, number, string, array and object, with number fixed at IEEE 754 double precision and member names fixed at strings. Anything outside that set must be encoded by convention: instants as RFC 3339 strings, binary as Base64 from RFC 4648, maps with non-string keys as arrays of pairs.

XML adds namespaces, attributes, mixed content and significant element order, and it has no number type: every value is text. XML to JSON therefore needs invented conventions, typically an @ attribute prefix and a reserved member for character data, and those conventions are not standard, so two converters produce two shapes for one document. Round-tripping also collapses a namespace prefix into whatever the converter chose.

YAML is the closest fit and still not free. Anchors and aliases express shared references, and a node that appears twice becomes two independent copies in JSON; a cyclic YAML cannot be represented at all, because JSON.stringify throws on a cycle. The YAML 1.1 boolean ambiguity, where yes and on parse as true, was removed in the YAML 1.2 core schema but survives in old documents, so a field meant to be a string arrives as a boolean.

CSV (RFC 4180) is text only and carries no types: an empty field is indistinguishable from a missing one, and type inference is a guess that eats leading zeros in postal codes and digits from identifiers. Quoting protects delimiters, newlines and doubled quotes, never types. Base64 inflates a payload to ceil(n/3)*4 characters, a third larger, and Base32 to ceil(n/5)*8, sixty percent larger, for an alphabet that survives case-insensitive channels.

A debugging order that actually works

A debugging order beats a debugging toolkit, because each step has to be completed before the next one can be trusted. Work from bytes inward, and stop at the first layer that misbehaves instead of inspecting all of them at once.

Start with the bytes: confirm UTF-8 with a fatal decoder and look for a BOM before any string processing, since a replacement character introduced here will be blamed on the grammar two steps later. Then the grammar, with a strict parser, reading the reported offset as UTF-16 units. Then the shape, against a schema rather than by eye, because a missing field and a null field look identical in a console log. Then the values, where the ranked suspects are integers above 2^53, duplicated keys and null versus absent. Then the size, where the only question is whether the whole document ever has to exist. Representation comes last, because it is the single step that is supposed to lose information.

Two rules keep the order effective. Never re-serialise before the original bytes are captured and hashed, because parsing and printing is the irreversible move in every failure listed above. And minimise before theorising: delete half the document and retry, repeatedly, which converges in a logarithmic number of runs on the smallest input that still fails.

Finally, the parser that matters is the consumer, not yours. Python json.loads accepts NaN and Infinity by default, while RFC 8259 defines no such literals and JSON.parse rejects them, so a payload that passes in a data script can fail in a browser. Reproduce the failure in the runtime that will actually read the file, then fix the producer rather than the copy.

Try it now