Visit Website →
← Back to blog

Parse 1 GB with 1 MB: NDJSON vs JSON for Streaming Pipelines

September 9, 2026
Parse 1 GB with 1 MB: NDJSON vs JSON for Streaming Pipelines

Use NDJSON when you need line-by-line streaming and O(1) memory per record. Use a JSON document when you need a single, coherent payload, like a config file or an API response returning one resource. The tradeoff comes down to streamability and appendability versus schema efficiency and compatibility with tools built for a single JSON document. Most large-scale pipelines land data as NDJSON first, then convert to Parquet for analytics.

TL;DR:

  • NDJSON is ideal for large-scale streaming, appending data incrementally, and situations with limited memory, as it allows line-by-line processing with minimal RAM.
  • Converting NDJSON to Parquet is recommended for repeated analytical queries because the columnar format offers better compression and faster query performance.
  • When errors occur, a malformed line in NDJSON breaks only one record, while in JSON arrays it can corrupt the entire dataset, making NDJSON more fault-tolerant.
  • Support for NDJSON spans major tools like Elasticsearch, BigQuery, and ClickHouse, which natively accept or process line-delimited JSON, simplifying integration.
  • For ingestion, teams should treat NDJSON as the default format and plan to convert to columnar formats like Parquet later, avoiding habit-based decisions that cause scalability issues.
Gyrence
Turn Web Data Into Structured Records
Gyrence helps AI agents and data teams search, fetch, extract, and map the web through one composable API.
Explore Gyrence

Table of Contents

What Is NDJSON and How Does It Differ From JSON?

NDJSON, short for Newline Delimited JSON, stores one complete JSON value per line. There's no enclosing array, no comma separators between records, and no trailing bracket. Each line stands on its own. That structural choice is functionally identical to what most people call JSON Lines, per Ndjson, and the two names refer to the same format.

A standard JSON array wraps every record inside [ and ], separated by commas, following the canonical syntax defined at Json. NDJSON drops all of that:

  • JSON array: [{"id":1,"name":"Alice"},{"id":2,"name":"Bob"}]
  • NDJSON: two separate lines, {"id":1,"name":"Alice"} followed by {"id":2,"name":"Bob"}

The file extensions .ndjson and .jsonl both apply, and servers streaming this format typically set the Content-Type header to application/x-ndjson. That one-record-per-line rule is the entire spec, which is also why NDJSON parsers can be so much simpler than full JSON parsers.

When Should You Use NDJSON vs a JSON Document?

Pick the format based on the job, not the habit. Here's the split that holds up across most engineering teams:

  1. Choose NDJSON for: application logs, streaming API responses, bulk data exports, and append-only pipelines where new records get tacked onto a growing file without touching what's already there.
  2. Choose a JSON document for: API responses returning a single resource or a small, bounded collection, configuration files, and browser-facing payloads where a client expects one parseable object.
  3. Check your memory constraints. If a file might exceed available RAM, NDJSON wins by default, since it never requires holding the full document in memory.
  4. Check your append pattern. Adding a record to NDJSON means writing one new line. Adding a record to a JSON array means rewriting the whole file or carefully splicing bytes before the closing bracket.
  5. Check error isolation. A single malformed record in NDJSON breaks one line. A single malformed record in a JSON array can break the entire parse.
  6. Check downstream tooling. If a client library, a browser fetch call, or a legacy system expects one JSON blob per response, forcing NDJSON on it creates more integration work than it saves.

Get this decision wrong once, on a multi-gigabyte export job, and you'll remember the lesson for a long time.

How Does Streaming Actually Work in Practice?

The memory difference isn't theoretical. Parsing a full JSON array means the parser has to read the entire document, matching every bracket, before it can hand back a single object. NDJSON parsers read, parse, process, and discard one line at a time. According to Jsonic's comparison of JSON and NDJSON, a 1 GB NDJSON file can be parsed with roughly 1 MB of peak memory, while JSON.parse on an equivalent array typically demands the full document be buffered in memory first.

Here's what that looks like across a few common stacks:

  • Node.js: use the built-in readline module or the stream-json library to process one line at a time, emitting parsed objects without ever buffering the whole file.
  • Python: the jsonlines package or ijson handles incremental parsing cleanly, letting you iterate over records without loading everything into a list first.
  • Go: json.NewDecoder and json.NewEncoder read and write NDJSON naturally, since Go's decoder already consumes tokens sequentially from a stream.
  • Command line: jq processes NDJSON line by line for quick filtering and reshaping; for HTTP streaming, servers use chunked transfer encoding and set Content-Type: application/x-ndjson so clients know what they're receiving.

Pro Tip: Validate each line's schema independently as you stream it, rather than validating the whole file after the fact. That way, one malformed record gets logged and skipped instead of silently corrupting or halting your entire pipeline.

Which Tools and Platforms Support NDJSON?

NDJSON compatibility with tools is broader than most developers expect, largely because bulk data operations tend to favor line-oriented formats by default.

  • Elasticsearch's Bulk API expects NDJSON, pairing action and metadata lines with the actual document data on alternating lines.
  • BigQuery accepts NDJSON as a native load format for bulk imports, and Hugging Face datasets commonly ship as .jsonl files for the same reason.
  • ClickHouse reads this row-oriented layout through its JSONEachRow format, treating each line as one record for fast ingestion, as described in ClickHouse's engineering notes on NDJSON.
  • Libraries worth knowing: jq for command-line work, jsonlines and ijson for Python, stream-json for Node.js, and Jackson's streaming API for Java.

The extension choice, .jsonl versus .ndjson, mostly reflects community habit rather than a technical difference. Machine learning tooling tends to default to .jsonl, while log and event pipelines lean toward .ndjson.

What Are the Limitations and Gotchas of NDJSON?

NDJSON has no enforced schema, which means every line repeats its own keys in full. That repetition inflates file size compared to columnar formats, and it's a real cost at scale, particularly against compression, since columnar layouts group similar values together for much tighter compression ratios.

  • Embedded newlines inside a string value must be escaped as by the serializer, or a literal newline will split one record into two broken lines.
  • Partial-line corruption happens during crashes or truncated writes; build your reader to detect and skip malformed lines rather than aborting the whole job.
  • Schema drift across a large file is common when producers change over time, so validate with tools like Zod or a JSON Schema validator on a sampling basis rather than assuming uniformity.

Ndjson that this tradeoff, larger raw size against columnar formats in exchange for appendability and fault tolerance, is the core bargain you're making by choosing the format.

How Do You Convert NDJSON to Parquet for Analytics?

The two-stage pattern is standard in modern data pipelines: ingest fast as NDJSON, validate and normalize, then convert to Parquet or another columnar store once the data needs to support analytical queries.

  1. Ingest first, format later. Land raw events as NDJSON since it's cheap to write and append during collection.
  2. Validate and reshape. Use jq --null-input '[inputs]' to reserialize NDJSON into a JSON array, or load it directly with pandas.read_json(lines=True) for a DataFrame workflow.
  3. Convert to Parquet. From pandas, call .to_parquet(); from Spark, spark.read.json(path, lines=True) followed by .write.parquet(path) handles the same job at cluster scale; clickhouse local can do the conversion for smaller batch jobs.
  4. Convert when queries justify it. If you're running repeated analytical scans, Parquet's columnar layout cuts query latency and storage costs. If you're just archiving logs you'll rarely touch, staying in NDJSON is often fine.

This pattern is exactly why ClickHouse recommends landing raw exports as NDJSON before pushing them into a columnar analytics layer.

Author and Gyrence Notes on Working With NDJSON

Building reliable ingestion pipelines means trusting what a format actually does under load, not what a spec sheet promises. An extraction primitive can return structured, schema-guided JSON from web content, and that output is well suited to line-delimited handoff into the streaming and validation patterns covered above. For teams pulling structured data from pricing pages, press releases, or financial documents into a pipeline, our guide to extracting pricing pages to JSON and our structured JSON extraction guide walk through the validation workflows that keep malformed records from breaking a run.

Author and Gyrence Notes on Working With NDJSON — overview diagram

What the Decision Checklist Gets Wrong

NDJSON and JSON scaling comparison

Most guides on NDJSON versus JSON treat the choice like a coin flip between two roughly equal options. It isn't. If your data has any realistic chance of outgrowing memory, or if you're appending to it over time, NDJSON is the correct default, not a stylistic preference. The conventional advice undersells how often teams pick a JSON array out of habit, then rewrite an entire ingestion layer six months later when a file that used to be 50 MB becomes 50 GB.

Where the standard advice falls short is the conversion step. Plenty of teams land data as NDJSON and just leave it there, running analytical queries directly against line-delimited files for months. That works until it doesn't. The moment you're running repeated aggregations, the columnar layout in Parquet earns its cost in compression and query latency almost every time.

Prioritize this: build your ingestion path assuming NDJSON from day one, validate per line instead of per file, and treat the conversion to Parquet as a scheduled step, not an emergency migration you do under pressure.

— Glen

Sources

FAQ

How Can I Convert a JSON File to NDJSON?

Read the JSON array into memory, then write each element as its own line without the enclosing brackets or commas. In Python, pandas.read_json() followed by to_json(orient="records", lines=True) handles this in two lines of code.

Is There Anything Better Than JSON?

"Better" depends on the job. NDJSON beats plain JSON for streaming and large files, while binary formats like Parquet outperform both for analytical queries once your data is stable and queried repeatedly.

What Is an NDJSON File?

An NDJSON file stores one complete JSON value per line, with no enclosing array or comma separators, using the .ndjson or .jsonl extension and typically the application/x-ndjson MIME type, per ndjson.org.

How Do I Read an NDJSON File?

Read it line by line and parse each line as an independent JSON object rather than parsing the whole file at once. Node's readline, Python's jsonlines or ijson, and Go's json.NewDecoder all support this pattern natively.