Unicode Normalization for Scraped Web Data: NFC vs NFKC

Use NFC for scraped text you plan to store, display or compare, and apply it once, right after decoding and before anything else touches the string. Reach for NFKC only on derived copies used for matching, never on the text you keep. The rest of this article shows why, with a worked example, a form-by-form comparison, and the places in a scraping pipeline where skipping the step quietly corrupts results.

TL;DR

• The same visible string can be stored as different code points, so equality checks, hashes and dedup keys silently disagree.
• Default to NFC for stored text. NFKC and NFKD discard compatibility distinctions, so they belong on throwaway match keys.
• Normalize at one point in the pipeline, after decoding and HTML cleanup, and keep the raw original if you may need it back.
• Normalize before dedup, hashing, selector matching and language detection, not after.
• Normalization fixes representation, not look-alike attacks: visually similar characters from different scripts stay different.

Table of Contents

• Why identical-looking text differs in scraped data
• A worked example
• The four forms, and what each one costs you
• Where normalization fits in a scraping pipeline
• Normalization is not a security control
• Our perspective
• Getting clean text into the pipeline
• FAQ
• Sources

Why identical-looking text differs in scraped data

A character like "é" can be one code point (U+00E9) or two: "e" followed by a combining acute accent (U+0301). Both render the same. Unicode normalization consistently picks composed or decomposed encodings so that canonically equivalent strings end up with the same binary representation.

Scraped data is where this bites, because you collect text from many sources: CMS exports, legacy-encoded pages, text pasted from word processors, macOS-originated filenames and user-generated content. Each source has its own habits. You end up with one product name stored three ways, and a == comparison that returns false for strings a human cannot tell apart.

A worked example

Take the word "café" scraped from two pages:

import unicodedata

a = "café"    # precomposed: 4 code points
b = "café"   # e + combining accent: 5 code points

print(a == b)                                  # False
print(len(a), len(b))                          # 4 5
print(unicodedata.normalize("NFC", b) == a)    # True

The strings render identically, yet they have different lengths, different hashes and different sort positions. If you build a dedup key with hash(text) before normalizing, the two pages count as distinct records. If you slice the first N characters for a preview, you can cut a combining mark off its base letter.

A second case is compatibility characters. The ligature "fi" (U+FB01), full-width digits like "１２３", and the non-breaking space all look like plain text but are different code points. NFC leaves them alone. NFKC folds them to "fi", "123" and a regular space.

The four forms, and what each one costs you

NFC and NFD are canonical forms. NFKC and NFKD remove compatibility distinctions and lose information, which is why the Unicode FAQ treats them as suited to restricted domains such as identifiers.

For general text, NFC is the safe default. The Unicode FAQ notes it is more compatible with strings converted from legacy encodings. For programming-language identifiers, UAX #31 generally prefers NFKC for case-insensitive systems and NFC for case-sensitive ones.

What NFKC can destroy in scraped data: superscripts and subscripts ("m²" becomes "m2", "H₂O" becomes "H2O"), circled numbers, fractions like "½" (which becomes "1⁄2"), and ligatures. In financial or scientific pages, that is a changed meaning, not a cosmetic tweak.

Where normalization fits in a scraping pipeline

Order matters. A workable sequence:

1. Decode bytes using the declared or detected charset.
2. Strip or convert markup to text or markdown.
3. Apply NFC to the text body.
4. Derive match keys (NFKC plus case folding plus whitespace collapse) as separate fields.
5. Hash, dedup, index and run language detection on the normalized output.

Keep the original. Converting between NFC and NFD is reversible, but a normalized string cannot always be returned to its original representation, so store the raw value explicitly if you need it for audit or source traceability.

Implementation support is broad. Per Microsoft's documentation, Win32 and .NET support all four forms, and Java ships normalization libraries. Python has unicodedata.normalize in the standard library.

Matching and selectors

The W3C recommends one consistent normalization form for all syntactic constructs in markup and content. The failure case: a CSS selector can miss an element whose class or ID uses a different normalization form than the stylesheet or your selector. If your extractor targets non-ASCII class names and returns nothing on pages that clearly have the element, compare code points before you blame the parser. A canonical tag checker is useful in the same triage, since the identifier problem often shows up alongside mismatched page variants.

URL handling has a similar trap: two URLs that differ only by normalization form of a path segment can look like one page to you and two to a crawler. Our notes on validating canonical URLs when scraping cover the dedup side of that.

Dedup and hashing

Hash after normalization, or you will store duplicates. For near-duplicate detection, hash the NFKC-folded, case-folded key rather than the stored NFC text. That gives you broad matching without altering the record you keep.

Language detection

Language detectors work on character n-grams or code point statistics. Decomposed text splits accented letters into base letters plus combining marks, which skews those statistics, and non-breaking spaces or full-width forms can add noise. Run detection on NFC text with markup already removed. We have not found an authoritative benchmark quantifying the effect, so treat this as sound practice and test it on your own corpus: detect on raw and on normalized text, and count the disagreements.

Also detect on the extracted body, not the whole page. Navigation, cookie banners and footers are often in a different language than the article. Our write-up on preserving HTML structure during content normalization for RAG explains how to separate body from boilerplate first.

Normalization is not a security control

Normalization removes representational differences. It does not make look-alikes equal: Cyrillic "а" and Latin "a" remain different characters. Unicode's security report says that for internationalized domain names, NFKC normalization and case folding prevent many spoofing attacks, but visual spoofing can still occur. Mitigation there rests with registries and user agents.

For scraped data, the practical rule is to normalize before you compare or validate, never after. UAX #31 warns that careless handling of normalization and case folding in identifier matching creates identity clashes that can compromise program logic and security. If scraped strings feed an allowlist, a permission check or a database key, normalize first, then validate.

Our perspective

Web data is a storm, and the representation of a string is one of the quietest ways it fails. Nothing throws an error. A dedup job reports fewer matches than it should, a selector returns empty, a language tag flips on a handful of pages, and you find out weeks later.

We think the right answer is the same one we apply everywhere: make the failure visible and keep the evidence. Normalize in one explicit step you own, store the raw text next to the normalized text, and log how many records changed. A pipeline that reports "4,112 of 60,000 strings changed under NFC" is a pipeline you can reason about. One that normalizes silently, or lossily, is guessing.

Gyrence's Fetch returns cleaned markdown and Extract returns typed JSON, each with structured failure responses. We do not claim to make your Unicode decisions for you, and you should not trust any vendor that says it does. Which form you pick depends on what the text is for. Treat our output as input to your own normalization step, and see how we think about structuring scraped data so agents don't have to guess.

— Glen

Getting clean text into the pipeline

If you want fetch, extraction and mapping with spending caps and typed responses, including the failure cases, Gyrence's Standard plan is $75 per month, with Growth at $299 and Scale at $549. Free and Pay-As-You-Go options are listed on the same page. LLM extraction is bundled, with no separate AI charge. To watch pages for changes, see WebDoppler.

FAQ

What is the difference between NFC and NFD?

Both are canonical forms and lose no information. NFC composes characters where a precomposed code point exists, so "e" plus an acute accent becomes "é". NFD does the reverse and splits it into base plus combining mark.

Should I use NFC or NFKC on scraped text?

Use NFC for text you store or show, since NFKC drops compatibility distinctions such as superscripts and ligatures. Use NFKC on a separate match key for dedup or search, where folding variants together is the goal. Keep the original string if you may need it later.

Does normalization change what a page says?

Canonical normalization (NFC, NFD) does not change meaning, only encoding. Compatibility forms can: "m²" becomes "m2" and "½" changes shape. That is why they should not be applied to the record you keep.

Does Unicode normalization affect language detection?

It can, because detectors count characters, and decomposed or compatibility variants change those counts. Normalize to NFC and strip markup before detecting. Then compare results on raw and normalized samples from your own data.

Sources

• Unicode FAQ: Normalization
• UAX #31: Unicode Identifiers and Syntax
• UTR #36: Unicode Security Considerations
• W3C: Normalization in HTML and CSS
• Microsoft: Using Unicode Normalization to Represent Strings

Recommended

• Engineers: Preserve HTML in Content Normalization for RAG Scraping
• Structured Scraping Output Formats: A Developer's Guide
• Markdown Isn't a Format: Structuring Scraped Web Data So Agents Don't Have to Guess