CKTIS / JavaScript
UTF-16 code units, surrogate pairs, and Unicode normalization
Distinguish code units, code points, and grapheme clusters. Check whether a string is well-formed UTF-16 and choose a normalization form for text comparison.
Version: ECMAScript 2024 and the current web platform; runtime support varies
Syntax
s.length
s[i]
s.slice(start, end)
s.charCodeAt(i)
s.codePointAt(i);
[...s]
s.isWellFormed()
s.toWellFormed()
s.normalize()
s.normalize(form) // "NFC", "NFD", "NFKC", or "NFKD"
a.normalize("NFC") === b.normalize("NFC")
new Intl.Segmenter(undefined, { granularity: "grapheme" }).segment(s)A JavaScript string is a sequence of 16-bit UTF-16 code units. length, s[i], and slice() work in code units, not bytes or user-perceived characters. Code points from U+10000 to U+10FFFF require a surrogate pair: a high surrogate (0xD800–0xDBFF) immediately followed by a low surrogate (0xDC00–0xDFFF). slice() can cut a pair in half.
charCodeAt(i) returns one code unit’s value. codePointAt(i) reads the whole pair only if a pair starts at the code-unit index i; otherwise it returns the single unit’s value, including when i points to a low surrogate. Out of bounds, they return NaN and undefined, respectively. for…of and [...s] keep valid pairs together, but iterate by code point, not by grapheme cluster. For clusters such as a letter plus a combining mark, use Intl.Segmenter with granularity: "grapheme" where supported.
A string can contain lone surrogates. isWellFormed() detects them by returning false, and toWellFormed() returns a string with each replaced by U+FFFD. Both methods are part of ECMAScript 2024; check support in your target runtime. Repair is lossy and is not normalization.
normalize() returns a string without changing the original. NFD performs canonical decomposition; NFC follows it with canonical composition where possible. NFKD performs compatibility decomposition; NFKC follows it with canonical composition. Compatibility forms can erase meaningful distinctions, such as the ligature ff versus ff. Omitting the argument or passing undefined selects NFC; an unrecognized form name throws RangeError.
=== compares code-unit sequences without normalizing. To treat canonically equivalent forms as equal, normalize both strings to NFC before comparing. Normalization can change lengths and offsets; it does not by itself provide case-insensitive matching or protect against spoofing with lookalike characters.
Examples
{
const face = "\u{1F600}";
const lone = face.slice(0, 1);
const composed = "\u00E9";
const decomposed = "e\u0301";
console.log(`${face.length} ${[...face].length}`);
console.log(`${face.charCodeAt(0).toString(16)} ${face.codePointAt(0).toString(16)} ${face.codePointAt(1).toString(16)}`);
console.log(`${lone.isWellFormed()} ${lone.toWellFormed().codePointAt(0).toString(16)}`);
console.log(`${composed.length} ${decomposed.length} ${composed === decomposed} ${composed.normalize() === decomposed.normalize()}`);
console.log(`${"\uFB00".normalize("NFC") === "ff"} ${"\uFB00".normalize("NFKC") === "ff"}`);
}Result
2 1 d83d 1f600 de00 false fffd 1 2 false true false true
The first two lines show an emoji with two code units but one code point. The third shows repair after splitting the pair. The fourth compares two representations of `é`, and the fifth distinguishes canonical from compatibility normalization.
Related lessons
- JavaScript · Lesson 3 · Advanced · Version: 4
Sources
- ECMAScript 2024 §6.1.4 — The String Type
- ECMAScript 2024 §22.1.3.3 — String.prototype.charCodeAt
- ECMAScript 2024 §22.1.3.4 — String.prototype.codePointAt
- ECMAScript 2024 §22.1.3.10 — String.prototype.isWellFormed
- ECMAScript 2024 §22.1.3.15 — String.prototype.normalize
- ECMAScript 2024 §22.1.3.31 — String.prototype.toWellFormed
- ECMAScript 2024 §22.1.3.36 — String.prototype [ @@iterator ]
- ECMAScript 2024 Internationalization API Specification §18 — Segmenter Objects