CKTIS / C++
UTF-8 code units, code points and grapheme clusters in C++
UTF-8 text has different lengths depending on what you count: code units, code points or grapheme clusters. The example counts and truncates well-formed UTF-8 without splitting a code point.
Version: ISO C++23; implementation support may vary
Syntax
// u8"text" has type const char8_t[5]; the bound includes the terminating zero code unit.
std::u8string_view text = u8"text"; // <string_view>
text.size(); // number of code units in the view
text.substr(0, n); // prefix of up to n code units; no Unicode boundary checkUTF-8 code units, code points and grapheme clusters measure different things. A UTF-8 code unit is an 8-bit octet. A code point is a numeric value in Unicode, such as U+0301. Each Unicode scalar value (a code point other than a surrogate) takes one to four code units in well-formed UTF-8. An extended grapheme cluster is a sequence of one or more code points grouped by Unicode segmentation rules (UAX #29). It approximates a user-perceived character, not a display width.
Since C++20, a u8"..." literal has type const char8_t[N], and N includes the terminating zero code unit. Unlike an ordinary "..." literal, it does not decay to a pointer that converts implicitly to const char*. std::u8string_view from <string_view> does not own its data. size() counts the code units in the view, and the pointer-only constructor stops at the first zero. To include embedded zeros, use the pointer-and-length constructor with a valid range. A literal lives for the whole program; other backing storage must remain valid while the view is used.
In well-formed UTF-8, each code point starts with exactly one non-continuation code unit; continuation units have the bit pattern 10xxxxxx. Counting the starts therefore works without decoding their values. substr(0, n) cuts by code units and may split a multicode-unit sequence. To keep the longest prefix within a code-unit budget, move the cut backward while it points at a continuation unit, stopping at a code-point boundary.
Both counting and trimming assume well-formed UTF-8: neither validates the encoding, normalizes text nor finds grapheme boundaries. For untrusted input, validate first and reject malformed data or apply an explicit replacement policy. To truncate by user-perceived characters, use a library that implements Unicode grapheme segmentation; C++23 provides no general-purpose API for it.
Examples
#include <cstddef>
#include <iostream>
#include <string_view>
constexpr bool is_continuation(char8_t unit) noexcept {
return (unit & 0xC0u) == 0x80u;
}
// Precondition: text is well-formed UTF-8.
constexpr std::size_t count_code_points(std::u8string_view text) noexcept {
std::size_t count = 0;
for (char8_t unit : text) {
if (!is_continuation(unit)) {
++count;
}
}
return count;
}
// Precondition: text is well-formed UTF-8.
constexpr std::u8string_view trim_to_unit_budget(
std::u8string_view text, std::size_t max_units) {
auto end = max_units < text.size() ? max_units : text.size();
while (end > 0 && end < text.size() && is_continuation(text[end])) {
--end;
}
return text.substr(0, end);
}
int main() {
constexpr std::u8string_view text = u8"e\u0301\U0001F642";
constexpr auto prefix = trim_to_unit_budget(text, 2);
static_assert(text.size() == 7);
static_assert(count_code_points(text) == 3);
static_assert(prefix == u8"e");
std::cout << "code_units: " << text.size() << '\n'
<< "code_points: " << count_code_points(text) << '\n'
<< "prefix_code_units: " << prefix.size() << '\n';
}Result
code_units: 7 code_points: 3 prefix_code_units: 1
The `e`, U+0301 and U+1F642 emoji take 1, 2 and 4 code units respectively: 7 code units, 3 code points and 2 grapheme clusters in total. A budget of 2 would cut inside the encoding of U+0301, so the function keeps only `e`. The prefix is well-formed UTF-8 but loses the accent. The program does not compute the cluster count.
Related lessons
- C++ · Lesson 49 · Advanced · Version: 3
Sources
- String literals [lex.string]
- Character sets [lex.charset]
- Compatibility with ISO C++ 2017: lexical conventions [diff.cpp17.lex]
- Header <string_view> synopsis [string.view.synop]
- Construction and assignment [string.view.cons]
- Capacity [string.view.capacity]
- String operations [string.view.ops]
- Standard format specifiers [format.string.std]
- N4950 — Working Draft, Standard for Programming Language C++
- Unicode Standard Annex #29: Unicode Text Segmentation