Knowledge baseExam Center
ENEnglish
Choose language13

CKTIS / C++

u8 string literal: code units, code points, and grapheme clusters

The `u8` prefix selects UTF-8 for a string literal. Its length in code units, its number of code points, and its number of grapheme clusters measure different things.

Version: ISO C++23; implementation support may vary

Syntax

u8"text"
u8R"tag(raw text)tag"

// C++23: a view over the UTF-8 code units of a literal
#include <string_view>
constexpr std::u8string_view text = u8"e\u0301";

A u8 string literal, such as u8"text", uses UTF-8 as its literal encoding. Since C++20 it is a nonmodifiable const char8_t[N] array with static storage duration; in C++11 through C++17 it was const char[N]. N includes the appended zero terminator. An unprefixed literal uses the ordinary literal encoding, which need not be UTF-8. In the raw form u8R"tag(...)tag", backslashes do not start escapes.

When the array decays to a pointer, the result is const char8_t*, not const char*. C++23 permits direct initialization of char and unsigned char arrays from a u8 literal, but adds no implicit conversion between these pointer types; implementation support for the fix may vary. std::cout << u8"x" selects a deleted overload. Before passing text to an interface that takes const char*, check what encoding that interface expects.

std::u8string_view from <string_view> does not own its data. A view over a literal can refer to that storage for the duration of the program. size() counts code units in the view. The pointer-only constructor stops before the first zero; pass a pointer and a length to retain embedded zeros.

A UTF-8 code unit is eight bits. A code point is a numeric value in the Unicode codespace; UTF-8 encodes each scalar value (a code point other than a surrogate) in one to four units. An extended grapheme cluster groups one or more code points according to Unicode segmentation rules and approximates a user-perceived character. e followed by combining acute accent U+0301 is three code units, two code points, and one such cluster.

Only for valid UTF-8, count code points by counting units that are not continuations (10xxxxxx). To keep the first k code points, end the prefix at the start of the next code point, or at the end of the data if there is no next one. Cutting at an arbitrary code-unit index can split a UTF-8 sequence; cutting at a code-point boundary can split a cluster. Neither this count nor this trim validates UTF-8, normalizes text, or finds grapheme boundaries. Even u8"\xFF" contains an invalid UTF-8 unit; validate untrusted input before using this shortcut, and segment the text separately when truncating by grapheme cluster.

Examples

#include <cstddef>
#include <iostream>
#include <string_view>

int main() {
    constexpr auto& literal = u8"e\u0301";
    constexpr std::u8string_view text{literal};
    static_assert(sizeof(literal) == 4 && text.size() == 3);

    std::size_t points = 0;
    std::size_t first_end = text.size();
    for (std::size_t i = 0; i < text.size(); ++i) {
        if ((static_cast<unsigned>(text[i]) & 0xC0u) != 0x80u) {
            ++points;
            if (points == 2) {
                first_end = i;
            }
        }
    }

    const auto prefix = text.substr(0, first_end);
    std::cout << "Code units: " << text.size() << '\n'
              << "Code points: " << points << '\n'
              << "First code point, code units: " << prefix.size() << '\n';
}

Result

Code units: 3
Code points: 2
First code point, code units: 1

The program prints counts, so the result does not depend on whether the terminal can render the accent. The prefix contains only `e`: it remains valid UTF-8 but drops the accent from the original cluster. The program does not compute a grapheme count.

Related lessons

  • C++ · Lesson 49 · Advanced · Version: 3

Sources