How UTF-8 and Unicode Work: Code Points, Bytes, and Why Length Is Three Numbers

A string that looks like one character can be one byte, two bytes, or several code points that a font draws as a single glyph. len in one language counts bytes. In another it counts UTF-16 code units. Neither number is how many characters a reader sees.

Unicode assigns numbers to abstract characters. UTF-8 is one encoding of those numbers as bytes. Length is a third question, and the answer depends on the unit you count. This article is about that split: scalar values, the UTF-8 byte grammar, and why byte length, code-point length, and user-perceived length are not interchangeable. It is not a survey of fonts, collation, or every legacy code page.

The same kind of trap shows up in numeric formats: the bits stored are an encoding of an abstract value, not the value you typed. How IEEE 754 floating point works is that problem for decimals. This page is that problem for text.

What Unicode assigns

Unicode gives each abstract character a code point, written U+NNNN. The assigned range runs from U+0000 to U+10FFFF, a little over a million numbers. The Basic Multilingual Plane is U+0000 through U+FFFF. Supplementary planes, including most emoji, sit at U+10000 and above.

Not every number in that range is a Unicode scalar value. U+D800 through U+DFFF are surrogates, reserved so UTF-16 can encode supplementary characters as a pair of 16-bit units. A scalar value is a code point that is not a surrogate. Encodings that claim to store Unicode text are defined on scalar values, not on the surrogate range.

A code point is not a glyph, not a grapheme, and not a byte. U+00E9 is the abstract character LATIN SMALL LETTER E WITH ACUTE. A font decides how to draw it. UTF-8 decides which bytes represent it. A renderer may also draw the two-code-point sequence U+0065 plus U+0301 (e, then combining acute) as the same mark. Those are different scalar sequences with the same appearance under canonical equivalence.

The UTF-8 byte grammar

UTF-8, as constrained by RFC 3629, encodes each scalar value in one to four bytes. The lead byte says how long the sequence is. Continuation bytes always start with the bits 10, which is why a decoder that loses sync can resynchronize on the next byte that does not match 10xxxxxx.

• U+0000 to U+007F: 0xxxxxxx. One byte, identical to ASCII. A file that is pure ASCII is valid UTF-8 with no conversion.
• U+0080 to U+07FF: 110xxxxx 10xxxxxx. Two bytes.
• U+0800 to U+FFFF, excluding surrogates: 1110xxxx 10xxxxxx 10xxxxxx. Three bytes.
• U+10000 to U+10FFFF: 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx. Four bytes.

The original UTF-8 design allowed five- and six-byte sequences. RFC 3629 forbids them. The largest legal lead byte is of the form 11110xxx, and the decoded value must be at most U+10FFFF. A lead byte of F5 or higher is ill-formed.

é (U+00E9) has the binary value 1110 1001. It does not fit in seven bits, so the two-byte form is required. The top five payload bits go in the lead byte and the bottom six go in the continuation:

U+00E9
110 00011  10 101001
C3          A9

The euro sign U+20AC needs three bytes: E2 82 AC. The grinning-face emoji U+1F600 needs four: F0 9F 98 80. Nothing in those sequences is a character count. They are the shortest legal encoding of one scalar value each.

What a decoder must reject

Shortest form is part of the grammar, not a style preference. ASCII solidus U+002F is the single byte 2F. The two-byte sequence C0 AF decodes to the same number if you ignore the length rule. Accepting it was a classic filter bypass: a check for / or .. on raw bytes missed the overlong form, and a later decoder turned it into a path separator. Conformant UTF-8 rejects overlong sequences.

Surrogate code points encoded as UTF-8 are also ill-formed. ED A0 80 would be U+D800 if the arithmetic were allowed. It is not a scalar value, so it is not UTF-8. The same rejection applies to a four-byte sequence that lands above U+10FFFF, and to a truncated sequence at the end of a buffer. A streaming decoder may hold a partial lead sequence until more bytes arrive. A finished buffer may not.

APIs disagree on the recovery policy, and that disagreement is part of the contract. Rust str::from_utf8 and Go's strict decode return an error. Python bytes.decode("utf-8") raises UnicodeDecodeError unless you pass errors="replace" or errors="surrogateescape". When a replacement policy is used, Unicode's recommended replacement character is U+FFFD. Replacing is not the same as accepting the bad bytes as characters.

Three lengths, one string

Take the word café, composed. The scalar values are U+0063 U+0061 U+0066 U+00E9. That is four code points. The UTF-8 bytes are 63 61 66 C3 A9, five bytes. A renderer draws four grapheme clusters.

The same word decomposed is U+0063 U+0061 U+0066 U+0065 U+0301. Five code points, six bytes (63 61 66 65 CC 81), and still four grapheme clusters if the display combines the acute. Byte equality fails. Visual equality holds. A unique index on the raw UTF-8 of a name will treat them as different keys.

Language runtimes pick a unit and then name it badly:

• Python 3 len("😀") is 1, because str is a sequence of scalar values. len("😀".encode("utf-8")) is 4.
• Go len("😀") is 4, because a Go string is bytes. utf8.RuneCountInString is 1. A range loop yields runes, not bytes.
• JavaScript "😀".length is 2. A JS string is a sequence of UTF-16 code units, and U+1F600 is the surrogate pair D83D DE00. Array.from("😀").length or [..."😀"].length is 1, because the iterator yields scalar values.
• Rust s.len() is bytes. s.chars().count() is scalar values. Grapheme segmentation is a separate crate and a separate algorithm.

A family emoji built with zero-width joiners is several scalar values and one extended grapheme cluster. Slicing it on code-point boundaries can leave a dangling joiner. Slicing it on byte boundaries can leave an invalid UTF-8 tail. Column width in a terminal is a fourth number again: East Asian wide characters and emoji presentation are font and locale questions, not encoding questions.

Normalization is not an encoding

Normalization Form C composes where a single code point exists. Normalization Form D decomposes to a base plus combining marks. Both forms are legal Unicode, and both encode cleanly as UTF-8. Encoding does not pick a form. Comparison, database uniqueness, and filenames have to pick one, or they will split one name into two keys.

macOS has historically stored filenames in a decomposed form. A Linux volume often stores the exact byte sequence the application wrote. Copy a composed é onto a volume that normalizes to NFD and the path bytes change even though no character was "lost." Hashing the file path without a normalization step then misses the file. NFC and NFD are canonical equivalents; they are not byte-identical, and compatibility forms (NFKC, NFKD) go further by folding characters that are only visually similar, which is the wrong tool for identifiers.

What this is not

UTF-16 stores 16-bit code units and uses surrogate pairs for supplementary planes. That is why Java and JavaScript lengths surprise people, but it is a different encoding. UTF-32 stores one 32-bit unit per scalar value and spends three zero bytes on every ASCII character. Neither is the subject here.

A byte-order mark, EF BB BF, is the UTF-8 encoding of U+FEFF. It is not required, and in the middle of a stream it is a word joiner, not a signature. Protocols that strip a leading BOM and protocols that treat it as data both exist. Guessing the encoding from a BOM is a separate problem from decoding a buffer you already know is UTF-8.

Takeaways

Unicode assigns scalar values. UTF-8 encodes each scalar value as one to four bytes, with ASCII unchanged and continuation bytes marked 10xxxxxx. Overlong forms, surrogate code points, and values above U+10FFFF are ill-formed, not alternate spellings. Byte length, code-point length, and grapheme length answer different questions, and JavaScript's length answers a fourth one: UTF-16 code units. Normalize before you compare, and slice only on a boundary your API actually promised.