Unicode, UTF-8, and Binary Encoding

Every character you see on screen — from the letter A to a snowman emoji — is ultimately stored as binary data. The mapping between human-readable characters and binary bits is defined by encoding standards, and UTF-8 is the dominant encoding on the web today. Understanding how Unicode code points translate to UTF-8 byte sequences helps you debug mojibake, validate input, and work with binary protocols.

Unicode Code Points

Unicode assigns every character a unique integer called a code point, written as U+XXXX where XXXX is a hexadecimal number. The Unicode standard currently defines over 149,000 characters across scripts, symbols, and emoji.

Code points range from U+0000 to U+10FFFF, giving a maximum of 1,114,112 possible values. This range is divided into planes — the first plane (U+0000 to U+FFFF) is called the Basic Multilingual Plane (BMP) and covers most everyday characters.

UTF-8 Encoding Rules

UTF-8 is a variable-length encoding: each code point is represented by 1 to 4 bytes. The number of bytes depends on the code point's value:

The leading bits of the first byte tell a decoder how many bytes follow. Continuation bytes all start with 10, making resynchronization possible even if you start reading mid-stream.

Encoding Example: € (U+20AC)

1. Code point U+20AC = decimal 8364 = binary 0010 0000 1010 1100
2. This falls in the 3-byte range (U+0800–U+FFFF)
3. Distribute the 16 bits into the pattern 1110xxxx 10xxxxxx 10xxxxxx:
   • First byte: 1110 + 0010 = 11100010
   • Second byte: 10 + 000010 = 10000010
   • Third byte: 10 + 101100 = 10101100
4. UTF-8 bytes: 0xE2 0x82 0xAC → binary: 11100010 10000010 10101100

Encoding Example: 😊 (U+1F60A)

1. Code point U+1F60A = decimal 128522 = binary 0001 1111 0110 0000 1010
2. This falls in the 4-byte range (U+10000–U+10FFFF)
3. Distribute 21 bits into the pattern 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx:
   • First byte: 11110 + 000 = 11110000
   • Second byte: 10 + 011111 = 10011111
   • Third byte: 10 + 100000 = 10100000
   • Fourth byte: 10 + 001010 = 10001010
4. UTF-8 bytes: 0xF0 0x9F 0x98 0x8A

Why UTF-8 Won the Web

UTF-8 has several properties that made it the de facto standard:

• ASCII compatibility. The first 128 code points (U+0000–U+007F) encode as single bytes identical to ASCII. Any valid ASCII text is also valid UTF-8, which eliminates migration effort for English-centric systems.
• Self-synchronization. Because leading bytes and continuation bytes have distinct bit patterns, a UTF-8 decoder can find the start of the next character from any position in a byte stream — even after data corruption.
• No endianness issues. Unlike UTF-16 and UTF-32, UTF-8 is byte-oriented and does not require a byte-order mark (BOM).
• Space efficiency for Latin text. English and other Latin-script languages use 1 byte per character, compared to 2 bytes minimum for UTF-16.

As of 2026, over 98% of web pages use UTF-8 encoding. The Content-Type: text/html; charset=utf-8 header and the <meta charset="utf-8"> tag are standard practice.

Common Pitfalls

• Confusing code points with bytes. JavaScript's String.length counts UTF-16 code units, not Unicode code points. The string "😊".length returns 2, not 1, because the emoji is encoded as a surrogate pair in UTF-16.
• Overlong encodings. An overlong encoding uses more bytes than necessary to represent a code point (e.g., encoding U+0041 as two bytes). The UTF-8 standard forbids overlong encodings, and conformant decoders reject them as a security measure.
• Invalid byte sequences. A lone continuation byte (0x80–0xBF) without a leading byte, or a leading byte not followed by enough continuation bytes, makes the stream invalid. How decoders handle this — replacing with U+FFFD (�) or raising an error — varies by implementation.

Key Takeaways

• Unicode code points are abstract integer identifiers; UTF-8 is one way to encode them as binary bytes.
• UTF-8 uses 1–4 bytes per character, with the byte count encoded in the leading bits of the first byte.
• ASCII compatibility and self-synchronization are the key reasons UTF-8 dominates the web.
• Confusing code points, UTF-16 code units, and bytes is the root cause of most string-handling bugs.

Try It Yourself

Want to see the binary representation of any text? Paste a string into the Text to Binary Converter to visualize its UTF-8 byte sequences and binary encoding in real time.