Unicode
Unicode is the universal character encoding standard for modern computing, with UTF-8 as its most widely used encoding form. It supports characters from all major world languages and has become the standard character encoding scheme for modern terminal environments.
Core Facts
Unicode defines multiple zero-width characters, such as U+200B (zero-width space), U+200C (zero-width non-joiner), U+200D (zero-width joiner), and U+FEFF (zero-width no-break space)
As of its latest version, Unicode has included more than 150,000 characters.
Timeline (last 90 days)
1 failure case involved email masking supporting only ASCII characters; when an email contained Cyrillic letters, the masking truncated the email into two parts, causing the lookup to fail.
The Persian letter ی (U+06CC) and the Arabic letter ي (U+064A) look identical in most fonts but are different Unicode code points, and the same applies to ک (U+06A9) and ك (U+0643).
If a language uses rare Unicode symbols, unconventional whitespace structures, or deliberately constructed ambiguous tokens, the tokenizer may split them into inefficient or even meaningless fragments, significantly increasing the difficulty of model understanding and generation.
The character é can be a single scalar value U+00E9, or it can consist of two scalar values, e plus the combining accent U+0301, but at the grapheme cluster level they both display as one character.
Unicode scalar values refer to all code points except those in the surrogate range U+D800–U+DFFF, and Rust's char type corresponds exactly to one scalar value.
The preprocessing that BertNormalizer is responsible for includes: cleaning the text, inserting spaces around Chinese characters, performing NFD decomposition on Unicode characters, removing non-spacing marks (removing accents), and lowercasing.
The Unicode code point space is divided into pages of 256 code points each (for example, U+0000–U+00FF is one page), characters of the same writing system usually fall within adjacent pages, and their normalization behavior is highly consistent within the same page.
Characters outside the Basic Multilingual Plane (such as most emoji) become surrogate pairs consisting of two escape sequences when processed by ensure_ascii escaping.
Typical techniques for linguistic illegibility attacks include replacing sensitive vocabulary with Unicode variant characters, mixing multiple languages, using low-resource languages to bypass filters, and fragmenting harmful intent across multiple conversation turns.
The underlying text content in web pages and documents remains standard Unicode character encoding; fonts only determine the visual rendering of characters, not their storage.
10 more timeline events
All Facts (20)
Unicode defines multiple zero-width characters, such as U+200B (zero-width space), U+200C (zero-width non-joiner), U+200D (zero-width joiner), and U+FEFF (zero-width no-break space)
80%VerifiedAs of its latest version, Unicode has included more than 150,000 characters.
65%UnverifiedGB2312标准收录6,763个汉字，Unicode中的CJK统一表意文字超过9万个
90%Unverified1 failure case involved email masking supporting only ASCII characters; when an email contained Cyrillic letters, the masking truncated the email into two parts, causing the lookup to fail.
50%UnverifiedThe Persian letter ی (U+06CC) and the Arabic letter ي (U+064A) look identical in most fonts but are different Unicode code points, and the same applies to ک (U+06A9) and ك (U+0643).
50%UnverifiedIf a language uses rare Unicode symbols, unconventional whitespace structures, or deliberately constructed ambiguous tokens, the tokenizer may split them into inefficient or even meaningless fragments, significantly increasing the difficulty of model understanding and generation.
50%UnverifiedThe character é can be a single scalar value U+00E9, or it can consist of two scalar values, e plus the combining accent U+0301, but at the grapheme cluster level they both display as one character.
50%UnverifiedUnicode scalar values refer to all code points except those in the surrogate range U+D800–U+DFFF, and Rust's char type corresponds exactly to one scalar value.
50%UnverifiedThe preprocessing that BertNormalizer is responsible for includes: cleaning the text, inserting spaces around Chinese characters, performing NFD decomposition on Unicode characters, removing non-spacing marks (removing accents), and lowercasing.
50%UnverifiedThe Unicode code point space is divided into pages of 256 code points each (for example, U+0000–U+00FF is one page), characters of the same writing system usually fall within adjacent pages, and their normalization behavior is highly consistent within the same page.
50%UnverifiedCharacters outside the Basic Multilingual Plane (such as most emoji) become surrogate pairs consisting of two escape sequences when processed by ensure_ascii escaping.
50%UnverifiedTypical techniques for linguistic illegibility attacks include replacing sensitive vocabulary with Unicode variant characters, mixing multiple languages, using low-resource languages to bypass filters, and fragmenting harmful intent across multiple conversation turns.
50%UnverifiedThe underlying text content in web pages and documents remains standard Unicode character encoding; fonts only determine the visual rendering of characters, not their storage.
50%UnverifiedASCII smuggling leverages special characters in the Unicode character set, including zero-width characters, control characters, and Unicode tag characters (in the range U+E0000 to U+E007F).
50%UnverifiedThese invisible characters are not displayed in most email clients but exist in the byte stream of the text.
50%UnverifiedDefending against ASCII smuggling requires anomalous Unicode character detection, character normalization processing, and the stripping of invisible characters and tag characters at the email gateway level.
50%UnverifiedThe family emoji "👨👩👧👦" is composed of 7 Unicode code points, namely four person emojis connected in series by three zero-width joiners (U+200D).
50%UnverifiedThe URL specification RFC 3986 stipulates that only a subset of ASCII characters is allowed in URLs, and non-ASCII characters must be percent-encoded to be used legally.
50%UnverifiedModern terminal emulators support 256 colors or even 24-bit true color, combined with the Unicode extended character set
50%Unverified零宽字符是Unicode标准中一组特殊控制字符，包括零宽空格（U+200B）、零宽连接符（U+200D）、零宽非连接符（U+200C）、左至右标记（U+200E）等，视觉上完全不可见但在字节层面真实存在
50%