Mapping of Unicode characters
From Wikinfo
| Unicode |
|---|
| Encodings |
| UCS |
| Mapping |
| Bi-directional text |
| BOM |
| Han unification |
| Unicode and HTML |
| Unicode and E-mail |
| Unicode typefaces |
Unicode’s Universal Character Set potentially supports over 1 million (1,114,112 = 220 + 216 or 17 × 216, hexadecimal 110000) code points.
As of Unicode 5.0.0, 102,012 (9.2%) of these code points are assigned, with another 137,468 (12.3%) reserved for private use, 2,048 for surrogates, and 66 designated noncharacters, leaving 872,582 (78.3%) unassigned. The number of assigned code points is made up as follows:
- 2,684 in reserve for designation within a particular block
- 98,893 graphical characters
- 435 special purpose characters for control, formatting, and glyph/character variation selection.
(See the summary table for a more detailed breakdown).
Unicode characters can be categorized in many ways. Every character is assigned a script (though many are assigned the common or inherited scripts where they inherit the script from the adjacent character). In Unicode a script is a coherent writing system that includes letters but also may include script specific punctuation, diacritic and other marks and numerals and symbols. A single script supports one or more languages.
Characters are assigned in blocks of characters. These blocks are usually groups of code points in some multiple of eight: many, for example, are grouped in blocks of 128 or 256 code points. Every character is also assigned a general category and subcategory. The general categories are: letter, mark, number, punctuation, symbol, or control (in other words a formatting or non-graphical character).
The blocks of characters are assigned according to various planes. Most characters are currently assigned to the first plane: the Basic Multilingual Plane. This is to help ease the transition for legacy software since the Basic Multilingual Plane is addressable with just two octet bytes. The characters outside the first plane usually have very specialized or rare use.
The first 256 code points correspond with those of ISO 8859-1, the most popular 8-bit character encoding in the Western world. As a result, the first 128 characters are also identical to ASCII. Though Unicode refers to these as a Latin script block, these two blocks contain many characters that are commonly useful outside of the Latin script.
Planes
The Unicode characters can be categorized in many different ways, Unicode code points can be logically divided into 17 planes, each with 65,536 (= 216) code points, although currently only a few planes are used:
- Plane 0 (0000–FFFF): Basic Multilingual Plane (BMP). This is the plane containing most of the character assignments so far. A primary objective for the BMP is to support the unification of prior character sets as well as characters for writing systems in current use.
- Plane 1 (10000–1FFFF): Supplementary Multilingual Plane (SMP).
- Plane 2 (20000–2FFFF): Supplementary Ideographic Plane (SIP)
- Planes 3 to 13 (30000–DFFFF) are unassigned
- Plane 14 (E0000–EFFFF): Supplementary Special-purpose Plane (SSP)
- Plane 15 (F0000–FFFFF) reserved for the Private Use Area (PUA)
- Plane 16 (100000–10FFFF), reserved for the Private Use Area (PUA)
Currently, about ten percent of the potential space is used. Furthermore, ranges of characters have been tentatively blocked out for every current and ancient writing system (script) the Unicode consortium has been able to identify: (see [1]). While Unicode may eventually need to use another of the spare 11 planes for ideographic characters, other planes remain, if previously unknown scripts with tens of thousands of characters are discovered. This 20 bit limit is therefore unlikely to be reached in the near future.
Basic Multilingual Plane
The first plane (plane 0), the Basic Multilingual Plane (BMP), is where most characters have been assigned so far. The BMP contains characters for almost all modern languages, and a large number of special characters. Most of the allocated code points in the BMP are used to encode Chinese, Japanese, and Korean (CJK) characters.
The graphic on the right is a visual roadmap to the Basic Multilingual Plane. The colours in use are:
- Black = Latin scripts and symbols
- Light Blue = Linguistic scripts
- Blue = Other European scripts
- Orange = Middle Eastern and SW Asian scripts
- Light Orange = African scripts
- Green = South Asian scripts
- Purple = Southeast Asian scripts
- Red = East Asian scripts
- Light Red = Unified CJK Han
- Yellow = Canadian Aboriginal scripts
- Magenta = Symbols
- Dark Grey = Diacritics
- Light Grey = UTF-16 surrogates and private use
- Cyan = Miscellaneous characters
- White = Unused
As of Unicode 5.0, The BMP includes the following scripts:
|
|
|
Future additions Several scripts are expected to be included in the BMP in the next revision of Unicode. These scripts, and their proposed code point ranges, are the following:
- Cham (18B0–18FF)
- Lanna (Old Tai Lue) (1A80–1AEF)
- Santali (Ol Cemet' / Ol Chiki) (2DE0–2DFF)
- Vai (A500–A61F)
- Saurashtra (AB00–AB5F)
Several other scripts are proposed for inclusion in the BMP, including:
- Avestan (0800–083F)
- Pahlavi (0840–087F)
- Batak (1A20–1A5F)
- Meitei Mayek / Meitei (1C80–1CDF)
- Varang Kshiti (AA00–AA3F)
- Sorang Sompeng (AA40–AA6F)
Supplementary Multilingual Plane
Plane 1, the Supplementary Multilingual Plane (SMP), is mostly used for historic scripts such as Linear B, but is also used for musical and mathematical symbols.
|
As of Unicode 5.0, Plane One includes the following scripts:
|
Many other scripts are proposed for inclusion in Plane One, including: |
Supplementary Ideographic Plane
Plane 2, the Supplementary Ideographic Plane (SIP), is used for about 40,000 Unified Han Ideographs that have previously been seldom used in daily written communications.
Unused planes
Unicode has not yet assigned any characters to Planes 3 through 13. It is not anticipated that these planes will be needed, given the total sizes of the known writing systems left to be encoded. However, the number of possible symbol characters that could arise outside of the context of writing systems is potentially limitless. The UCS and Unicode take requests for symbols on a case by case basis.
Supplementary Special-purpose Plane
Plane 14 (E in hexadecimal), the Supplementary Special-purpose Plane (SSP), currently contains non-graphical characters in two blocks of 128 and 240 characters. The first block is for language tag characters for use when language cannot be indicated through other protocols (such as the xml:lang attribute in XML). The other block contains glyph variation selectors to indicate an alternate glyph for a character that cannot be determined by context.
Private use planes
Two planes (planes 15 and 16) have been set aside for character assignment by parties outside the ISO and the Unicode Consortium. Use of such characters will have limited interoperability. Software and fonts that support Unicode will not necessarily support characters assignments by other parties. Especially if the characters have unusual properties such as right-to-left characters, other implementations may treat those characters inappropriately.
Plane mapping tables
Graphical characters
By far the most common Unicode characters are graphical characters. Graphical characters all have some visual representation or glyphs associated with them. While Unicode does not specify the concrete glyphs for these characters, it does specify recommended or prototypical glyphs. The actual glyph used by textual display software will depend on the font files used and whether those fonts provide support for contextual and non-contextual glyph variations
Script-specific characters
In Unicode, a script is an abstract coherent and unified writing system supporting one or more concrete writing systems which in turn support the written forms of one or more languages. Some scripts support one and only one language, for example: Armenian. Other scripts, like Latin, support many different writing systems: English, French, German, Italian, and Latin to name just a few. Some languages also make use of multiple alternate writing systems. Turkish, for example used Arabic before the 20th century and transitioned to Latin in the early part of the 20th century. For a list of languages supported by each script see the list of languages by writing system.
When multiple languages make use of the same script, there are frequently some differences: particularly in diacritics and other marks. For example, Swedish and English both use the Latin script. However, Swedish includes the character ‘å’ (sometimes called a “Swedish O”) while English has no such character. Nor does English make use of the diacritic combining circle above for any character. In general the languages sharing the same scripts share many of the same characters. Despite these peripheral differences in the Swedish and English writing systems they are said to use the same Latin script. So the Unicode abstraction of writing systems is a basic organizing technique. The differences between different alphabets or writing systems remain and are supported through Unicode’s flexible scripts, combining marks and collation algorithms.
While all characters have the property of belonging to a script, many characters, such as symbols, indicate “common” or “inherited” for their script property. The unified diacritical characters and unified punctuation characters frequently have the “common” or “inherited” script property. However, the individual scripts often have their own punctuation and diacritics. So many scripts include not only letters, but also diacritic and other marks, punctuation, numerals and even their own idiosyncratic symbols and space characters.
Unicode already includes over 60 scripts supporting hundreds or even thousands of languages throughout the World. Unicode is actively working on many more as indicated by its roadmap.
Unihan characters
Han unification is the process used by the authors of Unicode and the Universal Character Set to map multiple character sets of the CJK languages into a single set of unified characters. The Chinese characters are common to Chinese (where they are called hanzi), Japanese (where they are called kanji), and Korean (where they are called hanja). Modern Korean, Chinese and Japanese typefaces may represent a given Han character as somewhat different glyphs. However, in the formulation of Unicode, these different glyphs were treated as the same character. This unification is referred to as "Han unification", with the resulting character repertoire sometimes referred to as Unihan.
Besides the Unihan ideographs, Han unification also provides Han unified punctuation, symbols, numerals, ideograph stroke characters and ideographic description characters.
Phonetic characters
- It was thought by at least one person on Wikinfo that input from an expert in this subject would be helpful.
Unicode includes letters and marks from the International Phonetic Alphabet (IPA) and those supporting other phonetic writing systems too. Essentially these characters are used as graphemes for phonemes. In terms of script or writing system, these phonetic alphabets are basically one writing system. What distinguishes the various phonetic alphabets are their glyphs. However, as with numerals, the UCS often focus more on the presentational forms or glyphs given to these phonemes by the various phonetic alphabets. This is in contrast to the alternate names of these characters provided by Unicode NamesList property which typically reflects the common phoneme semantics shared by those various writing systems regardless of the glyphs used. So these differences manifest in the alternate names given to these characters: the canonical UCS name and the NamesList property names. Similarly, Unicode assignees the value of “Latin” to the script property of many of these characters. However, the primary purpose for these characters inclusion in the character set is to support the various phonetic writing systems. These phonetic writing system, in many ways, constitute a single unified writing system on its own: despite borrowing glyphs from other Latin, Greek and Cyrillic scripts.
Numerals
Numerals (often called numbers in Unicode) are characters that denote a number. The same Arabic-Indic numerals are used widely in various writing systems throughout the world and all share the same semantics for denoting numbers, However, the glyphs representing these numerals differ widely from one writing system to another. To support these glyph differences, Unicode includes duplicate encodings of these numerals within many of the script blocks. These digits are repeated in 23 separate blocks: twice in Arabic. Six additional blocks contain the digits again as rich text or legacy software compatibility characters.
Unicode also includes several less common numerals: Roman numerals, counting rod numerals, Cuneiform numerals and ancient Greek numerals.
Numerals invariably involve composition of glyphs as a limited number of characters are composed to make other numerals. For example the sequence 9 - 9 - 0 in Arabic-Indic numerals composes the numeral for nine hundred and ninety (990). In Roman numerals, the same number is expressed by the composed numeral Ⅹↀ or ⅩⅯ. Each of these is a distinct numeral for representing the same abstract number. The semantics of the numerals differ in particular in their composition. The Arabic-Indic decimal digits are positional-value compositions, while the Roman numerals are sign-value and they are additive and subtractive depending on their composition.
Punctuation and diacritics
Unicode includes several blocks for unified diacritics and other combining marks and also blocks for unified punctuation. However, when a mark or punctuation character is intended primarily for use within a particular script, the character is assigned to that particular script’s blocks. Therefore authors will find these types of characters throughout the Unicode character database. Unicode categorizes them as:
- Punctuation
- connector (Pc)
- dash (Pd)
- open (Po)
- close (Pe)
- initial (Pi)
- final (Pf)
- Mark
- non-spacing (Mn)
- spacing-combining (Mc
- enclosing (Me)
Symbols
Unicode has dozens of blocks dedicated to symbols that are useful regardless of one’s writing system. Other script-specific symbols are often included within a particular script’s blocks. Symbols are categorized as:
Symbols:
- math (Sm)
- currency (Sc)
- modifier (Sk)
- other (So)
Music notation
Unicode devotes a block of 256 characters for musical symbols. Since Unicode focuses on characters laid out in two dimensions, these characters do not encode pitch or other parts of Western music expressed in the vertical dimension. Therefore the music symbols are more suited for discussions of music symbols themselves or to discuss rhythm within the prose of a document. To encode more complex musical information some other data format is necessary, such as MusicXML or Midi.
Compatibility characters
In discussing Unicode and the UCS, many often refer to compatibility characters. Compatibility characters are graphical characters that are discouraged by the Unicode Consortium. As the Unicode consortium says:
A character that would not have been encoded except for compatibility and round-trip convertibility with other standards
However, the definition is more complicated that the glossary reveals. One of the properties given to characters by the Unicode consortium is the characters decomposition or compatibility decomposition. Most characters have no value for this property, but over 5 thousand characters do have a compatibility decomposition mapping that compatibility character to one or more other characters. By setting a characters decomposition property, Unicode establishes that character as a compatibility character. The reasons for these compatibility designations are varied and are discussed in further detail below. The term decomposition can sometimes confuse because a characters decomposition can, in some cases, be a singleton. In these cases the decomposition of one character is simply another equivalent or approximately equivalent character.
Canonical and Non-canonical
The compatibility decomposition property for the 5,402 Unicode compatibility characters includes a keyword that divides the compatibility characters into 17 logical groups. Those without a keyword are termed canonical equivalent or canonical decomposable characters. These characters have the closest relationship. Other keywords include: <initial>, <medial>, <final>, <isolated>, <wide>, <narrow, <small>, <square>, <vertical>, <circle>, <noBreak>, <fraction>, <subscript>, <superscript>, and <compat>. These keywords provide some indication of the relation between the compatibility character and its compatibility decomposition character sequence. However, the compatibility characters — whether canonical or not — fall in three basic categories: 1) characters corresponding to multiple alternate glyph forms and precomposed diacritics to support software and font implementations that do not include complete Unicode text layout capabilities; 2) characters included from other character sets or otherwise added to the UCS that constitute rich text rather than the plain text goals of Unicode; 3) some other characters that are semantically distinct, but visually similar. Because these semantically distinct characters may be displayed with glyphs similar to the glyphs of other characters, text processing software should try to address possible confusion for the sake of end users. When comparing and collating (sorting) text strings, different forms and rich text variants of characters should not alter the text processing results. For example, software users may be confused when performing a find on a page for a capital Latin letter ‘I’ and their software application fails to find the visually similar Roman numeral ‘Ⅰ’.
Compatibility Blocks
Several blocks of Unicode characters include either entirely or almost entirely all compatibility characters. These compatibility blocks contain none of the semantically distinct compatibility characters and so they fall unambiguously into the set of discouraged characters. Unicode recommends authors use the plain text compatibility decomposition equivalents instead and complement those characters with rich text markup. This approach is much more flexible and open-ended than using the finite set of circled or enclosed alphanumerics to give just one example.
Unfortunately, there are a small number of characters even within the compatibility blocks that themselves are not compatibility characters and therefore may confuse authors. The “Enclosed CJK Letters and Months” block contains a single non-compatibility character: the ‘Korean Standard Symbol’ (㉿ U+327F). This symbol and 12 other characters have been included in these blocks for no known reasons. The “CJK Compatibility Ideographs” block contains these non-compatibility unified Han ideographs:
- (U+FA0E): 﨎
- (U+FA0F): 﨏
- (U+FA11): 﨑
- (U+FA13): 﨓
- (U+FA14): 﨔
- (U+FA1F): 﨟
- (U+FA21): 﨡
- (U+FA23): 﨣
- (U+FA24): 﨤
- (U+FA27): 﨧
- (U+FA28): 﨨
- (U+FA29): 﨩
These thirteen characters are neither compatibility characters nor are their use discouraged in any way.
Several other characters in these blocks have no compatibility mapping but are clearly intended for legacy support:
Alphabetic Presentation Forms (1)
- Hebrew Point Judeo-Spanish Varika (U+FB1E): ﬞ. This is a glyph variant of Hebrew Point Rafe (U+05BF): ֿ , though Unicode provides no compatibility mapping.
Arabic Presentation Forms (4)
- “Ornate Left Parenthesis” (U+FD3E): ﴾. A glyph variant for U+0029 ‘)’
- “Ornate Right Parenthesis” (U+FD3F): ﴿. A glyph variant for U+0028 ‘ (’
- “Ligature Bismillah Ar-Rahman Ar-Raheem” (U+FDFD): ﷽. Bismillah Ar-Rahman Ar-Raheem is a ligature for Teh Marbuta (U+0629), Lam (U+0644), Meem (U+0645), Seen (U+0633), Beh (U+0628), (بسملة)
- “Arabic Tail Fragment” (U+FE73): ﹳ for supporting text systems without contextual glyph handling
CJK Compatibility Forms (2 that are both related to CJK Unified Ideograph: U+4E36 丶)
- Sesame Dot (U+FE45): ﹅
- White Sesame Dot (U+FE46): ﹆
Enclosed Alphanumerics (21 rich text variants)
- 10 Negative Circled Numbers (0 and 11 through 20) (U+24FF and U+24EB through U+24F4): ⓫ – ⓴
- 11 Double Circled Numbers (0 through 10) (U+24F5 through U+24FE): ⓵ – ⓾
Compatibility characters and normalization
Normalization is the process by which Unicode conforming software first performs compatibility decomposition before making comparisons or collating text strings. This is similar to other operations needed when, for example, a user performs a case or diacritic insensitive search within some text. In such cases software must equate or ignore characters it would not otherwise equate or ignore. Typically normalization is performed without altering the underlying stored text data (lossless). However, some software may potentially make permanent changes to text that eliminates the canonical or even non-canonical compatibility characters differences from text storage (lossy).
Non-graphical characters
Many characters are used to control the interpretation or display of text, but these characters themselves have no visual or spatial representation. For example, the null character (U+0000) is used in C-programming application environments to indicate the end of a string of characters. In this way, these programs only require a single starting memory address for a string. The string ends once the program reads the null character.
Legacy control characters
The legacy control characters come from ASCII and ISO 8859-1 character sets and are sometimes referred to as C0 and C1 respectively. Many of these characters play no explicit role in Unicode text handling, though they are still used in mainframe computing environments. Others, like the null character and many whitespace characters are still used commonly in text processing. Other common control characters are tabulation or tab (U+0009), linefeed (U+000A), carriage return (U+000D) and newline (U+0085). These are included among whitespace characters because, though they have no visual glyph, they do insert vertical or horizontal spacing between the display of characters.
Unicode introduced separators
In an attempt to simplify the several new line characters used in legacy text, UCS introduces its own new line characters to separate either lines or paragraphs: the line separator (U+2028) and paragraph separator (U+2029) characters.
Language tags
Unicode includes 128 characters as language tags. The characters essentially mirror the 128 ASCII characters except, when used they identify the subsequent text as belonging to a particular language according to BCP 47. For example, for indicating subsequent text as the variant of English as written in the United States, the initiating ‘Language Tag character’ (U+E0001) followed by the sequence ‘Tag Small Letter e’ (U+U+E0065), ‘Tag Small Letter n’ (U+E006E), “Tag Hyphen-minus’ (U+E002D), ‘Tag Small Letter u’ (U+E0075) and ‘Tag Small Letter s’ (U+E0073).
These language tag characters would not be displayed themselves. However, they would provide information for text processing or even for the display of other characters. For example the display of Unihan ideographs might substitute different glyphs if the language tags indicated Korean than if the tags indicated Japanese. Another example, might influence the display of decimal digits 0 through 9 differently depending on the language they appeared in.
Interlinear annotation
Three formatting characters provide support for interlinear annotation (U+FFF9, U+FFFA, U+FFFB). This may be used for providing notes that would typically be displayed between the lines of other text. Unicode considers such annotation to be rich text and recommends using other protocols for such annotation. The W3C ruby markup recommendation is an example of an alternate protocol supporting more advanced interlinear annotation.
Bidirectional text control
Unicode supports standard bidirectional text without any special characters. In other words Unicode conforming software should display right-to-left characters such as Hebrew letters as right-to-left simply from the properties of those characters. Similarly, the Unicode handles the mixture of left-to-right-text alongside right-to-left text without any special characters. For example, one can quote Arabic (“بسملة”) right alongside English and the Arabic letters will flow from right-to-left and the Latin letters left-to-right.. However, support for bidirectional text becomes more complicated when text flowing in opposite directions is embedded hierarchically. So that for example if one quotes an Arabic phrase that in turn quotes an English phrase. Other situations may complicate this when for example, an author wants the left-to-right characters overridden so that they flow from right-to-left. While these situations are fairly rare, Unicode provides seven characters ((U+200E, U+200F, U+202A, U+202B, U+202C, U+202D, U+202E) to help control these embedded bidirectional text levels up to 61 levels deep.
Variation Selectors
Many characters map to alternate glyphs depending on the context. For example Arabic and Latin cursive characters substitute different glyphs to connect glyphs together depending on whether the character is the initial character in a word, the final character, a medial character or an isolated character. These types of glyph substitution are easily handled by the context of the character with no other authoring input involved. Authors may also use special-purpose characters such as joiners and non-joiners to force an alternate form of glyph where it would not otherwise appear. Ligatures are similar instances where glyphs may be substituted simply by turning ligatures on or off as a rich text attribute.
However, for other glyph substitution, the authors intent may need to be encoded with the text and cannot be determined contextually. This is the case with character/glyphs referred to as gaiji where different glyphs are used for the same character either historically or for ideographs for family names. This is one of the gray areas in distinguishing between a glyph and a character. If a family name differs slightly from the ideograph character it derives from, then is that a simple glyph variant or a character variant? As of Unicode 3.2 and 4.0, the character set now includes 256 variation selectors so that these combining mark characters can select from 256 possible character/glyph variations for the preceding character. Unicode does not as yet provide any registry for these variations, so the issue of interoperable variation registration is left to other parties.
Other Special-purpose characters
Several characters fall between the non-graphical control and formatting characters and full-fledged graphical characters.
Joiners and Non-joiners
Word Joiner (U+2060), Zero-width joiner (U+200D), Zero-width non-joiner (U+200C), Zero-width space (U+200B), Combining Grapheme Joiner (U+034F).
Invisible Separator
Primarily for mathematics, the Invisible Separator (U+2063) provides a separator between characters where punctuation or space may be omitted such as in a two-dimensional index like ij.
Invisible Times and Function Application
Invisible Times (U+2062) and Function Application (U+2061) are useful in mathematics text where the multiplication of terms or the application of a function is implied without any glyph indicating the operation.
Spaces
The space character (U+0020) typically input by the space bar on a keyboard serves semantically as a word separator in many languages. For legacy reasons, the UCS also includes spaces of varying sizes that are compatibility equivalents for the space character. These spaces include:
- Space (U+0020)
- En Quad (U+2000)
- Em Quad (U+2001)
- En Space (U+2002)
- Em Space (U+2003)
- Three-Per-Em Space (U+2004)
- Four-Per-Em Space (U+2005)
- Six-Per-Em Space (U+2006)
- Figure Space (U+2007)
- Punctuation Space (U+2008)
- Thin Space (U+2009)
- Hair Space (U+200A)
- Mathematical Space (U+205F)
Aside from the original ASCII space, the other spaces are all compatibility characters. In this context this means that they effectively add no semantic content to the text, but instead provide styling control. Within Unicode, this non-semantic styling control is often referred to as rich text and is outside the thrust of Unicode’s goals. Rather than using different spaces in different contexts, this styling could instead be handled through intelligent text layout software.
Line-break control characters
Several characters are designed to help control line-breaks either by discouraging them (no-break characters) or suggesting line breaks such as the soft or shy hyphen (U+00AD). Such characters, though designed for styling, are probably indispensable for the intricate types of line-breaking they make possible.
- Shy Hyphen (U+00AD)
- Non-breaking Hyphen (U+2011)
- No-break Space (U+00A0)
- Narrow No-break Space (U+202F)
- Zero-width space (U+200B)
Whitespace characters
Whitespace characters are not a separate group of characters, but instead Unicode provides a list of characters it deems whitespace characters for interoperability support. Software Implementations and other standards may use the term to denote a slightly different set of characters. Whitespace characters are characters typically designated for programming environments. Often they have no syntactic meaning in such programming environments and are ignored by the machine interpreters. Unicode designates the legacy control characters U+0009 through U+000D and U+0085 as white space characters as well as the Unicode introduced line separator and paragraph separator. Also the core space character (U+0020) is designated as a whitespace character, but none of the other styling spaces.
Private use characters
The UCS includes over 100,000 code points for private use. This means these code points can be assigned characters with specific properties by individuals, organizations and software vendors outside the ISO and Unicode Consortium. A Private Use Area (PUA) is one of several ranges which are reserved for private use. For this range, the Unicode standard does not specify any characters.
The Basic Multilingual Plane includes a PUA in the range from U+E000 to U+F8FF (57344–63743). Plane Fifteen (U+F0000 to U+FFFFD), and Plane Sixteen (U+100000 to U+10FFFD) are completely reserved for private use as well.
The use of the PUA was a concept inherited from certain Asian encoding systems. These systems had private use areas to encode Japanese Gaiji (rare personal name characters) in application-specific ways. Similarly the ConScript Unicode Registry aims to coordinate the mapping of scripts not yet encoded in or rejected by Unicode in the PUAs. The Medieval Unicode Font Initiative uses the PUA to encode various ligatures, precomposed characters, and symbols found in medieval texts.
One example of usage of the Private Use Area is Apple's usage of U+F8FF for the Apple logo.
In Microsoft Windows, these character can be created using Private Character Editor, a limited font editor that comes with Windows.
Special code points
At the simplest level, each character in the UCS represents a code point and a particular semantic function: For graphical characters, the semantic function is often implied by its name, and the script or block it is included within. A graphical character may also have a recommended glyph that helps define the meaning of the character. Ideographs for languages in China, Japan, Korea and Vietnam include many other rich properties that participate in defining the semantic role for a character.
However, the UCS and Unicode designate other code points for other purposes. Those code points may have no or few character properties associated with them.
Surrogates
The 2,048 surrogates are not characters, but are reserved for use in UTF-16 to specify code points outside the Basic Multilingual Plane. They are divided into "high surrogates" (D800–DBFF) and "low surrogates" (DC00–DFFF). In UTF-16, they must always appear in pairs, as a high surrogate followed by a low surrogate, thus using 32 bits to denote one code point.
A surrogate pair denotes the code point
- 1000016 + (H - D80016 ) × 40016 + (L - DC0016)
where H and L are the numeric values of the high and low surrogates respectively.
Since high surrogate values in the range DB80 to DBFF always produce values in the Private Use planes, the high surrogate range can be further divided into (normal) high surrogates (D800–DB7F) and "high private use surrogates" (DB80–DBFF).
Noncharacters
Unicode reserves several code points as noncharacters. These code points are guaranteed to never have a character assigned to them. Software implementations are therefore free to use these code points for internal use. However, these noncharacters should never be included in text interchange between implementations. One inherently useful example of a noncharacter is the code point U+FFFE. This code point has the reverse binary sequence of the byte order mark (U+FEFF). If a stream of text contains this noncharacter, this is a good indication the text has been interpreted with the incorrect endianness.
Summary table of UCS characters assignments
Description of Table Columns and Rows
The following table lists all of the blocks currently assigned characters as of April 2007 (Unicode 5.0). Blocks are grouped according to their function.
- The first column lists the name of the group.
Working backwards:
- The last four columns indicate the boundaries of the block — both its starting code point and ending code point — in both hexdecimal and decimal notation.
- The prior column (labeed seq for sequence) indicates the order of the block in code point order. This sequence is jusst an ordering based on the current block assignments. As new blocks are assigned or broken down from the existing unassigned blocks, those sequences numbers would change (though the order would remain the same).
- Unalloc'd indicates the number of unallocated code points represented by a potential block.
- Alloc'd indicates the number of code points allocated to the block whether actually provisioned or reserved for potential future use of the block.
- Exxcl and Incl: Some bblocks contain unrelated characters best treated within other categories. In this casse the characters are all tallied in one place in terms of the allocated and reserved characters. The other unrelated characters are subtracted ('Excl ) from the present row and added (Incl) to another.
- Resrvd indicates the number of characters assigned to the block for related characters, but not yet assigned.
- Provd indicates the characters provisioned in the block: those actually assigned characters (Allocd − Excl + Incl − Resrvd; = Provd).
- Compat indicates the number of characters in the block considered compatibility characters. The issue of compatibility characters is complicated, however they generally represent characters included for compatibility with legacy text processing systems or legacy character sets. Unicode’s separation of glyph from character implies that far fewer characters are required for text processing. The various variant glyphs are instead stored as font data, rather than stored as text data (see Unicode compatibility character section). Compatibility characters are typically ligatures such as ffi or precomposed diacritic letters such as å.
- Core indicates the number of provisioned characters in the block less the discouraged compatibility characters and deprecated (strongly discouraged) characters.
Though the table name unallocated blocks, those blocks could potentially be allocated for any purpose. For example unused code point blocks within the general area of the BMP dedicated to Unihan ideographs could instead be allocated to modern scripts. The names merely indicate the general region of the plane in which they are situated.
Totals
| Allocd | Unallocd | Nonchars | Potential Code Points | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Grand Totals | 241,498 + | 872,548 | + 66 | = 1,114,112 | |||||||||||||
| Code Points in Unallocated Planes | 720,874 | ||||||||||||||||
| Non-Characters (Unallocated Planes) | 22 | Unicode designates 32 other non-characters in the Arabic Presentation Forms-A block from U+FDD0 to U+FDEF for a total of 66 noncharacters designated so far. | |||||||||||||||
| Non-Characters (Planes in Current Use) | 12 | ||||||||||||||||
| Non-Characters in BMP Arabic Presentation Forms-A block | -32 | 32 | |||||||||||||||
| Private Use Allocation | 137,468 | ||||||||||||||||
| Surrogates | 2,048 | ||||||||||||||||
| 93,978 | 5,349 | 99,327 | 2,684 | 155 | 155 | 102,014 | 151,674 | ||||||||||
| Core | Compat | Provd. | Resrvd | Incl. | Excl. | Allocd | Unallocd | ||||||||||
Modern Scripts
| Script-Block Name | Core | Compat | Provd. | Resrvd | Incl. | Excl. | Allocd | Unallocd | Seq | Hex Start |
Hex End |
Dec Start |
Dec End |
|||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| A | Modern Scripts | 17,722 | 999 | 18,721 | 1,454 | 14 | 79 | 20,240 | 944 | (sequences 1-4, 8-55, 99-100, 102-103, 105, 111-112) | ||||||
| 1 | Arabic Blocks | 253 | 12 | 265 | 39 | 304 | 0 | (sequences 13 & 15) | ||||||||
| 1.1 | Arabic | 223 | 12 | 235 | 21 | 256 | 13 | 0600 | 06FF | 1536 | 1791 | |||||
| 1.2 | Arabic Supplement | 30 | 30 | 18 | 48 | 15 | 0750 | 077F | 1872 | 1919 | ||||||
| 2 | Armenian | 85 | 1 | 86 | 10 | 96 | 11 | 0530 | 058F | 1328 | 1423 | |||||
| 3 | Balinese | 110 | 11 | 121 | 7 | 128 | 54 | 1B00 | 1B7F | 6912 | 7039 | |||||
| 4 | Bengali | 87 | 5 | 92 | 36 | 128 | 20 | 0980 | 09FF | 2432 | 2559 | |||||
| 5 | Bopomofo Blocks | 64 | 0 | 64 | 16 | 80 | (sequences 100, 103) | |||||||||
| 5.1 | Bopomofo | 40 | 40 | 8 | 48 | 100 | 3100 | 312F | 12544 | 12591 | ||||||
| 5.2 | Bopomofo Extended | 24 | 24 | 8 | 32 | 103 | 31A0 | 31BF | 12704 | 12735 | ||||||
| 6 | Buginese | 30 | 30 | 2 | 32 | 52 | 1A00 | 1A1F | 6656 | 6687 | ||||||
| 7 | Buhid | 20 | 20 | 12 | 32 | 43 | 1740 | 175F | 5952 | 5983 | ||||||
| 8 | Cherokee | 85 | 85 | 11 | 96 | 37 | 13A0 | 13FF | 5024 | 5119 | ||||||
| 9 | Coptic Blocks | 128 | 0 | 128 | 14 | 14 | 128 | (sequences 8, 87) | ||||||||
| 9.1 | Coptic | 114 | 114 | 14 | 128 | 87 | 2C80 | 2CFF | 11392 | 11519 | ||||||
| 9.2 | Coptic, Greek and | 14 | 14 | 0 | 14 | 0 | 8 | 0370 | 03FF | 880 | 1023 | |||||
| 10 | Cyrillic Blocks | 223 | 52 | 275 | 29 | 304 | (sequences 9-10) | |||||||||
| 10.1 | Cyrillic | 203 | 52 | 255 | 1 | 256 | 9 | 0400 | 04FF | 1024 | 1279 | |||||
| 10.2 | Cyrillic Supplement | 20 | 20 | 28 | 48 | 10 | 0500 | 052F | 1280 | 1327 | ||||||
| 11 | Devanagari | 99 | 11 | 110 | 18 | 128 | 19 | 0900 | 097F | 2304 | 2431 | |||||
| 12 | Ethiopic Blocks | 461 | 0 | 461 | 51 | 512 | (sequences 33, 88) | |||||||||
| 13.1 | Ethiopic | 356 | 356 | 28 | 384 | 35 | 1200 | 137F | 4608 | 4991 | ||||||
| 13.2 | Ethiopic Extended | 79 | 79 | 17 | 96 | 90 | 2D80 | 2DDF | 11648 | 11743 | ||||||
| 13.3 | Ethiopic Supplement | 26 | 26 | 6 | 32 | 36 | 1380 | 139F | 4992 | 5023 | ||||||
| 14 | Georgian Blocks | 121 | 0 | 121 | 23 | 144 | ||||||||||
| 14.1 | Georgian | 83 | 83 | 13 | 96 | 33 | 10A0 | 10FF | 4256 | 4351 | ||||||
| 14.2 | Georgian Supplement | 38 | 38 | 10 | 48 | 88 | 2D00 | 2D2F | 11520 | 11567 | ||||||
| 15 | Glagolitic | 94 | 94 | 2 | 96 | 85 | 2C00 | 2C5F | 11264 | 11359 | ||||||
| 16 | Greek Blocks | 79 | 267 | 346 | 40 | 14 | 400 | (sequences 33, 88) | ||||||||
| 16.1 | Greek and Coptic | 79 | 34 | 113 | 17 | 14 | 144 | 8 | 0370 | 03FF | 880 | 1023 | ||||
| 16.2 | Greek Extended | 0 | 233 | 233 | 23 | 256 | 60 | 1F00 | 1FFF | 7936 | 8191 | |||||
| 17 | Gujarati | 84 | 84 | 44 | 128 | 22 | 0A80 | 0AFF | 2688 | 2815 | ||||||
| 18 | Gurmukhi | 72 | 6 | 78 | 50 | 128 | 21 | 0A00 | 0A7F | 2560 | 2687 | |||||
| 19 | Hangul Blocks | 11,412 | 0 | 11,412 | ||||||||||||