什么是UTF8编码？

Posted 2023-03-24

UTF-8
一种字符集

介绍UTF-8编码规则

首先 UCS 和 Unicode 只是分配整数给字符的编码表. 现在存在好几种将一串字符表示为一串字节的方法. 最显而易见的两种方法是将 Unicode 文本存储为 2 个 或 4 个字节序列的串. 这两种方法的正式名称分别为 UCS-2 和 UCS-4. 除非另外指定, 否则大多数的字节都是这样的(Bigendian convention). 将一个 ASCII 或 Latin-1 的文件转换成 UCS-2 只需简单地在每个 ASCII 字节前插入 0x00. 如果要转换成 UCS-4, 则必须在每个 ASCII 字节前插入三个 0x00.

在 Unix 下使用 UCS-2 (或 UCS-4) 会导致非常严重的问题. 用这些编码的字符串会包含一些特殊的字符, 比如 '' 或 '/', 它们在 文件名和其他 C 库函数参数里都有特别的含义. 另外, 大多数使用 ASCII 文件的 UNIX 下的工具, 如果不进行重大修改是无法读取 16 位的字符的. 基于这些原因, 在文件名, 文本文件, 环境变量等地方, UCS-2 不适合作为 Unicode 的外部编码. 参考技术A UTF-8（8位元，Universal Character Set/Unicode Transformation Format）是针对Unicode的一种可变长度字符编码。它可以用来表示Unicode标准中的任何字符，而且其编码中的第一个字节仍与ASCII相容，使得原来处理ASCII字符的软件无须或只进行少部份修改后，便可继续使用。因此，它逐渐成为电子邮件、网页及其他存储或传送文字的应用中，优先采用的编码。 参考技术B UTF8是针对Unicode的一种可变长度字符编码，可以用来表示Unicode标准中的任何字符，适用性广泛。 参考技术C 在UTF-8中,字符是以8位序列来编码的,用一个或几个字节来表示一个字符。这种方式的最大好处,是UTF-8保留了ASCII字符的编码做为它的一部分。UTF-8 编码是用以解决国际上字符的一种多字节编码,它对...
2.
UTF-16以16位为单元对UCS进行编码。对于小于0x10000的UCS码,UTF-16编码就等于UCS码对应的16位无符号...
3.
GBK是在国家标准GB2312基础上扩容后兼容GB2312的标准(好像还不是国家标准)... 参考技术D 一种代码格式。多用于编码。编程。

关于 UTF7 和 UTF8编码的问题

在用到C#中的Encoding类库的时候遇到这两个编码，不懂它们到底是怎么进行编码的，有知道的可以介绍下吗？

UTF8其实和Unicode是同类，就是在编码方式上不同！
首先UTF8编码后的大小是不一定，不像Unicode编码后的大小是一样的！

我们先来看Unicode的编码：一个英文字母 “a” 和 一个汉字 “好”，编码后都是占用的空间大小是一样的，都是两个字节！

而UTF8编码：一个英文字母“a” 和 一个汉字 “好”，编码后占用的空间大小就不样了，前者是一个字节，后者是三个字节！

现在就让我们来看看UTF8编码的原理吧：
因为一个字母还有一些键盘上的符号加起来只用二进制七位就可以表示出来，而一个字节就是八位，所以UTF8就用一个字节来表式字母和一些键盘上的符号。然而当我们拿到被编码后的一个字节后怎么知道它的组成？它有可能是英文字母的一个字节，也有可能是汉字的三个字节中的一个字节！所以，UTF8是有标志位的！

当要表示的内容是 7位 的时候就用一个字节：0******* 第一个0为标志位，剩下的空间正好可以表示ASCII 0－127 的内容。

当要表示的内容在 8 到 11 位的时候就用两个字节：110***** 10****** 第一个字节的110和第二个字节的10为标志位。

当要表示的内容在 12 到 16 位的时候就用三个字节：1110***** 10****** 10****** 和上面一样，第一个字节的1110和第二、三个字节的10都是标志位，剩下的占湔�每梢员硎竞鹤帧?BR>
以此类推：
四个字节：11110**** 10****** 10****** 10******
五个字节：111110*** 10****** 10****** 10****** 10******
六个字节：1111110** 10****** 10****** 10****** 10****** 10******

UTF-7：A Mail-Safe Transformation Format of Unicode(RFC1642)。这是一种使用 7 位 ASCII 码对 Unicode 码进行转换的编码。它的设计目的仍然是为了在只能传递 7 为编码的邮件网关中传递信息。 UTF-7 对英语字母、数字和常见符号直接显示，而对其他符号用修正的 Base64 编码。符号 + 和 - 号控制编码过程的开始和暂停。所以乱码中如果夹有英文单词，并且相伴有 + 号和 - 号，这就有可能是 UTF-7 编码。

关于UTF7的更多资料如下（都是英语的，如果想具体了解再看）：
UTF-7

A Mail-Safe Transformation Format of Unicode

Status of this Memo

This memo provides information for the Internet community. This memo
does not specify an Internet standard of any kind. Distribution of
this memo is unlimited.

Abstract

The Unicode Standard, version 2.0, and ISO/IEC 10646-1:1993(E) (as
amended) jointly define a character set (hereafter referred to as
Unicode) which encompasses most of the world's writing systems.
However, Internet mail (STD 11, RFC 822) currently supports only 7-
bit US ASCII as a character set. MIME (RFC 2045 through 2049) extends
Internet mail to support different media types and character sets,
and thus could support Unicode in mail messages. MIME neither defines
Unicode as a permitted character set nor specifies how it would be
encoded, although it does provide for the registration of additional
character sets over time.

This document describes a transformation format of Unicode that
contains only 7-bit ASCII octets and is intended to be readable by
humans in the limiting case that the document consists of characters
from the US-ASCII repertoire. It also specifies how this
transformation format is used in the context of MIME and RFC 1641,
"Using Unicode with MIME".

Motivation

Although other transformation formats of Unicode exist and could
conceivably be used in this context (most notably UTF-8, also known
as UTF-2 or UTF-FSS), they suffer the disadvantage that they use
octets in the range decimal 128 through 255 to encode Unicode
characters outside the US-ASCII range. Thus, in the context of mail,
those octets must themselves be encoded. This requires putting text
through two successive encoding processes, and leads to a significant
expansion of characters outside the US-ASCII range, putting non-
English speakers at a disadvantage. For example, using UTF-8 together

with the Quoted-Printable content transfer encoding of MIME
represents US-ASCII characters in one octet, but other characters may
require up to nine octets.

Overview

UTF-7 encodes Unicode characters as US-ASCII octets, together with
shift sequences to encode characters outside that range. For this
purpose, one of the characters in the US-ASCII repertoire is reserved
for use as a shift character.

Many mail gateways and systems cannot handle the entire US-ASCII
character set (those based on EBCDIC, for example), and so UTF-7
contains provisions for encoding characters within US-ASCII in a way
that all mail systems can accomodate.

UTF-7 should normally be used only in the context of 7 bit
transports, such as mail. In other contexts, straight Unicode or
UTF-8 is preferred.

See RFC 1641, "Using Unicode with MIME" for the overall specification
on usage of Unicode transformation formats with MIME.

Definitions

First, the definition of Unicode:

The 16 bit character set Unicode is defined by "The Unicode
Standard, Version 2.0". This character set is identical with the
character repertoire and coding of the international standard
ISO/IEC 10646-1:1993(E); Coded Representation Form=UCS-2;
Subset=300; Implementation Level=3, including the first 7
amendments to 10646 plus editorial corrections.

Note. Unicode 2.0 further specifies the use and interaction of
these character codes beyond the ISO standard. However, any valid
10646 sequence is a valid Unicode sequence, and vice versa;
Unicode supplies interpretations of sequences on which the ISO
standard is silent as to interpretation.

Next, some handy definitions of US-ASCII character subsets:

Set D (directly encoded characters) consists of the following
characters (derived from RFC 1521, Appendix B, which no longer
appears in RFC 2045): the upper and lower case letters A through Z
and a through z, the 10 digits 0-9, and the following nine special
characters (note that "+" and "=" are omitted):

Character ASCII & Unicode Value (decimal)
' 39
( 40
) 41
, 44
- 45
. 46
/ 47
: 58
? 63

Set O (optional direct characters) consists of the following
characters (note that "\" and "~" are omitted):

Character ASCII & Unicode Value (decimal)
! 33
" 34
# 35
$ 36
% 37
& 38
* 42
; 59
< 60
= 61
> 62
@ 64
[ 91
] 93
^ 94
_ 95
' 96
123
| 124
125

Rationale. The characters "\" and "~" are omitted because they are
often redefined in variants of ASCII.

Set B (Modified Base 64) is the set of characters in the Base64
alphabet defined in RFC 2045, excluding the pad character "="
(decimal value 61).

Rationale. The pad character = is excluded because UTF-7 is designed
for use within header fields as set forth in RFC 2047. Since the only
readable encoding in RFC 2047 is "Q" (based on RFC 2045's Quoted-
Printable), the "=" character is not available for use (without a lot
of escape sequences). This was very unfortunate but unavoidable. The
"=" character could otherwise have been used as the UTF-7 escape
character as well (rather than using "+").

Note that all characters in US-ASCII have the same value in Unicode
when zero-extended to 16 bits.

UTF-7 Definition

A UTF-7 stream represents 16-bit Unicode characters using 7-bit US-
ASCII octets as follows:

Rule 1: (direct encoding) Unicode characters in set D above may be
encoded directly as their ASCII equivalents. Unicode characters in
Set O may optionally be encoded directly as their ASCII
equivalents, bearing in mind that many of these characters are
illegal in header fields, or may not pass correctly through some
mail gateways.

Rule 2: (Unicode shifted encoding) Any Unicode character sequence
may be encoded using a sequence of characters in set B, when
preceded by the shift character "+" (US-ASCII character value
decimal 43). The "+" signals that subsequent octets are to be
interpreted as elements of the Modified Base64 alphabet until a
character not in that alphabet is encountered. Such characters
include control characters such as carriage returns and line
feeds; thus, a Unicode shifted sequence always terminates at the
of a line. As a special case, if the sequence terminates with the
character "-" (US-ASCII decimal 45) then that character is
absorbed; other terminating characters are not absorbed and are
processed normally.

Note that if the first character after the shifted sequence is "-"
then an extra "-" must be present to terminate the shifted
sequence so that the actual "-" is not itself absorbed.

Rationale. A terminating character is necessary for cases where
the next character after the Modified Base64 sequence is part of
character set B or is itself the terminating character. It can
also enhance readability by delimiting encoded sequences.

Also as a special case, the sequence "+-" may be used to encode
the character "+". A "+" character followed immediately by any
character other than members of set B or "-" is an ill-formed
sequence.

Unicode is encoded using Modified Base64 by first converting
Unicode 16-bit quantities to an octet stream (with the most
significant octet first). Surrogate pairs (UTF-16) are converted
by treating each half of the pair as a separate 16 bit quantity
(i.e., no special treatment). Text with an odd number of octets is
ill-formed. ISO 10646 characters outside the range addressable via
surrogate pairs cannot be encoded.

Rationale. ISO/IEC 10646-1:1993(E) specifies that when characters
the UCS-2 form are serialized as octets, that the most significant
octet appear first. This is also in keeping with common network
practice of choosing a canonical format for transmission.

Rationale. The policy for code point allocation within ISO 10646
and Unicode is that the repertoires be kept synchronized. No code
points will be allocated in ISO 10646 outside the range
addressable by surrogate pairs.

Next, the octet stream is encoded by applying the Base64 content
transfer encoding algorithm as defined in RFC 2045, modified to
omit the "=" pad character. Instead, when encoding, zero bits are
added to pad to a Base64 character boundary. When decoding, any
bits at the end of the Modified Base64 sequence that do not
constitute a complete 16-bit Unicode character are discarded. If
such discarded bits are non-zero the sequence is ill-formed.

Rationale. The pad character "=" is not used when encoding
Modified Base64 because of the conflict with its use as an escape
character for the Q content transfer encoding in RFC 2047 header
fields, as mentioned above.

Rule 3: The space (decimal 32), tab (decimal 9), carriage return
(decimal 13), and line feed (decimal 10) characters may be
directly represented by their ASCII equivalents. However, note
that MIME content transfer encodings have rules concerning the use
of such characters. Usage that does not conform to the
restrictions of RFC 822, for example, would have to be encoded
using MIME content transfer encodings other than 7bit or 8bit,
such as quoted-printable, binary, or base64.

Given this set of rules, Unicode characters which may be encoded via
rules 1 or 3 take one octet per character, and other Unicode
characters are encoded on average with 2 2/3 octets per character

plus one octet to switch into Modified Base64 and an optional octet
to switch out.

Example. The Unicode sequence "A<NOT IDENTICAL TO><ALPHA>."
(hexadecimal 0041,2262,0391,002E) may be encoded as follows:

A+ImIDkQ.

Example. The Unicode sequence "Hi Mom -<WHITE SMILING FACE>-!"
(hexadecimal 0048, 0069, 0020, 004D, 006F, 006D, 0020, 002D, 263A,
002D, 0021) may be encoded as follows:

Hi Mom -+Jjo--!

Example. The Unicode sequence representing the Han characters for
the Japanese word "nihongo" (hexadecimal 65E5,672C,8A9E) may be
encoded as follows:

+ZeVnLIqe- 参考技术A UTF7
用于处理普通 ASCII 字符 , 主要普通 ASCII 的二进制编码最高位一般为 0. 在 UTF7 下处理 ASCII 字符的时候需要处理补位 . 补位规则 : 将后一字节的从尾取位补在前一字节的头 , 补满 8 位 .
UTF8
主要用于处理图片和铃声