关于 UTF7 和 UTF8编码的问题

Posted 2023-04-07

在用到C#中的Encoding类库的时候遇到这两个编码，不懂它们到底是怎么进行编码的，有知道的可以介绍下吗？

UTF8其实和Unicode是同类，就是在编码方式上不同！
首先UTF8编码后的大小是不一定，不像Unicode编码后的大小是一样的！

我们先来看Unicode的编码：一个英文字母 “a” 和 一个汉字 “好”，编码后都是占用的空间大小是一样的，都是两个字节！

而UTF8编码：一个英文字母“a” 和 一个汉字 “好”，编码后占用的空间大小就不样了，前者是一个字节，后者是三个字节！

现在就让我们来看看UTF8编码的原理吧：
因为一个字母还有一些键盘上的符号加起来只用二进制七位就可以表示出来，而一个字节就是八位，所以UTF8就用一个字节来表式字母和一些键盘上的符号。然而当我们拿到被编码后的一个字节后怎么知道它的组成？它有可能是英文字母的一个字节，也有可能是汉字的三个字节中的一个字节！所以，UTF8是有标志位的！

当要表示的内容是 7位 的时候就用一个字节：0******* 第一个0为标志位，剩下的空间正好可以表示ASCII 0－127 的内容。

当要表示的内容在 8 到 11 位的时候就用两个字节：110***** 10****** 第一个字节的110和第二个字节的10为标志位。

当要表示的内容在 12 到 16 位的时候就用三个字节：1110***** 10****** 10****** 和上面一样，第一个字节的1110和第二、三个字节的10都是标志位，剩下的占湔�每梢员硎竞鹤帧?BR>
以此类推：
四个字节：11110**** 10****** 10****** 10******
五个字节：111110*** 10****** 10****** 10****** 10******
六个字节：1111110** 10****** 10****** 10****** 10****** 10******

UTF-7：A Mail-Safe Transformation Format of Unicode(RFC1642)。这是一种使用 7 位 ASCII 码对 Unicode 码进行转换的编码。它的设计目的仍然是为了在只能传递 7 为编码的邮件网关中传递信息。 UTF-7 对英语字母、数字和常见符号直接显示，而对其他符号用修正的 Base64 编码。符号 + 和 - 号控制编码过程的开始和暂停。所以乱码中如果夹有英文单词，并且相伴有 + 号和 - 号，这就有可能是 UTF-7 编码。

关于UTF7的更多资料如下（都是英语的，如果想具体了解再看）：
UTF-7

A Mail-Safe Transformation Format of Unicode

Status of this Memo

This memo provides information for the Internet community. This memo
does not specify an Internet standard of any kind. Distribution of
this memo is unlimited.

Abstract

The Unicode Standard, version 2.0, and ISO/IEC 10646-1:1993(E) (as
amended) jointly define a character set (hereafter referred to as
Unicode) which encompasses most of the world's writing systems.
However, Internet mail (STD 11, RFC 822) currently supports only 7-
bit US ASCII as a character set. MIME (RFC 2045 through 2049) extends
Internet mail to support different media types and character sets,
and thus could support Unicode in mail messages. MIME neither defines
Unicode as a permitted character set nor specifies how it would be
encoded, although it does provide for the registration of additional
character sets over time.

This document describes a transformation format of Unicode that
contains only 7-bit ASCII octets and is intended to be readable by
humans in the limiting case that the document consists of characters
from the US-ASCII repertoire. It also specifies how this
transformation format is used in the context of MIME and RFC 1641,
"Using Unicode with MIME".

Motivation

Although other transformation formats of Unicode exist and could
conceivably be used in this context (most notably UTF-8, also known
as UTF-2 or UTF-FSS), they suffer the disadvantage that they use
octets in the range decimal 128 through 255 to encode Unicode
characters outside the US-ASCII range. Thus, in the context of mail,
those octets must themselves be encoded. This requires putting text
through two successive encoding processes, and leads to a significant
expansion of characters outside the US-ASCII range, putting non-
English speakers at a disadvantage. For example, using UTF-8 together

with the Quoted-Printable content transfer encoding of MIME
represents US-ASCII characters in one octet, but other characters may
require up to nine octets.

Overview

UTF-7 encodes Unicode characters as US-ASCII octets, together with
shift sequences to encode characters outside that range. For this
purpose, one of the characters in the US-ASCII repertoire is reserved
for use as a shift character.

Many mail gateways and systems cannot handle the entire US-ASCII
character set (those based on EBCDIC, for example), and so UTF-7
contains provisions for encoding characters within US-ASCII in a way
that all mail systems can accomodate.

UTF-7 should normally be used only in the context of 7 bit
transports, such as mail. In other contexts, straight Unicode or
UTF-8 is preferred.

See RFC 1641, "Using Unicode with MIME" for the overall specification
on usage of Unicode transformation formats with MIME.

Definitions

First, the definition of Unicode:

The 16 bit character set Unicode is defined by "The Unicode
Standard, Version 2.0". This character set is identical with the
character repertoire and coding of the international standard
ISO/IEC 10646-1:1993(E); Coded Representation Form=UCS-2;
Subset=300; Implementation Level=3, including the first 7
amendments to 10646 plus editorial corrections.

Note. Unicode 2.0 further specifies the use and interaction of
these character codes beyond the ISO standard. However, any valid
10646 sequence is a valid Unicode sequence, and vice versa;
Unicode supplies interpretations of sequences on which the ISO
standard is silent as to interpretation.

Next, some handy definitions of US-ASCII character subsets:

Set D (directly encoded characters) consists of the following
characters (derived from RFC 1521, Appendix B, which no longer
appears in RFC 2045): the upper and lower case letters A through Z
and a through z, the 10 digits 0-9, and the following nine special
characters (note that "+" and "=" are omitted):

Character ASCII & Unicode Value (decimal)
' 39
( 40
) 41
, 44
- 45
. 46
/ 47
: 58
? 63

Set O (optional direct characters) consists of the following
characters (note that "\" and "~" are omitted):

Character ASCII & Unicode Value (decimal)
! 33
" 34
# 35
$ 36
% 37
& 38
* 42
; 59
< 60
= 61
> 62
@ 64
[ 91
] 93
^ 94
_ 95
' 96
123
| 124
125

Rationale. The characters "\" and "~" are omitted because they are
often redefined in variants of ASCII.

Set B (Modified Base 64) is the set of characters in the Base64
alphabet defined in RFC 2045, excluding the pad character "="
(decimal value 61).

Rationale. The pad character = is excluded because UTF-7 is designed
for use within header fields as set forth in RFC 2047. Since the only
readable encoding in RFC 2047 is "Q" (based on RFC 2045's Quoted-
Printable), the "=" character is not available for use (without a lot
of escape sequences). This was very unfortunate but unavoidable. The
"=" character could otherwise have been used as the UTF-7 escape
character as well (rather than using "+").

Note that all characters in US-ASCII have the same value in Unicode
when zero-extended to 16 bits.

UTF-7 Definition

A UTF-7 stream represents 16-bit Unicode characters using 7-bit US-
ASCII octets as follows:

Rule 1: (direct encoding) Unicode characters in set D above may be
encoded directly as their ASCII equivalents. Unicode characters in
Set O may optionally be encoded directly as their ASCII
equivalents, bearing in mind that many of these characters are
illegal in header fields, or may not pass correctly through some
mail gateways.

Rule 2: (Unicode shifted encoding) Any Unicode character sequence
may be encoded using a sequence of characters in set B, when
preceded by the shift character "+" (US-ASCII character value
decimal 43). The "+" signals that subsequent octets are to be
interpreted as elements of the Modified Base64 alphabet until a
character not in that alphabet is encountered. Such characters
include control characters such as carriage returns and line
feeds; thus, a Unicode shifted sequence always terminates at the
of a line. As a special case, if the sequence terminates with the
character "-" (US-ASCII decimal 45) then that character is
absorbed; other terminating characters are not absorbed and are
processed normally.

Note that if the first character after the shifted sequence is "-"
then an extra "-" must be present to terminate the shifted
sequence so that the actual "-" is not itself absorbed.

Rationale. A terminating character is necessary for cases where
the next character after the Modified Base64 sequence is part of
character set B or is itself the terminating character. It can
also enhance readability by delimiting encoded sequences.

Also as a special case, the sequence "+-" may be used to encode
the character "+". A "+" character followed immediately by any
character other than members of set B or "-" is an ill-formed
sequence.

Unicode is encoded using Modified Base64 by first converting
Unicode 16-bit quantities to an octet stream (with the most
significant octet first). Surrogate pairs (UTF-16) are converted
by treating each half of the pair as a separate 16 bit quantity
(i.e., no special treatment). Text with an odd number of octets is
ill-formed. ISO 10646 characters outside the range addressable via
surrogate pairs cannot be encoded.

Rationale. ISO/IEC 10646-1:1993(E) specifies that when characters
the UCS-2 form are serialized as octets, that the most significant
octet appear first. This is also in keeping with common network
practice of choosing a canonical format for transmission.

Rationale. The policy for code point allocation within ISO 10646
and Unicode is that the repertoires be kept synchronized. No code
points will be allocated in ISO 10646 outside the range
addressable by surrogate pairs.

Next, the octet stream is encoded by applying the Base64 content
transfer encoding algorithm as defined in RFC 2045, modified to
omit the "=" pad character. Instead, when encoding, zero bits are
added to pad to a Base64 character boundary. When decoding, any
bits at the end of the Modified Base64 sequence that do not
constitute a complete 16-bit Unicode character are discarded. If
such discarded bits are non-zero the sequence is ill-formed.

Rationale. The pad character "=" is not used when encoding
Modified Base64 because of the conflict with its use as an escape
character for the Q content transfer encoding in RFC 2047 header
fields, as mentioned above.

Rule 3: The space (decimal 32), tab (decimal 9), carriage return
(decimal 13), and line feed (decimal 10) characters may be
directly represented by their ASCII equivalents. However, note
that MIME content transfer encodings have rules concerning the use
of such characters. Usage that does not conform to the
restrictions of RFC 822, for example, would have to be encoded
using MIME content transfer encodings other than 7bit or 8bit,
such as quoted-printable, binary, or base64.

Given this set of rules, Unicode characters which may be encoded via
rules 1 or 3 take one octet per character, and other Unicode
characters are encoded on average with 2 2/3 octets per character

plus one octet to switch into Modified Base64 and an optional octet
to switch out.

Example. The Unicode sequence "A<NOT IDENTICAL TO><ALPHA>."
(hexadecimal 0041,2262,0391,002E) may be encoded as follows:

A+ImIDkQ.

Example. The Unicode sequence "Hi Mom -<WHITE SMILING FACE>-!"
(hexadecimal 0048, 0069, 0020, 004D, 006F, 006D, 0020, 002D, 263A,
002D, 0021) may be encoded as follows:

Hi Mom -+Jjo--!

Example. The Unicode sequence representing the Han characters for
the Japanese word "nihongo" (hexadecimal 65E5,672C,8A9E) may be
encoded as follows:

+ZeVnLIqe- 参考技术A UTF7
用于处理普通 ASCII 字符 , 主要普通 ASCII 的二进制编码最高位一般为 0. 在 UTF7 下处理 ASCII 字符的时候需要处理补位 . 补位规则 : 将后一字节的从尾取位补在前一字节的头 , 补满 8 位 .
UTF8
主要用于处理图片和铃声

wchar_t 和编码

【中文标题】wchar_t 和编码

【英文标题】：wchar_t and encoding

【发布时间】：2012-05-13 10:59:36

【问题描述】：

如果我想将一段字符串转换为 UTF-16，比如char * xmlbuffer，是否必须在编码为 UTF-16 之前将类型转换为 wchar_t *？在编码为 UTF-8 之前是否需要 char* 类型？

wchar_t、char 与 UTF-8 或 UTF-16 或 UTF-32 或其他转换格式有何关系？

提前感谢您的帮助！

【参考方案1】：

不，您不必更改数据类型。

关于wchar_t：标准是这样说的

类型 wchar_t 是一个 distinct 类型，其值可以表示 distinct 指定的最大扩展字符集的所有成员的代码 在受支持的语言环境中。

不幸的是，它没有说明wchar_t 应该有什么编码；这取决于实现。所以例如给定

auto s = L"foo";

您绝对不能对表达式 *s 的值做出任何假设。

但是，您可以将std::string 用作不透明的字节序列，以您选择的任何转换格式表示文本，而不会出现问题。只是不要对其执行标准库字符串相关的操作。

【讨论】：

那么我能说在windows平台使用wchar_t转UTF-16只是为了方便的选择，理论上绝对可以用char转UTF-16？

@Hunter：理论上是的，但在 Windows 中，wchar_t 用于 UTF-16，char 用于 ASCII 和 UTF-8。

在 Windows 上，wchar_t 的已知大小为 16 位

@Hunter，如果您在 UTF-16 字符串上调用 strlen，它可能总是返回 0 或 1。strlen 只接受 8 位字符，并且会在第一个字符处停止高字节为 0。

@Mooing Duck：char16_t 更好，但只是最近才添加到 C++ 标准中。

【参考方案2】：

iconv 是一个 POSIX 函数，可以处理中间编码步骤。您可以使用 iconv_open 指定您有 UTF-8 输入并且您想要 UTF-16 输出。然后，使用从iconv_open 返回的句柄，您可以使用iconv（指定您的输入缓冲区和输出缓冲区）。完成后，您必须在从 iconv_open 返回的句柄上调用 iconv_close 以释放资源等。

您必须仔细阅读系统文档，了解iconv 支持哪些编码及其命名方案（即提供iconv_open 的内容）。例如，某些系统上的iconv 需要"utf-8"，而其他系统可能需要"UTF8" 等。

Windows 不提供 iconv 版本，而是提供了自己的 UTF 格式化函数：MultiByteToWideChar 和 WideCharToMultiByte。

//UTF8 to UTF16
std::string input = ...
int utf16len = MultiByteToWideChar(CP_UTF8, 0, input.c_str(), input.size(), 
                                               NULL, 0);
std::wstring output(utf16len);
MultiByteToWideChar(CP_UTF8, 0, input.c_str(), input.size(), 
                                &output[0], output.size());
//UTF16 to UTF8
std::wstring input = ...
int utf8len = WideCharToMultiByte(CP_UTF8, 0, input.c_str(), input.size(), 
                                              NULL, 0, NULL, NULL);
std::string output(utf8len);
WideCharToMultiByte(CP_UTF8, 0, input.c_str(), input.size(),
                                &output[0], output.size(), NULL, NULL);

【讨论】：

Hunter：注意Windows没有自带iconv，但是有办法得到。 @Dreamlax：您介意我们将我的答案作为 Windows 替代方案插入您的答案并删除我的答案吗？使用库的概念是正确的，您对此更清楚。

@MooingDuck：是的，绝对是，听起来是个好主意。将我的放入您的或您的放入我的，无论哪个。

【参考方案3】：

wchar_t 的大小取决于编译器，因此它与各种 unicode 格式的关系会有所不同。