Java检测文件是否UTF8编码

Posted zolo®

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Java检测文件是否UTF8编码相关的知识，希望对你有一定的参考价值。

介绍UTF-8编码规则

  1. UTF-8 编码字符理论上可以最多到 6 个字节长, 然而 16 位 BMP 字符最多只用到 3 字节长. Bigendian UCS-4 字节串的排列顺序是预定的.
  2. 字节 0xFE 和 0xFF 在 UTF-8 编码中从未用到.
  3. 下列字节串用来表示一个字符. 用到哪个串取决于该字符在 Unicode 中的序号.
  4. U-00000000 - U-0000007F: 0xxxxxxx
  5. U-00000080 - U-000007FF: 110xxxxx 10xxxxxx
  6. U-00000800 - U-0000FFFF: 1110xxxx 10xxxxxx 10xxxxxx
  7. U-00010000 - U-001FFFFF: 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
  8. U-00200000 - U-03FFFFFF: 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
  9. U-04000000 - U-7FFFFFFF: 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
  11. xxx 的位置由字符编码数的二进制表示的位填入. 越靠右的 x 具有越少的特殊意义. 只用最短的那个足够表达一个字符编码数的多字节串. 注意在多字节串中, 第一个字节的开头"1"的数目就是整个串中字节的数目.
  12. 例如: Unicode 字符 U+00A9 = 1010 1001 (版权符号) 在 UTF-8 里的编码为:
  13. 11000010 10101001 = 0xC2 0xA9
  14. 而字符 U+2260 = 0010 0010 0110 0000 (不等于) 编码为:
  15. 11100010 10001001 10100000 = 0xE2 0x89 0xA0
  17. 特殊规则: 文件头三个字节用16进制表示是EFBBBF, 此规则不通用, 由编辑工具定义.
  19. 这种编码的官方名字拼写为 UTF-8, 其中 UTF 代表 UCS Transformation Format. 请勿在任何文档中用其他名字 (比如 utf8 或 UTF_8) 来表示 UTF-8, 当然除非你指的是一个变量名而不是这种编码本身.

复制代码

源码实现:

  1. package com.yy.game.test;
  3. import java.io.BufferedInputStream;
  4. import java.io.File;
  5. import java.io.FileInputStream;
  6. import java.io.IOException;
  7. import java.io.InputStream;
  8. import java.nio.CharBuffer;
  9. import java.nio.MappedByteBuffer;
  10. import java.nio.channels.FileChannel;
  11. import java.nio.channels.FileChannel.MapMode;
  12. import java.nio.charset.Charset;
  13. import java.nio.charset.CharsetDecoder;
  14. import java.nio.charset.CoderResult;
  16. public class UTF8Checker {
  18.         public static void main(String[] args) throws IOException {
  19.                 File dir = new File("F:\\test");
  20.                 for (File file : dir.listFiles()) {
  21.                         System.out.format("%s: %s, %s%n", file, check(file), check2(file));
  22.                 }
  23.         }
  25.         /**
  26.          * JDK自带API实现
  27.          */
  28.         @SuppressWarnings("resource")
  29.         public static boolean check2(File file) throws IOException {
  30.                 long start = System.nanoTime();
  31.                 FileChannel fc = null;
  32.                 try {
  34.                         fc = new FileInputStream(file).getChannel();
  36.                         MappedByteBuffer buf = fc.map(MapMode.READ_ONLY, 0, fc.size());
  38.                         Charset utf8 = Charset.forName("UTF-8");
  39.                         CharsetDecoder decoder = utf8.newDecoder();
  40.                         CharBuffer cbuf = CharBuffer.allocate((int) (buf.limit() * decoder.averageCharsPerByte()));
  41.                         CoderResult result = decoder.decode(buf, cbuf, true);
  42.                         return !result.isError();
  43.                 } finally {
  44.                         if (fc != null) {
  45.                                 fc.close();
  46.                         }
  47.                         long end = System.nanoTime();
  48.                         System.out.println("used(ns):" + (end - start));
  49.                 }
  51.         }
  53.         /**
  54.          * 自定义实现
  55.          */
  56.         public static boolean check(File file) throws IOException {
  57.                 long start = System.nanoTime();
  58.                 InputStream in = null;
  59.                 try {
  60.                         in = new BufferedInputStream(new FileInputStream(file));
  61.                         StreamBuffer sbuf = new StreamBuffer(in, 1024);
  62.                         if (sbuf.next() == 0xEF && sbuf.next() == 0xBB && sbuf.next() == 0xBF) {
  63.                                 return true;
  64.                         }
  65.                         sbuf.redo();
  66.                         // 1. U-00000000 - U-0000007F: 0xxxxxxx
  67.                         // 2. U-00000080 - U-000007FF: 110xxxxx 10xxxxxx
  68.                         // 3. U-00000800 - U-0000FFFF: 1110xxxx 10xxxxxx 10xxxxxx
  69.                         // 4. U-00010000 - U-001FFFFF: 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
  70.                         // 5. U-00200000 - U-03FFFFFF: 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
  71.                         // 6. U-04000000 - U-7FFFFFFF: 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
  72.                         for (int ch = 0; (ch = sbuf.next()) != -1;) {
  73.                                 int n = 0;
  74.                                 if (ch <= 0x7F) {
  75.                                         n = 1;
  76.                                 } else if (ch <= 0xBF) {
  77.                                         return false;
  78.                                 } else if (ch <= 0xDF) {
  79.                                         n = 2;
  80.                                 } else if (ch <= 0xEF) {
  81.                                         n = 3;
  82.                                 } else if (ch <= 0xF7) {
  83.                                         n = 4;
  84.                                 } else if (ch <= 0xFB) {
  85.                                         n = 5;
  86.                                 } else if (ch <= 0xFD) {
  87.                                         n = 6;
  88.                                 } else {
  89.                                         return false;
  90.                                 }
  91.                                 while (--n > 0) {
  92.                                         if ((sbuf.next() & 0x80) != 0x80) {
  93.                                                 return false;
  94.                                         }
  95.                                 }
  96.                         }
  97.                         return true;
  98.                 } finally {
  99.                         if (in != null) {
  100.                                 in.close();
  101.                         }
  102.                         long end = System.nanoTime();
  103.                         System.out.println("used(ns):" + (end - start));
  104.                 }
  106.         }
  108.         static class StreamBuffer {
  110.                 final InputStream in;
  111.                 final byte[] buf;
  112.                 int pos = -1;// 初始值为-1,表示指针尚未移动.
  113.                 int len;
  115.                 public StreamBuffer(InputStream in, int size) {
  116.                         this.in = in;
  117.                         if (size < 3) {
  118.                                 size = 3;
  119.                         }
  120.                         this.buf = new byte[size];
  121.                 }
  123.                 public void redo() {
  124.                         this.pos = 0;
  125.                 }
  127.                 public int next() throws IOException {
  128.                         if (len > 0 || pos < 0) {
  129.                                 if (++pos == len) {
  130.                                         if ((len = in.read(buf)) == 0) {
  131.                                                 return -1;
  132.                                         }
  133.                                         pos = 0;
  134.                                 }
  135.                                 return this.buf[this.pos] & 0xFF;
  136.                         } else {
  137.                                 return -1;
  138.                         }
  139.                 }
  141.         }
  143. }

复制代码

在本机测试, JDK原生API需要创建CharBuffer,性能明显慢了25%以上.

  1. used(ns):472420
  2. used(ns):4490075
  3. F:\test\b334d5fd-b8a7-48f4-9099-f6011c7e5a48.sql: true, true
  4. used(ns):122515
  5. used(ns):343490
  6. F:\test\b334d5fd-b8a7-48f4-9099-f6011c7e5a482.sql: false, false
  7. used(ns):55164
  8. used(ns):82425
  9. F:\test\test.sql: false, false

复制代码

以上是关于Java检测文件是否UTF8编码的主要内容，如果未能解决你的问题，请参考以下文章

检测字节流是否是UTF8编码

如何检测上传的csv文件的编码

怎么转换为utf8有BOM编码

iOS小技能：URL编码

iOS小技能：URL编码

iOS小技能：URL编码