MyCat线程模型分析

Posted 2020-08-04 ZzzCrazyPig

参考MyCat权威指南，对MyCat-Server里面的线程模型做简要分析:

1. 线程模型图

根据MyCat权威指南，做出以下线程模型图:

MyCat的线程模型主要分为三部分（: 网络通讯线程、业务线程和定时任务线程，下面分别介绍这些线程的使用:
【温馨提示】
这里排除JVM自身使用的线程，只关注MyCat服务所使用的线程，如果需要详细了解MyCat里面使用的所有线程，请参考《MyCat权威指南》-> 开发篇 -> MyCat线程模型分析

2. 网络通讯线程

MyCat-Server使用了原生Java NIO和AIO来构建其最重要的网络通讯层，在MyCat-Server启动之前可以通过配置文件来选择使用NIO网络模型或者AIO网络模型。MyCat官方推荐使用NIO模型。所以，这里仅针对NIO网络通讯模型来分析，网络通讯层线程的使用体现在NIOAcceptor、NIOConnector和NIOReactor。

2.1 NIOAcceptor

NIOAcceptor线程负责处理客户端连接MyCat的事件（在NIO网络编程中，对应于NIO服务端），包括默认的8066服务端口（Server）和9066管理端口（Manager）。

NIOAcceptor继承Thread类，在run方法里面处理客户端连接MyCat的连接请求，代码如下所示 :

@Override
    public void run() {
        final Selector tSelector = this.selector;
        for (;;) {
            ++acceptCount;
            try {
                tSelector.select(1000L);
                Set<SelectionKey> keys = tSelector.selectedKeys();
                try {
                    for (SelectionKey key : keys) {
                        if (key.isValid() && key.isAcceptable()) {
                            accept();
                        } else {
                            key.cancel();
                        }
                    }
                } finally {
                    keys.clear();
                }
            } catch (Exception e) {
                LOGGER.warn(getName(), e);
            }
        }
    }

对于来自客户端的连接，将被Selector事件选择器捕获，对于成功的连接，将调用accept方法做进一步处理，下面是accept方法的代码 :

private void accept() {
        SocketChannel channel = null;
        try {
            channel = serverChannel.accept();
            channel.configureBlocking(false);
            FrontendConnection c = factory.make(channel);
            c.setAccepted(true);
            c.setId(ID_GENERATOR.getId());
            NIOProcessor processor = (NIOProcessor) MycatServer.getInstance()
                    .nextProcessor();
            c.setProcessor(processor);

            NIOReactor reactor = reactorPool.getNextReactor();
            reactor.postRegister(c);

        } catch (Exception e) {
            LOGGER.warn(getName(), e);
            closeChannel(channel);
        }
    }

accept方法将客户端连接封装成FrontendConnection对象，分配相关资源，然后将客户端连接注册到NIOReactor上。

【温馨提示】
这里注意到，NIOProcessor（持有处理业务逻辑的线程池）和NIOReactor（负责处理NIO网络通讯的读写事件）都进行了池化，目的是让对象得以共享这些稀缺的资源。

2.2 NIOConnector

NIOConnector线程负责处理MyCat连接后端mysql实例的事件（在NIO网络编程中，对应于NIO Client）。与NIOAcceptor类似，该类继承Thread类，在run方法里面处理连接事件，代码如下所示 :

@Override
    public void run() {
        final Selector tSelector = this.selector;
        for (;;) {
            ++connectCount;
            try {
                //查看有无连接就绪
                tSelector.select(1000L);
                connect(tSelector);
                Set<SelectionKey> keys = tSelector.selectedKeys();
                try {
                    for (SelectionKey key : keys) {
                        Object att = key.attachment();
                        if (att != null && key.isValid() && key.isConnectable()) {
                            finishConnect(key, att);
                        } else {
                            key.cancel();
                        }
                    }
                } finally {
                    keys.clear();
                }
            } catch (Exception e) {
                LOGGER.warn(name, e);
            }
        }
    }

在connect方法里面会判断连接队列（元素入队列体现在postConnect方法方法里）是否有连接需要进行处理，如果队列有值，那么取得连接对象主动去跟后端的MySQL实例建立连接。代码如下所示 :

private void connect(Selector selector) {
        AbstractConnection c = null;
        while ((c = connectQueue.poll()) != null) {
            try {
                SocketChannel channel = (SocketChannel) c.getChannel();
                //注册OP_CONNECT监听与后端连接是否真正建立
                channel.register(selector, SelectionKey.OP_CONNECT, c);
                //主动连接
                channel.connect(new InetSocketAddress(c.host, c.port));
            } catch (Exception e) {
                c.close(e.toString());
            }
        }
    }

连接建立成功后调用finishConnect方法注册对应的读写事件，然后与NIOAcceptor一样，将读写事件的处理交由NIOReactor处理。代码如下所示 :

private void finishConnect(SelectionKey key, Object att) {
        BackendAIOConnection c = (BackendAIOConnection) att;
        try {
            if (finishConnect(c, (SocketChannel) c.channel)) { //做原生NIO连接是否完成的判断和操作
                clearSelectionKey(key);
                c.setId(ID_GENERATOR.getId());
                //绑定特定的NIOProcessor以作idle清理
                NIOProcessor processor = MycatServer.getInstance()
                        .nextProcessor();
                c.setProcessor(processor);
                //与特定NIOReactor绑定监听读写
                NIOReactor reactor = reactorPool.getNextReactor();
                reactor.postRegister(c);
            }
        } catch (Exception e) {
            //如有异常，将key清空
            clearSelectionKey(key);
            c.close(e.toString());
            c.onConnectFailed(e);
        }
    }

2.3 NIOReactor

一般的NIO网络编程示例，在Server accept客户端连接之后，会直接在该线程里注册读写事件，最后再分发给逻辑处理线程池去处理逻辑，示例代码如下所示 :

for (;;) {
    try {
        tSelector.select();
        Set<SelectionKey> keys = tSelector.selectedKeys();
        try {
            for (SelectionKey key : keys) {
                if (key.isValid() && key.isAcceptable()) {
                    accept(); // accept register OP_READ and OP_WRITE
                } else if(key.isReadable) {
                    read(); // read data and do business using thread pool
                } else if(key.isWritable) {
                    write(); // write data using thread pool
                }
            }
        } finally {
            keys.clear();
        }
    } catch (Exception e) {
        // 
    }
}

在高性能IO框架实现中，大多采用Reactor模型，将处理连接和处理读写事件分离，模型图如下所示 :

MyCat中采用了reactor模式，将读写事件的注册和接收分离到另外的线程去处理。这个另外的线程指的就是NIOReactor，并且，为了优化并发下频繁进行读写，NIOReactor进行了池化（NIOReactorPool）。

3. 业务线程

BusinessExecutor采用Java线程池来实现，线程池大小默认计算方式如下所示：

this.processorExecutor = (DEFAULT_PROCESSORS != 1) ? DEFAULT_PROCESSORS * 2 : 4;

其中DEFAULT_PROCESSORS为系统CPU可用核心数。单核心的线程池大小为4，非单核心线程池大小为系统CPU可用核心数的2倍。

线程池的任务队列使用LinkedTransferQueue。

业务线程主要用在以下方面：

• 负责对读取到的网络字节数据做进一步处理
• 创建后台mysql连接
• 多分片节点结果集包返回汇聚后处理

4. 定时任务线程

TimerExecutor 定时任务线程池，默认线程大小为2。TimerExecutor结合Timer类进行定时任务调度，主要功能为：

• 定时检查前端连接和后端空闲连接，回收不可用的连接。
• 定期发送心跳包以及心跳检测

【温馨提示】 定时逻辑代码入口在MyCatServer.java类里面

5. jstack dump线程

我们可以使用jdk自带工具jstack来查看MyCat-Server线程的使用情况:

首先使用jps查看MyCat-Server的进程号:

然后使用jstack [mycat-server-pid]打印出线程使用情况:

C:\WINDOWS\system32>jstack 8648
2016-08-11 22:34:47
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.66-b18 mixed mode):

"Timer1" #31 daemon prio=5 os_prio=0 tid=0x000000001a75a800 nid=0x21dc waiting on condition [0x000000001e0df000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000d7c00730> (a java.util.concurrent.LinkedTransferQueue)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
        at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
        at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"Timer0" #30 daemon prio=5 os_prio=0 tid=0x000000001a759000 nid=0x1ce0 waiting on condition [0x000000001dfde000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000d7c00730> (a java.util.concurrent.LinkedTransferQueue)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
        at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
        at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"BusinessExecutor7" #29 daemon prio=5 os_prio=0 tid=0x000000001a759800 nid=0x13b4 waiting on condition [0x000000001dbde000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000d7c18178> (a java.util.concurrent.LinkedTransferQueue)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
        at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
        at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"BusinessExecutor6" #28 daemon prio=5 os_prio=0 tid=0x000000001a75b000 nid=0x1090 waiting on condition [0x000000001dade000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000d7c18178> (a java.util.concurrent.LinkedTransferQueue)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
        at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
        at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"BusinessExecutor5" #27 daemon prio=5 os_prio=0 tid=0x000000001a753800 nid=0xf08 waiting on condition [0x000000001d9de000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000d7c18178> (a java.util.concurrent.LinkedTransferQueue)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
        at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
        at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"BusinessExecutor4" #26 daemon prio=5 os_prio=0 tid=0x000000001a758000 nid=0x14a0 waiting on condition [0x000000001d8df000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000d7c18178> (a java.util.concurrent.LinkedTransferQueue)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
        at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
        at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"BusinessExecutor3" #25 daemon prio=5 os_prio=0 tid=0x000000001a756800 nid=0x21c4 waiting on condition [0x000000001d7de000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000d7c18178> (a java.util.concurrent.LinkedTransferQueue)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
        at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
        at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"BusinessExecutor2" #24 daemon prio=5 os_prio=0 tid=0x000000001a757800 nid=0x21c0 waiting on condition [0x000000001d6de000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000d7c18178> (a java.util.concurrent.LinkedTransferQueue)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
        at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
        at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"BusinessExecutor1" #23 daemon prio=5 os_prio=0 tid=0x000000001a755000 nid=0x21b4 waiting on condition [0x000000001d5de000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000d7c18178> (a java.util.concurrent.LinkedTransferQueue)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
        at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
        at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"BusinessExecutor0" #22 daemon prio=5 os_prio=0 tid=0x000000001a4e5000 nid=0x21b0 waiting on condition [0x000000001d4de000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000d7c18178> (a java.util.concurrent.LinkedTransferQueue)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
        at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
        at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"$_MyCatServer" #21 prio=5 os_prio=0 tid=0x000000001a4ec000 nid=0x21b8 runnable [0x000000001d3de000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000d7c0bb68> (a sun.nio.ch.Util$2)
        - locked <0x00000000d7c0bb78> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000d7c0bae8> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.opencloudb.net.NIOAcceptor.run(NIOAcceptor.java:86)

"$_MyCatManager" #20 prio=5 os_prio=0 tid=0x000000001a4ea800 nid=0x21bc runnable [0x000000001d2df000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000d7c00de8> (a sun.nio.ch.Util$2)
        - locked <0x00000000d7c00df8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000d7c00d68> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.opencloudb.net.NIOAcceptor.run(NIOAcceptor.java:86)

"$_NIOConnector" #19 prio=5 os_prio=0 tid=0x000000001a4e9000 nid=0x20f0 runnable [0x000000001d1de000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000d7c1b600> (a sun.nio.ch.Util$2)
        - locked <0x00000000d7c1b610> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000d7c1b580> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.opencloudb.net.NIOConnector.run(NIOConnector.java:76)

"$_NIOREACTOR-3-RW" #18 prio=5 os_prio=0 tid=0x000000001a4e8000 nid=0x2314 runnable [0x000000001d0de000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000d7c13868> (a sun.nio.ch.Util$2)
        - locked <0x00000000d7c13878> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000d7c137e8> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.opencloudb.net.NIOReactor$RW.run(NIOReactor.java:89)
        at java.lang.Thread.run(Thread.java:745)

"$_NIOREACTOR-2-RW" #17 prio=5 os_prio=0 tid=0x000000001a4e6800 nid=0x1774 runnable [0x000000001cfdf000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000d7c1e880> (a sun.nio.ch.Util$2)
        - locked <0x00000000d7c1e890> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000d7c1e800> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.opencloudb.net.NIOReactor$RW.run(NIOReactor.java:89)
        at java.lang.Thread.run(Thread.java:745)

"$_NIOREACTOR-1-RW" #16 prio=5 os_prio=0 tid=0x000000001a4e7800 nid=0x2320 runnable [0x000000001cede000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000d7c07a68> (a sun.nio.ch.Util$2)
        - locked <0x00000000d7c07a78> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000d7c079e8> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.opencloudb.net.NIOReactor$RW.run(NIOReactor.java:89)
        at java.lang.Thread.run(Thread.java:745)

"$_NIOREACTOR-0-RW" #15 prio=5 os_prio=0 tid=0x000000001a4e6000 nid=0x20ac runnable [0x000000001cdde000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000d7c0f0c0> (a sun.nio.ch.Util$2)
        - locked <0x00000000d7c0f0d0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000d7c0f040> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.opencloudb.net.NIOReactor$RW.run(NIOReactor.java:89)
        at java.lang.Thread.run(Thread.java:745)

"Log4jWatchdog" #14 daemon prio=5 os_prio=0 tid=0x000000001a4e9800 nid=0x16d0 waiting on condition [0x000000001bd0f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:104)

"[email protected]" #12 daemon prio=5 os_prio=0 tid=0x000000001a4eb000 nid=0x2298 in Object.wait() [0x000000001bb0f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000d7c454a8> (a java.util.TaskQueue)
        at java.lang.Object.wait(Object.java:502)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        - locked <0x00000000d7c454a8> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

"MyCatTimer" #11 daemon prio=5 os_prio=0 tid=0x000000001a372000 nid=0x1f48 in Object.wait() [0x000000001b7bf000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        - locked <0x00000000d7c46428> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

"Thread-0" #10 prio=5 os_prio=0 tid=0x0000000019e9b000 nid=0x2304 waiting on condition [0x000000001a6ee000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000d7cda8b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at org.opencloudb.route.MyCATSequnceProcessor$ExecuteThread.run(MyCATSequnceProcessor.java:102)

"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x000000001967d000 nid=0x1f5c runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread2" #8 daemon prio=9 os_prio=2 tid=0x00000000195f7800 nid=0x213c waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #7 daemon prio=9 os_prio=2 tid=0x00000000195f1000 nid=0x20fc waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #6 daemon prio=9 os_prio=2 tid=0x0000000017f9a800 nid=0x1cdc waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x0000000017f4f000 nid=0x72c waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x0000000017f4e000 nid=0x214c runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x00000000033b7000 nid=0x2150 in Object.wait() [0x000000001928f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        - locked <0x00000000d7d301b0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x0000000017f09000 nid=0x1328 in Object.wait() [0x000000001918f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
        - locked <0x00000000d7d303c8> (a java.lang.ref.Reference$Lock)

"main" #1 prio=5 os_prio=0 tid=0x00000000030be000 nid=0x22c8 waiting on condition [0x00000000032bf000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.opencloudb.MycatStartup.main(MycatStartup.java:57)

"VM Thread" os_prio=2 tid=0x0000000017f07000 nid=0x1218 runnable

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00000000032d7800 nid=0x1d54 runnable

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00000000032d9000 nid=0x23f4 runnable

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00000000032da800 nid=0x1600 runnable

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00000000032dc000 nid=0x1aac runnable

"VM Periodic Task Thread" os_prio=2 tid=0x00000000196b9800 nid=0x2308 waiting on condition

JNI global references: 253