ProxySQL 排错 Max connect timeout reached while reaching hostgroup 10 after 10000ms
Posted 2森
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了ProxySQL 排错 Max connect timeout reached while reaching hostgroup 10 after 10000ms相关的知识，希望对你有一定的参考价值。
ProxySQL 排错
- 问题分析:
在ProxySQL在集群下,因未知原因导致误测到所有节点OFFLINE_HARD,并runtime_mysql_servers表清空,从而导致前端查询无法传递到后端对应的servers上,重新load mysql servers to runtime可以暂时解决,但没有找到长期解决办法,只好退回单节点模式。如有解决的兄弟希望你分享给我(k2865#qq.com,或讨论区留言 ) - 版本和环境
ProxySQL:
[email protected] [(none)]> select @@Version;
+--------------------+
| @@version |
+--------------------+
| v1.4.10-1-g5eb0f3e |
+--------------------+
1 row in set (0.00 sec)
[email protected] [(none)]>select * from disk.proxysql_servers;
+----------+------+--------+-----------+
| hostname | port | weight | comment |
+----------+------+--------+-----------+
| db211 | 6032 | 1 | secondary |
| db210 | 6032 | 1 | primary |
+----------+------+--------+-----------+
2 rows in set (0.00 sec)
--
--
MGR:
[email protected] [sysbench_testdata]>select version();
+------------+
| version() |
+------------+
| 5.7.23-log |
+------------+
1 row in set (0.00 sec)
[email protected] [sysbench_testdata]>select * from performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| group_replication_applier | 33a6c4f8-a67a-11e8-b7f6-0050568beaf6 | db211 | 3507 | ONLINE |
| group_replication_applier | 951678d2-91a2-11e8-be71-0050568be82c | db210 | 3507 | ONLINE |
| group_replication_applier | abac4f08-a683-11e8-9447-0050568b744a | db212 | 3507 | ONLINE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
3 rows in set (0.00 sec)
- 故障现像:
ProxySQL 开始最先是正常的.可以读写,通过sysbench 也能正常压测,但随后报以下错误:
FATAL: MySQL error: 9001 "Max connect timeout reached while reaching hostgroup 10 after 10000ms"
(last message repeated 1 times)
FATAL: `thread_init‘ function failed: /usr/share/sysbench/oltp_common.lua:273: SQL API error
(last message repeated 1 times)
FATAL: mysql_stmt_prepare() failed
FATAL: MySQL error: 9001 "Max connect timeout reached while reaching hostgroup 10 after 10000ms"
FATAL: mysql_stmt_prepare() failed
FATAL: MySQL error: 9001 "Max connect timeout reached while reaching hostgroup 10 after 10000ms"
FATAL: mysql_stmt_prepare() failed
FATAL: MySQL error: 9001 "Max connect timeout reached while reaching hostgroup 10 after 10001ms"
FATAL: mysql_stmt_prepare() failed
- 手工连接,发现直接连接MGR节点是可以正常连接并查询的,连接proxysql,可以登录但无法查询:
[email protected] [(none)]>select * from wenyz.t2;
ERROR 9001 (HY000): Max connect timeout reached while reaching hostgroup 11 after 10000ms
[email protected] [(none)]>select * from wenyz.t2;
ERROR 9001 (HY000): Max connect timeout reached while reaching hostgroup 11 after 10000ms查看官方问题库,得以下解释:
11. No Hostgroup 0 has been configured , then why we get "Max connect timeout reached while reaching hostgroup 0 after 10000ms" ?
This is how MySQL Query Rules work while selecting hostgroups.
When you set any query rules inside table mysql_query_rules , Then your query get analyzed by the Query Processor to decide which destination hostgroup it should forward. (according to mysql_query_rules.destination_hostgroup)
When your Query Processor doesn‘t find any query matching to query rule then the default hostgroup for the specific user is applied (according to mysql_users.default_hostgroup)
Example : Default Hostgroup for User根据这一信息,查看ProxySQL状态,发现runtime_mysql_servers没有记录.
[email protected] [(none)]>select * from mysql_servers;
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 10 | db212 | 3507 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 11 | db210 | 3507 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 11 | db212 | 3507 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 11 | db211 | 3507 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 13 | db211 | 3507 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 13 | db210 | 3507 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
6 rows in set (0.00 sec)
[email protected] [(none)]>select * from runtime_mysql_servers;
Empty set (0.00 sec)
[email protected] [(none)]>- 重新执行load mysql servers to runtime后恢复正常
[email protected] [(none)]>load mysql servers to runtime;
Query OK, 0 rows affected (0.00 sec)
- 问题重现时proxysql.log中出现以下信息:
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:308 , MyDS:308) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:311 , MyDS:311) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:312 , MyDS:312) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:315 , MyDS:315) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:320 , MyDS:320) , 1040: Too many connections.
2018-08-25 23:33:06 MySQL_HostGroups_Manager.cpp:303:connect_error(): [ERROR] Shunning server db212:3507 with 5 errors/sec. Shunning for 10 seconds
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:322 , MyDS:322) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:309 , MyDS:309) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:326 , MyDS:326) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:313 , MyDS:313) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:328 , MyDS:328) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:317 , MyDS:317) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:325 , MyDS:325) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:334 , MyDS:334) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:332 , MyDS:332) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:331 , MyDS:331) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:307 , MyDS:307) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:310 , MyDS:310) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:314 , MyDS:314) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:316 , MyDS:316) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:318 , MyDS:318) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:321 , MyDS:321) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:324 , MyDS:324) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:327 , MyDS:327) , 1040: Too many connections.
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:330 , MyDS:330) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:333 , MyDS:333) , 1040: Too many connections.
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:335 , MyDS:335) , 1040: Too many connections.
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:336 , MyDS:336) , 1040: Too many connections.
2018-08-25 23:33:06 mysql_connection.cpp:634:handler(): [ERROR] Failed to mysql_real_connect() on db212:3507 , FD (Conn:337 , MyDS:337) , 1040: Too many connections.
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 MySQL_Session.cpp:2884:handler(): [WARNING] Error during query on (10,db212,3507): 1295, This command is not supported in the prepared statement protocol yet
2018-08-25 23:33:06 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: SHUNNED , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-25 23:33:06 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 1 | 0 | 1000 | 0 | 0 | 0 | | 140420659150080 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659545088 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659544960 |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659544832 |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548032 |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549696 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-25 23:33:11 MySQL_HostGroups_Manager.cpp:2580:update_group_replication_set_writer(): [WARNING] Group Replication: setting host db211:3507 as writer
2018-08-25 23:33:11 MySQL_HostGroups_Manager.cpp:2580:update_group_replication_set_writer(): [WARNING] Group Replication: setting host db210:3507 as writer
2018-08-25 23:33:11 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: SHUNNED , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-25 23:33:11 [INFO] Dumping mysql_servers
- 出现以上信息时,sysbench端报ERROR 9001 (HY000): Max connect timeout reached while reaching hostgroup 11 after 10000ms了.但proxysql的runtime_mysql_server中还是正常的.在proxysql.log中有大量的deadlock 或lock timeout,输出的runtime_mysql_server信息是全的.但到第二天早上才有如下:
2018-08-26 08:16:27 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 08:16:27 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659150080 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659545088 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420789185920 |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659544832 |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548032 |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549696 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 08:16:27 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 08:16:27 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659150080 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659545088 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420789185920 |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659544832 |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548032 |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549696 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 08:16:27 [INFO] Dumping mysql_servers_incoming
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2018-08-26 08:16:27 MySQL_HostGroups_Manager.cpp:602:commit(): [WARNING] Removed server at address 140420659150080, hostgroup 10, address db212 port 3507. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2018-08-26 08:16:27 MySQL_HostGroups_Manager.cpp:602:commit(): [WARNING] Removed server at address 140420659545088, hostgroup 13, address db210 port 3507. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2018-08-26 08:16:27 MySQL_HostGroups_Manager.cpp:602:commit(): [WARNING] Removed server at address 140420789185920, hostgroup 13, address db211 port 3507. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2018-08-26 08:16:27 MySQL_HostGroups_Manager.cpp:602:commit(): [WARNING] Removed server at address 140420659544832, hostgroup 11, address db211 port 3507. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2018-08-26 08:16:27 MySQL_HostGroups_Manager.cpp:602:commit(): [WARNING] Removed server at address 140420659548032, hostgroup 11, address db212 port 3507. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2018-08-26 08:16:27 MySQL_HostGroups_Manager.cpp:602:commit(): [WARNING] Removed server at address 140420659549696, hostgroup 11, address db210 port 3507. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2018-08-26 08:16:27 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 08:16:27 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659150080 |
| 13 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659545088 |
| 13 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420789185920 |
| 11 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659544832 |
| 11 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548032 |
| 11 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549696 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 08:16:27 [INFO] Dumping current MySQL Servers structures for hostgroup 10
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 08:16:27 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659150080 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 [INFO] Dumping current MySQL Servers structures for hostgroup 13
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 08:16:27 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659150080 |
| 13 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420789185920 |
| 13 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659545088 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 08:16:27 [INFO] Dumping current MySQL Servers structures for hostgroup 11
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 08:16:27 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659150080 |
| 13 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420789185920 |
| 13 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659545088 |
| 11 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549696 |
| 11 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659544832 |
| 11 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548032 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 08:16:27 [INFO] Dumping current MySQL Servers structures for hostgroup 15
2018-08-26 08:16:27 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659150080 |
| 13 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420789185920 |
| 13 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659545088 |
| 11 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549696 |
| 11 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659544832 |
| 11 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548032 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:27 MySQL_Session.cpp:2798:handler(): [ERROR] Detected an offline server during query: db212, 3507
2018-08-26 08:16:28 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 08:16:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 11 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659544832 |
| 13 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659545088 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 08:16:28 [INFO] Cluster: detected a new checksum for mysql_servers from peer db210:6032, version 16925, epoch 1535242587, checksum 0xEA6375A74FCB9E77 . Not syncing yet ...
2018-08-26 08:16:28 [INFO] Cluster: checksum for mysql_servers from peer db210:6032 matches with local checksum 0xEA6375A74FCB9E77 , we won‘t sync.
2018-08-26 08:16:30 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
2018-08-26 08:16:30 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+
\ 另一次故障日志
2018-08-26 12:22:23 [INFO] Dumping current MySQL Servers structures for hostgroup 10
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:23 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:23 [INFO] Dumping current MySQL Servers structures for hostgroup 13
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:23 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:23 [INFO] Dumping current MySQL Servers structures for hostgroup 11
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:23 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548544 |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548672 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:23 [INFO] Dumping current MySQL Servers structures for hostgroup 15
2018-08-26 12:22:23 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548544 |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548672 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:24 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:24 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548672 |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548544 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 MySQL_HostGroups_Manager.cpp:2580:update_group_replication_set_writer(): [WARNING] Group Replication: setting host db211:3507 as writer
2018-08-26 12:22:28 [INFO] Group replication: max_writers=1 , moving 1 nodes from writer HG 10 to backup HG 13
2018-08-26 12:22:28 MySQL_HostGroups_Manager.cpp:2580:update_group_replication_set_writer(): [WARNING] Group Replication: setting host db210:3507 as writer
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548672 |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548544 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 [INFO] Dumping mysql_servers_incoming
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548672 |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548544 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup 10
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup 13
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup 11
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548544 |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548672 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup 15
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548544 |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548672 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548672 |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548544 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
| 13 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 11 | db211 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 11 | db212 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548672 |
| 11 | db210 | 3507 | 1 | 0 | 0 | 1000 | 0 | 0 | 0 | | 140420659548544 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 [INFO] Dumping mysql_servers_incoming
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2018-08-26 12:22:28 MySQL_HostGroups_Manager.cpp:602:commit(): [WARNING] Removed server at address 140420659549056, hostgroup 10, address db212 port 3507. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2018-08-26 12:22:28 MySQL_HostGroups_Manager.cpp:602:commit(): [WARNING] Removed server at address 140420659549184, hostgroup 13, address db210 port 3507. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2018-08-26 12:22:28 MySQL_HostGroups_Manager.cpp:602:commit(): [WARNING] Removed server at address 140420659549568, hostgroup 13, address db211 port 3507. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2018-08-26 12:22:28 MySQL_HostGroups_Manager.cpp:602:commit(): [WARNING] Removed server at address 140420659548800, hostgroup 11, address db211 port 3507. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2018-08-26 12:22:28 MySQL_HostGroups_Manager.cpp:602:commit(): [WARNING] Removed server at address 140420659548672, hostgroup 11, address db212 port 3507. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2018-08-26 12:22:28 MySQL_HostGroups_Manager.cpp:602:commit(): [WARNING] Removed server at address 140420659548544, hostgroup 11, address db210 port 3507. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
| 13 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 11 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 11 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548672 |
| 11 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548544 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup 10
HID: 10 , address: db212 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup 13
HID: 13 , address: db211 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 13 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup 11
HID: 11 , address: db210 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db212 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 13 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
| 11 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548544 |
| 11 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 11 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548672 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:28 [INFO] Dumping current MySQL Servers structures for hostgroup 15
2018-08-26 12:22:28 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549056 |
| 13 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549568 |
| 13 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
| 11 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548544 |
| 11 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 11 | db212 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548672 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:29 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 11 , address: db211 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 13 , address: db210 , port: 3507 , weight: 1 , status: OFFLINE_HARD , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment:
2018-08-26 12:22:29 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 11 | db211 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659548800 |
| 13 | db210 | 3507 | 1 | 3 | 0 | 1000 | 0 | 0 | 0 | | 140420659549184 |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
2018-08-26 12:22:29 [INFO] Cluster: detected a new checksum for mysql_servers from peer db210:6032, version 17848, epoch 1535257348, checksum 0xEA6375A74FCB9E77 . Not syncing yet ...
2018-08-26 12:22:29 [INFO] Cluster: checksum for mysql_servers from peer db210:6032 matches with local checksum 0xEA6375A74FCB9E77 , we won‘t sync.
2018-08-26 12:22:32 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
2018-08-26 12:22:32 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+
2018-08-26 12:22:32 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
2018-08-26 12:22:32 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+
2018-08-26 12:22:33 [INFO] Cluster: detected a new checksum for mysql_servers from peer db211:6032, version 14580, epoch 1535257352, checksum 0xEA6375A74FCB9E77 . Not syncing yet ...
2018-08-26 12:22:33 [INFO] Cluster: checksum for mysql_servers from peer db211:6032 matches with local checksum 0xEA6375A74FCB9E77 , we won‘t sync.
查看 集群状态 发现没有监测到runtime的变化:
[email protected] [(none)]>select *,from_unixtime(changed_at),from_unixtime(updated_at) from stats_proxysql_servers_checksums;
+----------+------+-------------------+---------+------------+--------------------+------------+------------+------------+---------------------------+---------------------------+
| hostname | port | name | version | epoch | checksum | changed_at | updated_at | diff_check | from_unixtime(changed_at) | from_unixtime(updated_at) |
+----------+------+-------------------+---------+------------+--------------------+------------+------------+------------+---------------------------+---------------------------+
| db211 | 6032 | admin_variables | 0 | 0 | | 0 | 1535258863 | 0 | 1970-01-01 00:00:00 | 2018-08-26 04:47:43 |
| db211 | 6032 | mysql_query_rules | 2 | 1535111193 | 0xC2127DA38220E59D | 1535174701 | 1535258863 | 0 | 2018-08-25 05:25:01 | 2018-08-26 04:47:43 |
| db211 | 6032 | mysql_servers | 14580 | 1535257352 | 0xEA6375A74FCB9E77 | 1535257353 | 1535258863 | 0 | 2018-08-26 04:22:33 | 2018-08-26 04:47:43 |
| db211 | 6032 | mysql_users | 2 | 1535105812 | 0x6C619C922DBF30FF | 1535174701 | 1535258863 | 0 | 2018-08-25 05:25:01 | 2018-08-26 04:47:43 |
| db211 | 6032 | mysql_variables | 0 | 0 | | 0 | 1535258863 | 0 | 1970-01-01 00:00:00 | 2018-08-26 04:47:43 |
| db211 | 6032 | proxysql_servers | 8 | 1535125307 | 0x53CAAA0FF8478223 | 1535174701 | 1535258863 | 0 | 2018-08-25 05:25:01 | 2018-08-26 04:47:43 |
| db210 | 6032 | admin_variables | 0 | 0 | | 0 | 1535258863 | 0 | 1970-01-01 00:00:00 | 2018-08-26 04:47:43 |
| db210 | 6032 | mysql_query_rules | 1 | 1535174701 | 0xC2127DA38220E59D | 1535174702 | 1535258863 | 0 | 2018-08-25 05:25:02 | 2018-08-26 04:47:43 |
| db210 | 6032 | mysql_servers | 17848 | 1535257348 | 0xEA6375A74FCB9E77 | 1535257349 | 1535258863 | 0 | 2018-08-26 04:22:29 | 2018-08-26 04:47:43 |
| db210 | 6032 | mysql_users | 1 | 1535174701 | 0x6C619C922DBF30FF | 1535174702 | 1535258863 | 0 | 2018-08-25 05:25:02 | 2018-08-26 04:47:43 |
| db210 | 6032 | mysql_variables | 0 | 0 | | 0 | 1535258863 | 0 | 1970-01-01 00:00:00 | 2018-08-26 04:47:43 |
| db210 | 6032 | proxysql_servers | 1 | 1535174701 | 0x53CAAA0FF8478223 | 1535174702 | 1535258863 | 0 | 2018-08-25 05:25:02 | 2018-08-26 04:47:43 |
+----------+------+-------------------+---------+------------+--------------------+------------+------------+------------+---------------------------+---------------------------+
12 rows in set (0.00 sec)ProxySQL全配置信息
[email protected] [(none)]>show global variables;
+-----------------------------------------------------+--------------------------------+
| Variable_name | Value |
+-----------------------------------------------------+--------------------------------+
| admin-admin_credentials | admin:admin;cluster_21x:zstxxx |
| admin-checksum_mysql_query_rules | true |
| admin-checksum_mysql_servers | true |
| admin-checksum_mysql_users | true |
| admin-cluster_check_interval_ms | 1000 |
| admin-cluster_check_status_frequency | 10 |
| admin-cluster_mysql_query_rules_diffs_before_sync | 3 |
| admin-cluster_mysql_query_rules_save_to_disk | true |
| admin-cluster_mysql_servers_diffs_before_sync | 3 |
| admin-cluster_mysql_servers_save_to_disk | true |
| admin-cluster_mysql_users_diffs_before_sync | 3 |
| admin-cluster_mysql_users_save_to_disk | true |
| admin-cluster_password | zstxxx |
| admin-cluster_proxysql_servers_diffs_before_sync | 3 |
| admin-cluster_proxysql_servers_save_to_disk | true |
| admin-cluster_username | cluster_21x |
| admin-hash_passwords | true |
| admin-mysql_ifaces | 0.0.0.0:6032 |
| admin-read_only | false |
| admin-refresh_interval | 2000 |
| admin-stats_credentials | stats:stats |
| admin-stats_mysql_connection_pool | 60 |
| admin-stats_mysql_connections | 60 |
| admin-stats_mysql_query_cache | 60 |
| admin-stats_system_cpu | 60 |
| admin-stats_system_memory | 60 |
| admin-telnet_admin_ifaces | (null) |
| admin-telnet_stats_ifaces | (null) |
| admin-version | v1.4.10-1-g5eb0f3e |
| admin-web_enabled | false |
| admin-web_port | 6080 |
| mysql-autocommit_false_is_transaction | false |
| mysql-autocommit_false_not_reusable | false |
| mysql-client_found_rows | true |
| mysql-client_multi_statements | true |
| mysql-commands_stats | true |
| mysql-connect_retries_delay | 1 |
| mysql-connect_retries_on_failure | 10 |
| mysql-connect_timeout_server | 3000 |
| mysql-connect_timeout_server_max | 10000 |
| mysql-connection_delay_multiplex_ms | 0 |
| mysql-connection_max_age_ms | 0 |
| mysql-connpoll_reset_queue_length | 50 |
| mysql-default_charset | utf8 |
| mysql-default_max_latency_ms | 1000 |
| mysql-default_query_delay | 0 |
| mysql-default_query_timeout | 36000000 |
| mysql-default_reconnect | true |
| mysql-default_schema | information_schema |
| mysql-default_sql_mode | |
| mysql-default_time_zone | SYSTEM |
| mysql-enforce_autocommit_on_reads | false |
| mysql-eventslog_filename | |
| mysql-eventslog_filesize | 104857600 |
| mysql-forward_autocommit | false |
| mysql-free_connections_pct | 10 |
| mysql-have_compress | true |
| mysql-hostgroup_manager_verbose | 1 |
| mysql-init_connect | |
| mysql-interfaces | 0.0.0.0:6033 |
| mysql-kill_backend_connection_when_disconnect | true |
| mysql-long_query_time | 1000 |
| mysql-max_allowed_packet | 4194304 |
| mysql-max_connections | 2048 |
| mysql-max_stmts_cache | 10000 |
| mysql-max_stmts_per_connection | 20 |
| mysql-max_transaction_time | 14400000 |
| mysql-mirror_max_concurrency | 16 |
| mysql-mirror_max_queue_length | 32000 |
| mysql-monitor_connect_interval | 60000 |
| mysql-monitor_connect_timeout | 600 |
| mysql-monitor_enabled | true |
| mysql-monitor_groupreplication_healthcheck_interval | 5000 |
| mysql-monitor_groupreplication_healthcheck_timeout | 800 |
| mysql-monitor_history | 600000 |
| mysql-monitor_password | zstzst |
| mysql-monitor_ping_interval | 10000 |
| mysql-monitor_ping_max_failures | 3 |
| mysql-monitor_ping_timeout | 1000 |
| mysql-monitor_query_interval | 60000 |
| mysql-monitor_query_timeout | 100 |
| mysql-monitor_read_only_interval | 1500 |
| mysql-monitor_read_only_max_timeout_count | 3 |
| mysql-monitor_read_only_timeout | 500 |
| mysql-monitor_replication_lag_interval | 10000 |
| mysql-monitor_replication_lag_timeout | 1000 |
| mysql-monitor_replication_lag_use_percona_heartbeat | |
| mysql-monitor_slave_lag_when_null | 60 |
| mysql-monitor_username | proxysql |
| mysql-monitor_wait_timeout | true |
| mysql-monitor_writer_is_also_reader | true |
| mysql-multiplexing | true |
| mysql-ping_interval_server_msec | 120000 |
| mysql-ping_timeout_server | 500 |
| mysql-poll_timeout | 2000 |
| mysql-poll_timeout_on_failure | 100 |
| mysql-query_cache_size_MB | 256 |
| mysql-query_digests | true |
| mysql-query_digests_lowercase | false |
| mysql-query_digests_max_digest_length | 2048 |
| mysql-query_digests_max_query_length | 65000 |
| mysql-query_processor_iterations | 0 |
| mysql-query_processor_regex | 1 |
| mysql-query_retries_on_failure | 1 |
| mysql-server_capabilities | 45578 |
| mysql-server_version | 5.5.30 |
| mysql-servers_stats | true |
| mysql-session_idle_ms | 1000 |
| mysql-session_idle_show_processlist | true |
| mysql-sessions_sort | true |
| mysql-shun_on_failures | 5 |
| mysql-shun_recovery_time_sec | 10 |
| mysql-ssl_p2s_ca | |
| mysql-ssl_p2s_cert | |
| mysql-ssl_p2s_cipher | |
| mysql-ssl_p2s_key | |
| mysql-stacksize | 1048576 |
| mysql-stats_time_backend_query | false |
| mysql-stats_time_query_processor | false |
| mysql-threads | 4 |
| mysql-threshold_query_length | 524288 |
| mysql-threshold_resultset_size | 4194304 |
| mysql-throttle_connections_per_sec_to_hostgroup | 1000000 |
| mysql-throttle_max_bytes_per_second_to_client | 2147483647 |
| mysql-throttle_ratio_server_to_client | 0 |
| mysql-verbose_query_error | false |
| mysql-wait_timeout | 28800000 |
+-----------------------------------------------------+--------------------------------+
127 rows in set (0.00 sec)
最后ProxySQL切换成单节点,运行正常
以上是关于ProxySQL 排错 Max connect timeout reached while reaching hostgroup 10 after 10000ms的主要内容，如果未能解决你的问题，请参考以下文章
mysql 的max_connections和max_user_connections 的区别
max_connections 与 max_used_connections --ERROR 1040: Too many connections