Mysql 中间件 oneProxy总结
Posted coder_up
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Mysql 中间件 oneProxy总结相关的知识,希望对你有一定的参考价值。
建议使用之前把官方的文档全部通读一遍 这里提供一个我的网盘地址oneproxy 百度网盘
0. 先对 oneproxy 有个大概的了解,知道他所处的位置
1. mysql服务器
- 创建test用户,密码test
- 存在test数据库
- 确认oneproxy能登录
1.1 注意事项
- 所有MySQL节点有相同的用户名和口令
- 所有MySQL节点有相同的数据库名
- 所有MySQL节点使用相同的字符集设置
- 前端应用程序可以切换操作集群
- 命令:
use groupname
- 命令:
- 前端应用程序不能切换MySQL数据库
2. OneProxy 服务器
- 可以和Mysql共同一台机器
- 默认client port
3307
mysql -utest -ptest -h 127.0.0.1 -P3307
- 默认admin port
4041
mysql -uadmin -pOneProxy -h 127.0.0.1 -P4041
2.1 OneProxy 单机多实例
- OneProxy 单实例占用两个端口
3307
,4041
,所有只要保证端口不重复, 就可以保证单机启动多个实例 多实例只要换不同端口即可
2.2 关闭OneProxy
- 后台运行
- –keepalive
启动守护进程
- 遇到内存Bug Crash 时候,可以自动重启,应用自动重新连接(
开启keepalive后,会有一个单独的进程来监控oneproxy
)
- –keepalive
- 杀死进程
ps -ef | grep oneproxy
kill -9 所有oneproxy 进程号
- 管理端口里面
shutdown force
3. OneProxy 需要自己的验证机制
3.1 链接后端和client的配置
--proxy-user-list=username/password@dbname
--proxy-database: 指定默认数据库
--proxy-charset: 指定数据库字符集, 查看支持的字符集可以登陆到4041的命令窗口执行 list charset; 查看
这里的用户名和密码是提供oneproxy 链接后端mysql使用的,同时也是 client(php,java, python。。。)链接oneproxy 使用的
3.2 管理接口的用户名和密码配置
admin-user= admin
admin-password= OneProxy
admin-address=ip:port
密码必须加密,使用管理端口passwd命令生成
mysql -uadmin -pOneProxy -P4041 -c -h ip:port
> passwd test
4. 健康检查
- 每秒钟Proxy会和后端进行通信,检测后端数据库状态。
- Markup: 正常状态,可以提供服务
- Markdown:异常状态,不可以提供服务
- 查看状态
- 查看后端机器:
list backend;
- 查看组:
list group;
- 查看链接池:
list pool;
- 查看日志文件:tail -100f oneproxy.log
- 查看后端机器:
5. 多个数据库
- OneProxy的use命令变成切换集群
- OneProxy不支持数据库切换
- 程序里不可以有use命令(不良习惯)
- 可以设定多个用户(不同的默认库)
--proxy-user-list.1=user/pass@数据库名
--proxy-user-list.2=user/pass@数据库名
- 如果不指定,则默认的数据库名为
--proxy-database (默认为test)
6. 连接回收
- OneProxy 10秒钟检一次连接池。
- 自动回收5分钟不使用的连接。
要求数据库的超时设置大于300秒 + 10秒
- 确认
interactive_timeout
设置[mysqld]
- 确认
net_read_timeout
设置 - 确认
net_write_timeout
设置
- 确认
7. 配置主从
proxy-master-addresses.1 = 192.168.2.190:3306@server1:2
proxy-master-addresses.2 = 192.168.2.190:3306@server1:1
proxy-slave-addresses.1 = 192.168.2.190:3306@server1:1
proxy-slave-addresses.2 = 192.168.2.190:3306@server1:1
proxy-slave-addresses.3 = 192.168.2.190:3306@server1:1
addresses.(1,2,3,4):
这代表指定多台主从,
server1:(1,2,3,4):
这个代表该节点的power, 可以理解为负载均衡中的weight值
8. 常用命令
COMMAND | SUMMARY | SAMPLE |
---|---|---|
LIST HELP | Display the command list. | list |
LIST CHARSET | Display the charset list. | list charset |
LIST POLICY | Display the traffic policy for server group. | list policy |
LIST BACKEND | Display info for all the backends. | list backend |
LIST SEQUENCE | Display info for all the sequences. | list sequence |
LIST GROUP | Display info for all the server groups. | list group |
LIST POOL | Display info for all the connection pools. | list pool |
LIST THREADS | Display statitics data for all the threads. | list threads |
LIST TABLEMAP | Display the map of tables and server groups. | list tablemap |
LIST USERS | Display the user list. | list users |
LIST SQLSTATS | Display the sql statistics. | list sqlstats |
LIST TABSTATS | Display the table statistics. | list tabstats |
LIST USERSTATS | Display the user statistics. | list userstats |
LIST IPSTATS | Display the IP based perfstats . | list ipstats |
LIST IPQOS | Display the IP based request for last 10 seconds . | list IPQOS |
LIST DBQOS | Display the backend request for last 10 seconds . | list DBQOS |
LIST IPTABSTATS | Display the IP based table statistics. | list iptabstats |
LIST IPSQLSTATS | Display the IP based SQL statistics. | list ipsqlstats |
LIST SQLTEXT | Display the sql text. | list sqltext |
LIST DMLTEXT | Display the sql text of dml operation. | list dmltext |
LIST NEWTEXT | Display the sql text gathered by runtime. | list newtext |
LIST OLDTEXT | Display the sql text loaded by firewall. | list oldtext |
LIST SQLFAIL | Display the failed sql in dry run. | list sqlfail |
LIST BLACKSQL | Display the black sql text loaded by firewall. | list blacksql |
LIST TRANS_DEBUG | Display the table map of transaction. | list trans_debug |
LIST TABLES | Display the partition tables. | list tables |
LIST PARTITIONS | Display the table partitions. | list partitions |
LIST OPTION | Display the current options. | list option |
SET MASTER | Set a given backend as master backend. | set master ‘192.168.1.119:3306’ |
SET SLAVE | Set a given backend as slave backend. | set slave ‘192.168.1.119:3306’ |
SET OFFLINE | Set a given backend as offline backend. | set offline ‘192.168.1.119:3306’ |
SET ONLINE | Set a given backend as online backend. | set online ‘192.168.1.119:3306’ |
SET POWER | Set the power/capacity number for a given backend. | set power ‘192.168.1.119:3306’ [1-5] |
SET DBQOS | Set the QoS limit for a given backend. | set power ‘192.168.1.119:3306’ num |
SET GPOLICY | Set a feature policy for given server group. | set gpolicy default 1 |
SET GMASTER | Set the master backend for given server group. | set gmaster default 1 |
SET GACCESS | Set the security level for given server group. | set gaccess default 1 |
SET GDELAY | Set the maximum slave delay for given server group. | set gdelay default 1 |
SET GMANUAL | Set manual master mode for given server group. | set gmanual default 1 |
SET POOLMIN | Set the min idle connections for given backend. | set poolmin ‘192.168.1.119:3306’ 5 |
SET POOLMAX | Set the max idle connections for given backend. | set poolmax ‘192.168.1.119:3306’ 300 |
SET SQLSTATS | Enable or disable the sql statistics feature. | set sqlstats on |
SET IPSTATS | Clear the ip statistics data. | set ipstats clear |
SET TABSTATS | Enable or disable the table statistics feature. | set tabstats on |
SET USERSTATS | Enable or disable the user statistics feature. | set userstats on |
SET TRANS_DEBUG | Enable or disable the transaction debug mode. | set trans_debug on |
SET LOG_SQLERROR | Enable or disable the sql error log feature. | set log_sqlerror on |
SET SQL_DEBUG | Enable or disable the sql debug mode. | set sql_debug on |
SET SECURITY_LEVEL | set the security level of the proxy. | set security_level 0-maximum |
SET SQL_FIREWALL | Enable or disable the SQL Firewall feature. | set sql_firewall on |
SET DML_FIREWALL | Enable or disable the DML only Firewall feature. | set dml_firewall on |
SET WATCH_MODE | Enable or disable the Firewall watch mode. | set watch_mode on |
SET SQL_SIGNATURE | Enable or disable the SQL signature feature. | set sql_signature on |
SET IP_FIREWALL | Enable or disable the IP Firewall feature. | set ip_firewall on |
SET LOGIN_RESTRICT | Enable or disable the login restrict mode. | set login_restrict on |
SET STRICT_POOLMAX | Enable or disable the stricted pool max mode. | set strict_poolmax on |
SET UNSAFE_RESULT | Enalbe or disable the unsafe result operation. | set unsafe_group on |
SET MULTI_INSERTS | Enalbe or disable cross partition operation. | set multi_inserts on |
SET FORWARD_CLIENTIP | Comment the client ip into the SQL text. | set forward_clientip on |
SET BLACKIP | Disable the access of specific ip address. | set blackip ‘192.168.0.1’ |
SET GREENIP | Enable the access of specific ip address. | set greenip ‘192.168.0.1’ |
SET READ_ONLY | Put a partition into read only mode. | set read_only ‘my_range_0’ |
SET READ_WRITE | Put a partition into writable mode. | set read_write ‘my_range_0’ |
SET PARALLEL_DEGREE | Set the maximum degree for parallel query. | set parallel_degree 8 |
SET PARALLEL_REQUESTS | Set the maximum requests for parallel query. | set parallel_requests 512 |
SET MAX_CACHE_ROWS | Set the maximum rows for cache resultset. | set max_cache_rows 8 |
SET MAX_IDLE_TIME | Set the maximum idle time for pooled connection. | set max_idle_time 60 |
ADD MASTER | Add a new master(rw) backend to oneproxy. | add master ‘192.168.1.120:3306@default’ |
ADD SLAVE | Add a new slave(ro) backend to oneproxy. | add slave ‘192.168.1.120:3306@default’ |
ADD SEQUENCE | Add a new sequence into oneproxy. | add sequence ‘seq_name’ |
SAVE TABLES | save partition tables to a text file. | save tables ‘/tmp/part.txt’ |
SAVE SQL | save sql list to a text file. | save sql ‘/tmp/firewall.sql’ |
SAVE BLACKSQL | save black sql list to a text file. | save blacksql ‘/tmp/blacksql.sql’ |
SAVE DML | save dml sql list to a text file. | save dml ‘/tmp/firewall_dml.sql’ |
SAVE NEW | save new sql list to a text file. | save new ‘/tmp/firewall_new.sql’ |
SAVE IP | save ip list to a text file. | save ip ‘/tmp/firewall.sql’ |
LOAD TABLES | load partition tables from text file. | load tables ‘/tmp/part.txt’ |
LOAD SQL | load firewall sql from text file. | load sql ‘/tmp/firewall.sql’ |
LOAD BLACKSQL | load black sql from text file. | load blacksql ‘/tmp/blacksql.sql’ |
LOAD IP | load firewall ip list from text file. | load ip ‘/tmp/firewall_ip.sql’ |
MAP | Map a table name to a given server group. | map my_test1_0 default |
UNMAP | Remove the server gruop for a given table. | unmap my_test1_0 |
PASSWD | Encrypt the password of command options. | passwd test |
SHUTDOWN | Shutdown the proxy. | shutdown force |
9. 开发中的注意事项 (这个是在网上找的,可供参考)
1、不支持 Server Side Cursor 接口,比如 mysql C API 里的 Prepare、Bind、 Execute调用接口
2、不支持 use命令切换数据库
3、默认禁止 CALL, PREPARE, EXECUTE, DEALLOCATE 命令,也就是说不能用存储过程
4、单库(单实例)分表–insert/update/delete要加字段名,如insert into t1(id,name) values(1,’张三’);
5、单库(单实例)分表–目前分了N张表,如果以自增id做关联查询,那么每张表的自增id都是从1开始,在与其他表join关联查询时,数据会不准确
6、单库(单实例)分表–当where条件有分区列时,值不能有函数转换,也不能有算术表达式,必须是原子值,否则结果不准确
7、分库分表(多实例)–不支持垮库join,例如user_0表在10.0.0.1机器里,现在要join关联查询10.0.0.2机器里的money_detail表,不支持
8、分库分表(多实例)–不支持分布式事务,例如user_0表在10.0.0.1机器里,user_1表在10.0.0.2机器里,现在想同时update更新两张表,不支持
9、读写分离 –默认读操作全部访问slave,如果想强制走主库,例如涉及金钱类的查询操作,SQL改为select /master/ from t1 where id=1;
10、分库分表/分表 –where条件带分区列时,直接命中该表,如果未带分区列,会逐一扫描所有分表(单线程),考虑性能问题,要加并行查询(多线程),SQL改为select /parallel/ from t1 where name=’李四’; 并行查询会增加额外的CPU消耗
11、分表规则:支持range(范围),hash(取模),hash规则要提前规划好,具体分多少张表,如前期分64张表,1年后想扩容128张表,数据需要重新导出导入,成本非常高,目前二级分表还不支持。
以上是关于Mysql 中间件 oneProxy总结的主要内容,如果未能解决你的问题,请参考以下文章