MyCat 学习笔记 第十篇.数据分片 之 ER分片

Posted 2020-06-10

 1 应用场景

这篇来说下mycat中自带的er关系分片，所谓er关系分片即可以理解为有关联关系表之间数据分片。类似于订单主表与订单详情表间的分片存储规则。

本文所说的er分片分为两种：

a. 依据主键进行数据分片，验证发现主表数据保存在第1个datanode中，子表数据根据分片规则存储。

b. 依据分片关键字段进行分片，验证发现主表与子表根据分片规则存储，且保存在相同的分片内。

 

接下来，可以下实际配置与数据验证

 

2 环境说明

 参考  《MyCat 学习笔记》第六篇.数据分片 之 按月数据分片  http://www.cnblogs.com/kaye0110/p/5160826.html

 

3 参数配置

3.1 server.xml 配置

同上参考

3.2 schema.xml 配置

 

<schema name="RANGEDB" checkSQLschema="false" sqlMaxLimit="100">
　　<!-- 方案a. 以主键进行分片 ，采用 sharding-by-pid 规则（根据 t_er_child_1.cid 的值 / 256 进行分片） -->

　　<table name="t_er_parent_1" dataNode="dn$4-7" rule="sharding-by-pid">
　　　　<childTable name="t_er_child_1" primaryKey="cid" joinKey="pid" parentKey="pid" />
　　</table>

　　<!-- 方案b.根据主表分片字段分页，子表数据会跟着主表走 -->

　　<table name="t_er_order" dataNode="dn$4-7" rule="sharding-by-long">
　　　　<childTable name="t_er_detail" primaryKey="detail_id" joinKey="order_id" parentKey="order_id" />
　　</table>

</schema>

 

3.3 rule.xml 配置

 

<tableRule name="sharding-by-long">
　　<rule>
　　　　<columns>sharding_long</columns>
　　　　<algorithm>func2</algorithm>
　　</rule>
</tableRule>

<!--  partitionCount 与 partitionLength 相乘的值需要等于 1024  -->

<function name="func2" class="org.opencloudb.route.function.PartitionByLong">
　　<property name="partitionCount">4</property>
　　<property name="partitionLength">256</property>
</function>

 

4 数据验证

4.1 方案a的验证

 

CREATE TABLE `t_er_parent_1` (
`pid` INT NOT NULL,
`pname` VARCHAR(45) NULL,
PRIMARY KEY (`pid`));

CREATE TABLE `t_er_child_1` (
`cid` INT NOT NULL,
`pid` INT NOT NULL,
`cname` VARCHAR(45) NULL,
PRIMARY KEY (`cid`));

 

insert into t_er_parent_1 (pid,pname) values (1,‘parent 255‘);
insert into t_er_child_1(cid,pid,cname) values (1,255,‘child 255‘);

insert into t_er_parent_1 (pid,pname) values (2,‘parent 256‘);
insert into t_er_child_1(cid,pid,cname) values (2,256,‘child 256‘);

insert into t_er_parent_1 (pid,pname) values (3,‘parent 512‘);
insert into t_er_child_1(cid,pid,cname) values (3,512,‘child 512‘);

insert into t_er_parent_1 (pid,pname) values (4,‘parent 768‘);
insert into t_er_child_1(cid,pid,cname) values (4,768,‘child 768‘);

mysql> select * from t_er_parent_1;
+-----+------------+
| pid | pname |
+-----+------------+
| 1 | parent 255 |
| 2 | parent 256 |
| 3 | parent 512 |
| 4 | parent 768 |
+-----+------------+
4 rows in set (0.01 sec)

mysql> select * from t_er_child_1;
+-----+-----+-----------+
| cid | pid | cname |
+-----+-----+-----------+
| 1 | 255 | child 255 |
| 4 | 768 | child 768 |
| 2 | 256 | child 256 |
| 3 | 512 | child 512 |
+-----+-----+-----------+
4 rows in set (0.01 sec)

 

看下方案a的日志 ，所有主表数据都进入了第1个数据结点dn4，不过子表数据进入dn7

01/28 21:44:25.559 DEBUG [$_NIOREACTOR-0-RW] (ServerQueryHandler.java:56) -ServerConnection [id=2, schema=RANGEDB, host=0:0:0:0:0:0:0:1, user=test,txIsolation=3, autocommit=true, schema=RANGEDB]insert into t_er_parent_1 (pid,pname) values (4,‘parent 768‘)
01/28 21:44:25.559 DEBUG [$_NIOREACTOR-0-RW] (NonBlockingSession.java:113) -ServerConnection [id=2, schema=RANGEDB, host=0:0:0:0:0:0:0:1, user=test,txIsolation=3, autocommit=true, schema=RANGEDB]insert into t_er_parent_1 (pid,pname) values (4,‘parent 768‘), route={
1 -> dn4{insert into t_er_parent_1 (pid,pname) values (4,‘parent 768‘)}
} rrs
01/28 21:44:25.562 DEBUG [$_NIOREACTOR-1-RW] (NonBlockingSession.java:229) -release connection MySQLConnection [id=15, lastTime=1453988665553, user=root, schema=range_db_4, old shema=range_db_4, borrowed=true, fromSlaveDB=false, threadId=179, charset=utf8, txIsolation=3, autocommit=true, attachment=dn4{insert into t_er_parent_1 (pid,pname) values (4,‘parent 768‘)}, respHandler=SingleNodeHandler [node=dn4{insert into t_er_parent_1 (pid,pname) values (4,‘parent 768‘)}, packetId=0], host=localhost, port=3306, statusSync=null, writeQueue=0, modifiedSQLExecuted=true]
01/28 21:44:25.562 DEBUG [$_NIOREACTOR-1-RW] (PhysicalDatasource.java:403) -release channel MySQLConnection [id=15, lastTime=1453988665553, user=root, schema=range_db_4, old shema=range_db_4, borrowed=true, fromSlaveDB=false, threadId=179, charset=utf8, txIsolation=3, autocommit=true, attachment=null, respHandler=null, host=localhost, port=3306, statusSync=null, writeQueue=0, modifiedSQLExecuted=false]
01/28 21:44:25.564 DEBUG [$_NIOREACTOR-0-RW] (ServerQueryHandler.java:56) -ServerConnection [id=2, schema=RANGEDB, host=0:0:0:0:0:0:0:1, user=test,txIsolation=3, autocommit=true, schema=RANGEDB]insert into t_er_child_1(cid,pid,cname) values (4,768,‘child 768‘)
01/28 21:44:25.565 DEBUG [$_NIOREACTOR-0-RW] (RouterUtil.java:650) -found partion node (using parent partion rule directly) for child table to insert dn7 sql :INSERT INTO t_er_child_1 (cid, pid, cname)
VALUES (4, 768, ‘child 768‘)
01/28 21:44:25.565 DEBUG [$_NIOREACTOR-0-RW] (NonBlockingSession.java:113) -ServerConnection [id=2, schema=RANGEDB, host=0:0:0:0:0:0:0:1, user=test,txIsolation=3, autocommit=true, schema=RANGEDB]insert into t_er_child_1(cid,pid,cname) values (4,768,‘child 768‘), route={
1 -> dn7{INSERT INTO t_er_child_1 (cid, pid, cname)
VALUES (4, 768, ‘child 768‘)}
} rrs

 

看下物理表的数据 range_db_4 对应的就是dn4结点。

mysql> select * from range_db_4.t_er_parent_1;

select * from range_db_4.t_er_child_1;
+-----+------------+
| pid | pname |
+-----+------------+
| 1 | parent 255 |
| 2 | parent 256 |
| 3 | parent 512 |
| 4 | parent 768 |
+-----+------------+
4 rows in set (0.00 sec)

+-----+-----+-----------+
| cid | pid | cname |
+-----+-----+-----------+
| 1 | 255 | child 255 |
+-----+-----+-----------+
1 row in set (0.00 sec)

 

 

4.2 方案b的验证

CREATE TABLE `t_er_order` (
`order_id` INT NOT NULL,
`prod_info` VARCHAR(45) NULL,
`sharding_long` VARCHAR(45) NULL,
PRIMARY KEY (`order_id`));

CREATE TABLE `t_er_detail` (
`detail_id` INT NOT NULL,
`order_id` INT NOT NULL,
`detail_info` INT NOT NULL,
PRIMARY KEY (`detail_id`));

insert into t_er_order (order_id,prod_info,sharding_long) values (1,‘prod_1‘,200);
insert into t_er_order (order_id,prod_info,sharding_long) values (2,‘prod_256‘,256);
insert into t_er_order (order_id,prod_info,sharding_long) values (3,‘prod_512‘,512);
insert into t_er_order (order_id,prod_info,sharding_long) values (4,‘prod_1024‘,1024);
insert into t_er_order (order_id,prod_info,sharding_long) values (5,‘prod_1025‘,1025);
insert into t_er_order (order_id,prod_info,sharding_long) values (6,‘prod_795‘,795);

insert into t_er_detail (detail_id,order_id,detail_info) values (1,1,200);
insert into t_er_detail (detail_id,order_id,detail_info) values (2,2,256);
insert into t_er_detail (detail_id,order_id,detail_info) values (3,3,512);
insert into t_er_detail (detail_id,order_id,detail_info) values (4,4,1024);
insert into t_er_detail (detail_id,order_id,detail_info) values (5,5,1025);
insert into t_er_detail (detail_id,order_id,detail_info) values (6,6,795);

 

mysql> select * from t_er_order;
+----------+-----------+---------------+
| order_id | prod_info | sharding_long |
+----------+-----------+---------------+
| 1 | prod_1 | 200 |
| 4 | prod_1024 | 1024 |
| 5 | prod_1025 | 1025 |
| 2 | prod_256 | 256 |
| 3 | prod_512 | 512 |
| 6 | prod_795 | 795 |
+----------+-----------+---------------+
6 rows in set (0.08 sec)

mysql> select * from t_er_detail;
+-----------+----------+-------------+
| detail_id | order_id | detail_info |
+-----------+----------+-------------+
| 2 | 2 | 256 |
| 1 | 1 | 200 |
| 4 | 4 | 1024 |
| 5 | 5 | 1025 |
| 3 | 3 | 512 |
| 6 | 6 | 795 |
+-----------+----------+-------------+
6 rows in set (0.01 sec)

 

 

查询物理表

mysql> select * from range_db_6.t_er_order;select * from range_db_6.t_er_detail;

+----------+-----------+---------------+
| order_id | prod_info | sharding_long |
+----------+-----------+---------------+
| 3 | prod_512 | 512 |
+----------+-----------+---------------+
1 row in set (0.00 sec)

+-----------+----------+-------------+
| detail_id | order_id | detail_info |
+-----------+----------+-------------+
| 3 | 3 | 512 |
+-----------+----------+-------------+
1 row in set (0.00 sec)

 

再来看下日志

新增主表数据，根据分片规则数据应该进入第2个数据结点 dn5

01/28 20:50:10.274 DEBUG [$_NIOREACTOR-0-RW] (NonBlockingSession.java:113) -ServerConnection [id=1, schema=RANGEDB, host=0:0:0:0:0:0:0:1, user=test,txIsolation=3, autocommit=true, schema=RANGEDB]insert into t_er_order (order_id,prod_info,sharding_long) values (2,‘prod_256‘,256), route={
1 -> dn5{insert into t_er_order (order_id,prod_info,sharding_long) values (2,‘prod_256‘,256)}
} rrs

份数据库中进行主键搜索确认，并进入数据缓存

 

01/28 20:50:10.544 DEBUG [BusinessExecutor2] (EnchachePool.java:76) -ER_SQL2PARENTID miss cache ,key:RANGEDB:select t_er_order.order_id from t_er_order where t_er_order.order_id=2
01/28 20:50:10.544 DEBUG [BusinessExecutor2] (FetchStoreNodeOfChildTableHandler.java:73) -find child node with sql:select t_er_order.order_id from t_er_order where t_er_order.order_id=2
01/28 20:50:10.545 DEBUG [BusinessExecutor2] (FetchStoreNodeOfChildTableHandler.java:81) -execute in datanode dn4
01/28 20:50:10.545 DEBUG [BusinessExecutor2] (PhysicalDBPool.java:452) -select read source hostM3306 for dataHost:localhost3306
01/28 20:50:10.545 DEBUG [$_NIOREACTOR-0-RW] (PhysicalDatasource.java:403) -release channel MySQLConnection [id=16, lastTime=1453985410540, user=root, schema=range_db_4, old shema=range_db_4, borrowed=true, fromSlaveDB=false, threadId=175, charset=utf8, txIsolation=3, autocommit=true, attachment=null, respHandler=null, host=localhost, port=3306, statusSync=null, writeQueue=0, modifiedSQLExecuted=false]
01/28 20:50:10.745 DEBUG [BusinessExecutor2] (FetchStoreNodeOfChildTableHandler.java:81) -execute in datanode dn5
01/28 20:50:10.745 DEBUG [BusinessExecutor2] (PhysicalDBPool.java:452) -select read source hostM3306 for dataHost:localhost3306
01/28 20:50:10.746 DEBUG [$_NIOREACTOR-1-RW] (FetchStoreNodeOfChildTableHandler.java:154) -received rowResponse response,2 from MySQLConnection [id=19, lastTime=1453985410743, user=root, schema=range_db_5, old shema=range_db_5, borrowed=true, fromSlaveDB=false, threadId=176, charset=utf8, txIsolation=3, autocommit=true, attachment=dn5, respHandler=org[email protected]6be897cc, host=localhost, port=3306, statusSync=null, writeQueue=0, modifiedSQLExecuted=false]
01/28 20:50:10.746 DEBUG [$_NIOREACTOR-1-RW] (PhysicalDatasource.java:403) -release channel MySQLConnection [id=19, lastTime=1453985410743, user=root, schema=range_db_5, old shema=range_db_5, borrowed=true, fromSlaveDB=false, threadId=176, charset=utf8, txIsolation=3, autocommit=true, attachment=null, respHandler=null, host=localhost, port=3306, statusSync=null, writeQueue=0, modifiedSQLExecuted=false]

明确插入dn5结点
01/28 20:50:10.945 DEBUG [BusinessExecutor0] (RouterUtil.java:1213) -found partion node for child table to insert dn5 sql :insert into t_er_detail (detail_id,order_id,detail_info) values (2,2,256)

 

 

本篇完。