javaer你还在手写分表分库?来看看这个框架怎么做的 干货满满
Posted 薛家明
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了javaer你还在手写分表分库?来看看这个框架怎么做的 干货满满相关的知识，希望对你有一定的参考价值。
java orm框架easy-query分库分表之分表
高并发三驾马车:分库分表、MQ、缓存。今天给大家带来的就是分库分表的干货解决方案,哪怕你不用我的框架也可以从中听到不一样的结局方案和实现。
一款支持自动分表分库的orm框架easy-query 帮助您解脱跨库带来的复杂业务代码,并且提供多种结局方案和自定义路由来实现比中间件更高性能的数据库访问。
目前市面上有的分库分表JAVA组件有很多：中间件代理有:sharding-sphere(proxy),mycat 客户端JDBC:sharding-sphere（jdbc）等等，中间件因为代理了一层会导致所有的sql执行都要经过中间件，性能会大大折扣,但是因为中间部署可以提供更加省的连接池，客户端无需代理，仅需对sql进行分析即可实现，但是越靠近客户的模式可以优化的性能越高，所以本次带来的框架可以提供前所未有的分片规则自由和前所未有的便捷高性能。
本文 demo地址 https://github.com/xuejmnet/easy-sharding-test
怎么样的orm算是支持分表分库
首先orm是否支持分表分库不仅仅是看框架是否支持动态修改表名，让数据正确存入对应的表或者修改对应的数据，这些说实话都是最最简单的实现，真正需要支持分库分表那么需要orm实现复杂的跨表聚合查询，这才是分表分库的精髓，很显然目前的orm很少有支持的。接下来我将给大家演示基于springboot3.x的分表分库演示,取模分片和时间分片。本章我们主要以使用为主后面下一章我们来讲解优化方案,包括原理解析,后续有更多的关于分表分库的经验是博主多年下来的实战经验分享给大家保证大家的happy coding。
初始化项目
进入 https://start.spring.io/ 官网直接下载
安装依赖
<!-- https://mvnrepository.com/artifact/com.alibaba/druid -->
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>druid</artifactId>
<version>1.2.15</version>
</dependency>
<!-- mysql驱动 -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.17</version>
</dependency>
<dependency>
<groupId>com.easy-query</groupId>
<artifactId>sql-springboot-starter</artifactId>
<version>0.9.7</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.18</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
application.yml配置
server:
port: 8080
spring:
datasource:
type: com.alibaba.druid.pool.DruidDataSource
driver-class-name: com.mysql.cj.jdbc.Driver
url: jdbc:mysql://127.0.0.1:3306/easy-sharding-test?serverTimezone=GMT%2B8&characterEncoding=utf-8&useSSL=false&allowMultiQueries=true&rewriteBatchedStatements=true
username: root
password: root
logging:
level:
com.easy.query.core: debug
easy-query:
enable: true
name-conversion: underlined
database: mysql
取模
常见的分片方式之一就是取模分片,取模分片可以让以分片键为条件的处理完美路由到对应的表,性能上来说非常非常高，但是局限性也是很大的因为无意义的id路由会导致仅支持这一个id条件而不支持其他条件的路由，只能全分片表扫描来获取对应的数据，但是他的实现和理解也是最容易的，当然后续还有基因分片一种可以部分解决仅支持id带来的问题不过也并不是非常的完美。
简单的取模分片
我们本次测试案例采用order表对其进行5表拆分:order_00,order_01,order_02,order_03,order_04,采用订单id取模进行分表
数据库脚本
CREATE DATABASE IF NOT EXISTS `easy-sharding-test` CHARACTER SET \'utf8mb4\';
USE `easy-sharding-test`;
create table order_00
(
id varchar(32) not null comment \'主键ID\'primary key,
uid varchar(50) not null comment \'用户id\',
order_no int null comment \'订单号\'
)comment \'订单表\';
create table order_01
(
id varchar(32) not null comment \'主键ID\'primary key,
uid varchar(50) not null comment \'用户id\',
order_no int null comment \'订单号\'
)comment \'订单表\';
create table order_02
(
id varchar(32) not null comment \'主键ID\'primary key,
uid varchar(50) not null comment \'用户id\',
order_no int null comment \'订单号\'
)comment \'订单表\';
create table order_03
(
id varchar(32) not null comment \'主键ID\'primary key,
uid varchar(50) not null comment \'用户id\',
order_no int null comment \'订单号\'
)comment \'订单表\';
create table order_04
(
id varchar(32) not null comment \'主键ID\'primary key,
uid varchar(50) not null comment \'用户id\',
order_no int null comment \'订单号\'
)comment \'订单表\';
//定义了一个对象并且设置表名和分片初始化器`shardingInitializer`,设置id为主键，并且设置id为分表建
@Data
@Table(value = "order",shardingInitializer = OrderShardingInitializer.class)
public class OrderEntity
@Column(primaryKey = true)
@ShardingTableKey
private String id;
private String uid;
private Integer orderNo;
//编写订单取模初始化器,只需要实现两个方法，当然你也可以自己实现对应的`EntityShardingInitializer`这边是继承`easy-query`框架提供的分片取模初始化器
@Component
public class OrderShardingInitializer extends AbstractShardingModInitializer<OrderEntity>
/**
* 设置模几我们模5就设置5
* @return
*/
@Override
protected int mod()
return 5;
/**
* 编写模5后的尾巴长度默认我们设置2就是左补0
* @return
*/
@Override
protected int tailLength()
return 2;
//编写分片规则`AbstractModTableRule`由框架提供取模分片路由规则,如果需要自己实现可以继承`AbstractTableRouteRule`这个抽象类
@Component
public class OrderTableRouteRule extends AbstractModTableRule<OrderEntity>
@Override
protected int mod()
return 5;
@Override
protected int tailLength()
return 2;
初始化工作做好了开始编写代码
新增初始化
@RestController
@RequestMapping("/order")
@RequiredArgsConstructor(onConstructor_ = @Autowired)
public class OrderController
private final EasyQuery easyQuery;
@GetMapping("/init")
public Object init()
ArrayList<OrderEntity> orderEntities = new ArrayList<>(100);
List<String> users = Arrays.asList("xiaoming", "xiaohong", "xiaolan");
for (int i = 0; i < 100; i++)
OrderEntity orderEntity = new OrderEntity();
orderEntity.setId(String.valueOf(i));
int i1 = i % 3;
String uid = users.get(i1);
orderEntity.setUid(uid);
orderEntity.setOrderNo(i);
orderEntities.add(orderEntity);
long l = easyQuery.insertable(orderEntities).executeRows();
return "成功插入:"+l;
查询单条
按分片键查询
可以完美的路由到对应的数据库表和操作单表拥有一样的性能
@GetMapping("/first")
public Object first(@RequestParam("id") String id)
OrderEntity orderEntity = easyQuery.queryable(OrderEntity.class)
.whereById(id).firstOrNull();
return orderEntity;
http://localhost:8080/order/first?id=20
"id":"20","uid":"xiaolan","orderNo":20
http-nio-8080-exec-1, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no` FROM `order_03` t WHERE t.`id` = ? LIMIT 1
==> http-nio-8080-exec-1, name:ds0, Parameters: 20(String)
<== Total: 1
日志稍微解释一下
- http-nio-8080-exec-1表示当前语句执行的线程,默认多个分片聚合后需要再线程池中查询数据后聚合返回。
- name:ds0 表示数据源叫做ds0，如果不分库那么这个数据源可以忽略,也可以自己指定配置文件中或者设置
defaultDataSourceName
全程无需您去计算路由到哪里,并且规则和业务代码已经脱离解耦
不按分片键查询
当我们的查询为非分片键查询那么会导致路由需要进行全分片扫描然后来获取对应的数据进行判断哪个时我们要的
@GetMapping("/firstByUid")
public Object firstByUid(@RequestParam("uid") String uid)
OrderEntity orderEntity = easyQuery.queryable(OrderEntity.class)
.where(o->o.eq(OrderEntity::getUid,uid)).firstOrNull();
return orderEntity;
http://localhost:8080/order/firstByUid?uid=xiaoming
"id":"18","uid":"xiaoming","orderNo":18
//这边把日志精简了一下可以看到他是开启了5个线程进行分片查询
==> SHARDING_EXECUTOR_1, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no` FROM `order_00` t WHERE t.`uid` = ? LIMIT 1
==> SHARDING_EXECUTOR_5, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no` FROM `order_03` t WHERE t.`uid` = ? LIMIT 1
==> SHARDING_EXECUTOR_4, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no` FROM `order_04` t WHERE t.`uid` = ? LIMIT 1
==> SHARDING_EXECUTOR_3, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no` FROM `order_02` t WHERE t.`uid` = ? LIMIT 1
==> SHARDING_EXECUTOR_2, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no` FROM `order_01` t WHERE t.`uid` = ? LIMIT 1
==> SHARDING_EXECUTOR_3, name:ds0, Parameters: xiaoming(String)
==> SHARDING_EXECUTOR_4, name:ds0, Parameters: xiaoming(String)
==> SHARDING_EXECUTOR_5, name:ds0, Parameters: xiaoming(String)
==> SHARDING_EXECUTOR_1, name:ds0, Parameters: xiaoming(String)
==> SHARDING_EXECUTOR_2, name:ds0, Parameters: xiaoming(String)
<== Total: 1
因为uid不是分片键所以在分片查询的时候需要遍历所有的表然后返回对应的数据，可能有同学会问就这?当然这只是简单演示后续下一篇我会给出具体的优化方案来进行处理。
分页查询
分片后的分页查询是分片下的一个难点,这边框架自带功能，分片后分页之所以难是因为如果是自行实现业务代码会变得非常复杂，有一种非常简易的方式就是把分页重写pageIndex永远为1，然后全部取到内存后在进行stream过滤，但是带来的另一个问题就是pageIndex不能便宜过大不然内存会完全存不下导致内存爆炸，并且如果翻页到最后几页那将是灾难性的，给程序带来极其不稳定,但是easy-query提供了和sharding-sphere一样的分片聚合方式并且因为靠近业务的关系所以可以有效的优化深度分页pageIndex过大
@GetMapping("/page")
public Object page(@RequestParam("pageIndex") Integer pageIndex,@RequestParam("pageSize") Integer pageSize)
EasyPageResult<OrderEntity> pageResult = easyQuery.queryable(OrderEntity.class)
.orderByAsc(o -> o.column(OrderEntity::getOrderNo))
.toPageResult(pageIndex, pageSize);
return pageResult;
http://localhost:8080/order/page?pageIndex=1&pageSize=10
"total":100,"data":["id":"0","uid":"xiaoming","orderNo":0,"id":"1","uid":"xiaohong","orderNo":1,"id":"2","uid":"xiaolan","orderNo":2,"id":"3","uid":"xiaoming","orderNo":3,"id":"4","uid":"xiaohong","orderNo":4,"id":"5","uid":"xiaolan","orderNo":5,"id":"6","uid":"xiaoming","orderNo":6,"id":"7","uid":"xiaohong","orderNo":7,"id":"8","uid":"xiaolan","orderNo":8,"id":"9","uid":"xiaoming","orderNo":9]
==> SHARDING_EXECUTOR_3, name:ds0, Preparing: SELECT COUNT(1) FROM `order_02` t
==> SHARDING_EXECUTOR_2, name:ds0, Preparing: SELECT COUNT(1) FROM `order_03` t
==> SHARDING_EXECUTOR_5, name:ds0, Preparing: SELECT COUNT(1) FROM `order_04` t
==> SHARDING_EXECUTOR_1, name:ds0, Preparing: SELECT COUNT(1) FROM `order_01` t
==> SHARDING_EXECUTOR_4, name:ds0, Preparing: SELECT COUNT(1) FROM `order_00` t
<== Total: 1
==> SHARDING_EXECUTOR_5, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no` FROM `order_04` t ORDER BY t.`order_no` ASC LIMIT 10
==> SHARDING_EXECUTOR_2, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no` FROM `order_03` t ORDER BY t.`order_no` ASC LIMIT 10
==> SHARDING_EXECUTOR_4, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no` FROM `order_00` t ORDER BY t.`order_no` ASC LIMIT 10
==> SHARDING_EXECUTOR_1, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no` FROM `order_01` t ORDER BY t.`order_no` ASC LIMIT 10
==> SHARDING_EXECUTOR_3, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no` FROM `order_02` t ORDER BY t.`order_no` ASC LIMIT 10
<== Total: 10
这边可以看到一行代码实现分页，下面是第二页
http://localhost:8080/order/page?pageIndex=2&pageSize=10
"total":100,"data":["id":"10","uid":"xiaohong","orderNo":10,"id":"11","uid":"xiaolan","orderNo":11,"id":"12","uid":"xiaoming","orderNo":12,"id":"13","uid":"xiaohong","orderNo":13,"id":"14","uid":"xiaolan","orderNo":14,"id":"15","uid":"xiaoming","orderNo":15,"id":"16","uid":"xiaohong","orderNo":16,"id":"17","uid":"xiaolan","orderNo":17,"id":"18","uid":"xiaoming","orderNo":18,"id":"19","uid":"xiaohong","orderNo":19]
==> SHARDING_EXECUTOR_9, name:ds0, Preparing: SELECT COUNT(1) FROM `order_02` t
==> SHARDING_EXECUTOR_8, name:ds0, Preparing: SELECT COUNT(1) FROM `order_01` t
==> SHARDING_EXECUTOR_10, name:ds0, Preparing: SELECT COUNT(1) FROM `order_04` t
==> SHARDING_EXECUTOR_7, name:ds0, Preparing: SELECT COUNT(1) FROM `order_03` t
==> SHARDING_EXECUTOR_6, name:ds0, Preparing: SELECT COUNT(1) FROM `order_00` t
<== Total: 1
==> SHARDING_EXECUTOR_9, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no` FROM `order_01` t ORDER BY t.`order_no` ASC LIMIT 20
==> SHARDING_EXECUTOR_8, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no` FROM `order_03` t ORDER BY t.`order_no` ASC LIMIT 20
==> SHARDING_EXECUTOR_10, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no` FROM `order_04` t ORDER BY t.`order_no` ASC LIMIT 20
==> SHARDING_EXECUTOR_6, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no` FROM `order_02` t ORDER BY t.`order_no` ASC LIMIT 20
==> SHARDING_EXECUTOR_7, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no` FROM `order_00` t ORDER BY t.`order_no` ASC LIMIT 20
<== Total: 10
按时间分表
这边我们简单还是以order订单为例,按月进行分片假设我们从2022年1月到2023年5月一共17个月表名为t_order_202201、t_order_202202、t_order_202203...t_order_202304、t_order_202305
数据库脚本
create table t_order_202201
(
id varchar(32) not null comment \'主键ID\'primary key,
uid varchar(50) not null comment \'用户id\',
order_no int not null comment \'订单号\',
create_time datetime not null comment \'创建时间\'
)comment \'订单表\';
create table t_order_202202
(
id varchar(32) not null comment \'主键ID\'primary key,
uid varchar(50) not null comment \'用户id\',
order_no int not null comment \'订单号\',
create_time datetime not null comment \'创建时间\'
)comment \'订单表\';
....
create table t_order_202304
(
id varchar(32) not null comment \'主键ID\'primary key,
uid varchar(50) not null comment \'用户id\',
order_no int not null comment \'订单号\',
create_time datetime not null comment \'创建时间\'
)comment \'订单表\';
create table t_order_202305
(
id varchar(32) not null comment \'主键ID\'primary key,
uid varchar(50) not null comment \'用户id\',
order_no int not null comment \'订单号\',
create_time datetime not null comment \'创建时间\'
)comment \'订单表\';
@Data
@Table(value = "t_order",shardingInitializer = OrderByMonthShardingInitializer.class)
public class OrderByMonthEntity
@Column(primaryKey = true)
private String id;
private String uid;
private Integer orderNo;
/**
* 分片键改为时间
*/
@ShardingTableKey
private LocalDateTime createTime;
//路由规则可以直接继承AbstractShardingMonthInitializer也可以自己实现
@Component
public class OrderByMonthShardingInitializer extends AbstractShardingMonthInitializer<OrderByMonthEntity>
/**
* 开始时间不可以使用LocalDateTime.now()因为会导致每次启动开始时间都不一样
* @return
*/
@Override
protected LocalDateTime getBeginTime()
return LocalDateTime.of(2022,1,1,0,0);
/**
* 如果不设置那么就是当前时间,用于程序启动后自动计算应该有的表包括最后时间
* @return
*/
@Override
protected LocalDateTime getEndTime()
return LocalDateTime.of(2023,5,31,0,0);
@Override
public void configure0(ShardingEntityBuilder<OrderByMonthEntity> builder)
//后续用来实现优化分表
//按月分片路由规则也可以自己实现因为框架已经封装好了所以可以用框架自带的
@Component
public class OrderByMonthTableRouteRule extends AbstractMonthTableRule<OrderByMonthEntity>
@Override
protected LocalDateTime convertLocalDateTime(Object shardingValue)
return (LocalDateTime)shardingValue;
初始化
@RestController
@RequestMapping("/orderMonth")
@RequiredArgsConstructor(onConstructor_ = @Autowired)
public class OrderMonthController
private final EasyQuery easyQuery;
@GetMapping("/init")
public Object init()
ArrayList<OrderByMonthEntity> orderEntities = new ArrayList<>(100);
List<String> users = Arrays.asList("xiaoming", "xiaohong", "xiaolan");
LocalDateTime beginTime=LocalDateTime.of(2022,1,1,0,0);
LocalDateTime endTime=LocalDateTime.of(2023,5,31,0,0);
int i=0;
while(!beginTime.isAfter(endTime))
OrderByMonthEntity orderEntity = new OrderByMonthEntity();
orderEntity.setId(String.valueOf(i));
int i1 = i % 3;
String uid = users.get(i1);
orderEntity.setUid(uid);
orderEntity.setOrderNo(i);
orderEntity.setCreateTime(beginTime);
orderEntities.add(orderEntity);
beginTime=beginTime.plusDays(1);
i++;
long l = easyQuery.insertable(orderEntities).executeRows();
return "成功插入:"+l;
http://localhost:8080/orderMonth/init
成功插入:516
获取第一条数据
@GetMapping("/first")
public Object first(@RequestParam("id") String id)
OrderEntity orderEntity = easyQuery.queryable(OrderEntity.class)
.whereById(id).firstOrNull();
return orderEntity;
http://localhost:8080/orderMonth/first?id=11
"id":"11","uid":"xiaolan","orderNo":11,"createTime":"2022-01-12T00:00:00"
//以每5组一个次并发执行聚合
==> SHARDING_EXECUTOR_1, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no`,t.`create_time` FROM `t_order_202205` t WHERE t.`id` = ? LIMIT 1
==> SHARDING_EXECUTOR_1, name:ds0, Parameters: 11(String)
==> SHARDING_EXECUTOR_2, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no`,t.`create_time` FROM `t_order_202207` t WHERE t.`id` = ? LIMIT 1
==> SHARDING_EXECUTOR_2, name:ds0, Parameters: 11(String)
==> SHARDING_EXECUTOR_3, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no`,t.`create_time` FROM `t_order_202303` t WHERE t.`id` = ? LIMIT 1
==> SHARDING_EXECUTOR_3, name:ds0, Parameters: 11(String)
==> SHARDING_EXECUTOR_4, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no`,t.`create_time` FROM `t_order_202212` t WHERE t.`id` = ? LIMIT 1
==> SHARDING_EXECUTOR_4, name:ds0, Parameters: 11(String)
==> SHARDING_EXECUTOR_5, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no`,t.`create_time` FROM `t_order_202302` t WHERE t.`id` = ? LIMIT 1
==> SHARDING_EXECUTOR_5, name:ds0, Parameters: 11(String)
==> SHARDING_EXECUTOR_1, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no`,t.`create_time` FROM `t_order_202304` t WHERE t.`id` = ? LIMIT 1
==> SHARDING_EXECUTOR_5, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no`,t.`create_time` FROM `t_order_202206` t WHERE t.`id` = ? LIMIT 1
==> SHARDING_EXECUTOR_2, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no`,t.`create_time` FROM `t_order_202305` t WHERE t.`id` = ? LIMIT 1
==> SHARDING_EXECUTOR_1, name:ds0, Parameters: 11(String)
==> SHARDING_EXECUTOR_2, name:ds0, Parameters: 11(String)
==> SHARDING_EXECUTOR_4, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no`,t.`create_time` FROM `t_order_202209` t WHERE t.`id` = ? LIMIT 1
==> SHARDING_EXECUTOR_3, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no`,t.`create_time` FROM `t_order_202204` t WHERE t.`id` = ? LIMIT 1
==> SHARDING_EXECUTOR_5, name:ds0, Parameters: 11(String)
==> SHARDING_EXECUTOR_3, name:ds0, Parameters: 11(String)
==> SHARDING_EXECUTOR_4, name:ds0, Parameters: 11(String)
==> SHARDING_EXECUTOR_2, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no`,t.`create_time` FROM `t_order_202208` t WHERE t.`id` = ? LIMIT 1
==> SHARDING_EXECUTOR_5, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no`,t.`create_time` FROM `t_order_202201` t WHERE t.`id` = ? LIMIT 1
==> SHARDING_EXECUTOR_3, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no`,t.`create_time` FROM `t_order_202210` t WHERE t.`id` = ? LIMIT 1
==> SHARDING_EXECUTOR_5, name:ds0, Parameters: 11(String)
==> SHARDING_EXECUTOR_4, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no`,t.`create_time` FROM `t_order_202202` t WHERE t.`id` = ? LIMIT 1
==> SHARDING_EXECUTOR_3, name:ds0, Parameters: 11(String)
==> SHARDING_EXECUTOR_2, name:ds0, Parameters: 11(String)
==> SHARDING_EXECUTOR_4, name:ds0, Parameters: 11(String)
==> SHARDING_EXECUTOR_1, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no`,t.`create_time` FROM `t_order_202211` t WHERE t.`id` = ? LIMIT 1
==> SHARDING_EXECUTOR_1, name:ds0, Parameters: 11(String)
==> SHARDING_EXECUTOR_2, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no`,t.`create_time` FROM `t_order_202203` t WHERE t.`id` = ? LIMIT 1
==> SHARDING_EXECUTOR_5, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no`,t.`create_time` FROM `t_order_202301` t WHERE t.`id` = ? LIMIT 1
==> SHARDING_EXECUTOR_2, name:ds0, Parameters: 11(String)
==> SHARDING_EXECUTOR_5, name:ds0, Parameters: 11(String)
<== Total: 1
获取范围内的数据
@GetMapping("/range")
public Object first()
List<OrderByMonthEntity> list = easyQuery.queryable(OrderByMonthEntity.class)
.where(o -> o.rangeClosed(OrderByMonthEntity::getCreateTime, LocalDateTime.of(2022, 3, 1, 0, 0), LocalDateTime.of(2022, 9, 1, 0, 0)))
.toList();
return list;
http://localhost:8080/orderMonth/range
["id":"181","uid":"xiaohong","orderNo":181,"createTime":"2022-07-01T00:00:00","id":"182","uid":"xiaolan","orderNo":182,"createTime":"2022-07-02T00:00:00","id":"183","uid":"xiaoming","orderNo":183,"createTime":"2022-07-03T00:00:00",...........,"id":"239","uid":"xiaolan","orderNo":239,"createTime":"2022-08-28T00:00:00","id":"240","uid":"xiaoming","orderNo":240,"createTime":"2022-08-29T00:00:00","id":"241","uid":"xiaohong","orderNo":241,"createTime":"2022-08-30T00:00:00","id":"242","uid":"xiaolan","orderNo":242,"createTime":"2022-08-31T00:00:00"]
//可以精准定位到对应的分片路由上获取数据
==> SHARDING_EXECUTOR_1, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no`,t.`create_time` FROM `t_order_202207` t WHERE t.`create_time` >= ? AND t.`create_time` <= ?
==> SHARDING_EXECUTOR_5, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no`,t.`create_time` FROM `t_order_202209` t WHERE t.`create_time` >= ? AND t.`create_time` <= ?
==> SHARDING_EXECUTOR_2, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no`,t.`create_time` FROM `t_order_202206` t WHERE t.`create_time` >= ? AND t.`create_time` <= ?
==> SHARDING_EXECUTOR_4, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no`,t.`create_time` FROM `t_order_202203` t WHERE t.`create_time` >= ? AND t.`create_time` <= ?
==> SHARDING_EXECUTOR_3, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no`,t.`create_time` FROM `t_order_202205` t WHERE t.`create_time` >= ? AND t.`create_time` <= ?
==> SHARDING_EXECUTOR_4, name:ds0, Parameters: 2022-03-01T00:00(LocalDateTime),2022-09-01T00:00(LocalDateTime)
==> SHARDING_EXECUTOR_3, name:ds0, Parameters: 2022-03-01T00:00(LocalDateTime),2022-09-01T00:00(LocalDateTime)
==> SHARDING_EXECUTOR_2, name:ds0, Parameters: 2022-03-01T00:00(LocalDateTime),2022-09-01T00:00(LocalDateTime)
==> SHARDING_EXECUTOR_5, name:ds0, Parameters: 2022-03-01T00:00(LocalDateTime),2022-09-01T00:00(LocalDateTime)
==> SHARDING_EXECUTOR_1, name:ds0, Parameters: 2022-03-01T00:00(LocalDateTime),2022-09-01T00:00(LocalDateTime)
==> SHARDING_EXECUTOR_4, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no`,t.`create_time` FROM `t_order_202208` t WHERE t.`create_time` >= ? AND t.`create_time` <= ?
==> SHARDING_EXECUTOR_2, name:ds0, Preparing: SELECT t.`id`,t.`uid`,t.`order_no`,t.`create_time` FROM `t_order_202204` t WHERE t.`create_time` >= ? AND t.`create_time` <= ?
==> SHARDING_EXECUTOR_4, name:ds0, Parameters: 2022-03-01T00:00(LocalDateTime),2022-09-01T00:00(LocalDateTime)
==> SHARDING_EXECUTOR_2, name:ds0, Parameters: 2022-03-01T00:00(LocalDateTime),2022-09-01T00:00(LocalDateTime)
<== Total: 185
最后
目前为止你已经看到了easy-query对于分片的便捷性，但是本章只是开胃小菜,相信了解分库分表的小伙伴肯定会说就这？不是和sharding-jdbc一样吗为什么要用你的呢。我想说第一篇只是给大家了解一下如何使用,后续的文章才是分表分库的精髓相信我你一定没看过
.Net下你不得不看的分表分库解决方案-多字段分片
.Net下你不得不看的分表分库解决方案-多字段分片
介绍
本期主角:ShardingCore 一款ef-core下高性能、轻量级针对分表分库读写分离的解决方案，具有零依赖、零学习成本、零业务代码入侵
dotnet下唯一一款全自动分表,多字段分表框架,拥有高性能,零依赖、零学习成本、零业务代码入侵,并且支持读写分离动态分表分库,同一种路由可以完全自定义的新星组件,通过本框架你不但可以学到很多分片的思想和技巧,并且更能学到Expression的奇思妙用
你的star和点赞是我坚持下去的最大动力,一起为.net生态提供更好的解决方案
项目地址
- github地址 https://github.com/xuejmnet/sharding-core
- gitee地址 https://gitee.com/dotnetchina/sharding-core
背景
直接开门见山,你有没有这种情况你需要将一批数据用时间分片来进行存储比如订单表,订单表的分片字段是订单的创建时间,并且id是雪花id，订单编号是带时间信息的编号,因为.net下的所有分片方案几乎都是只支持单分片字段,所以当我们不使用分片字段查询也就是订单创建时间查询的话会带来全表查询,导致性能下降,譬如我想用雪花id或者订单编号进行查询，但是带来的却是内部低效的结果,针对这种情况是否有一个好的解决方案呢,有但是需要侵入业务代码,根据雪花id或者订单编号进行解析出对应的时间然后手动指定分片前提是框架支持手动指定.基于上述原因ShardingCore 带来了全新版本 x.3.2.x+ 支持多字段分片路由,并且拥有很完美的实现,废话不多说我们直接开始吧！！！！！！！！！！！
原理
我们现在假定一个很简单的场景,依然是订单时间按月分片,查询进行如下语句
//这边演示不使用雪花id因为雪花id很难在演示中展示所以使用订单编号进行演示格式:yyyyMMddHHmmss+new Random().Next(0,10000).ToString().PadLeft(4,\'0\')
var dateTime = new DateTime(2021, 11, 1);
var order = await _myDbContext.Set<Order>().Where(o => o.OrderNo== 202112201900001111&&o.CreateTime< dateTime).FirstOrDefaultAsync();
上述语句OrderNo会查询Order_202112这张表，然后时间索引会查询......Order_202108、Order_202109、Order_202110,然后两者取一个交集我们发现其实是没有结果的,这个时候应该是返回默认值null或者直接报错
这就是一个简单的原理
直接开始
接下来我将用订单编号和创建时间来为大演示,数据库采用sqlserver(你也可以换成任意efcore支持的数据库),其中编号格式yyyyMMddHHmmss+new Random().Next(0,10000).ToString().PadLeft(4,\'0\'),创建时间是DateTime格式并且创建时间按月分表,这边不采用雪花id是因为雪花id的实现会根据workid和centerid的不一样而出现不一样的效果,接下来我们通过简单的5步操作实现多字段分片
添加依赖
首先我们添加两个依赖,一个是ShardingCore一个EFCore.SqlServer
//请安装最新版本目前x.3.2.x+,第一个版本号6代表efcore的版本号
Install-Package ShardingCore -Version 6.3.2
Install-Package Microsoft.EntityFrameworkCore.SqlServer -Version 6.0.1
创建一个订单对象
public class Order
public string Id get; set;
public string OrderNo get; set;
public string Name get; set;
public DateTime CreateTime get; set;
创建DbContext
这边就简单的创建了一个dbcontext,并且设置了一下order如何映射到数据库,当然你可以采用attribute的方式而不是一定要fluentapi
/// <summary>
/// 如果需要支持分表必须要实现<see cref="IShardingTableDbContext"/>
/// </summary>
public class DefaultDbContext:AbstractShardingDbContext,IShardingTableDbContext
public DefaultDbContext(DbContextOptions options) : base(options)
protected override void OnModelCreating(ModelBuilder modelBuilder)
base.OnModelCreating(modelBuilder);
modelBuilder.Entity<Order>(o =>
o.HasKey(p => p.Id);
o.Property(p => p.OrderNo).IsRequired().HasMaxLength(128).IsUnicode(false);
o.Property(p => p.Name).IsRequired().HasMaxLength(128).IsUnicode(false);
o.ToTable(nameof(Order));
);
public IRouteTail RouteTail get; set;
创建分片路由
这边我们采用订单创建时间按月分表
public class OrderVirtualRoute : AbstractSimpleShardingMonthKeyDateTimeVirtualTableRoute<Order>
/// <summary>
/// 配置主分表字段是CreateTime,额外分表字段是OrderNo
/// </summary>
/// <param name="builder"></param>
public override void Configure(EntityMetadataTableBuilder<Order> builder)
builder.ShardingProperty(o => o.CreateTime);
builder.ShardingExtraProperty(o => o.OrderNo);
/// <summary>
/// 是否要在程序运行期间自动创建每月的表
/// </summary>
/// <returns></returns>
public override bool AutoCreateTableByTime()
return true;
/// <summary>
/// 分表从何时起创建
/// </summary>
/// <returns></returns>
public override DateTime GetBeginTime()
return new DateTime(2021, 9, 1);
/// <summary>
/// 配置额外分片路由规则
/// </summary>
/// <param name="shardingKey"></param>
/// <param name="shardingOperator"></param>
/// <param name="shardingPropertyName"></param>
/// <returns></returns>
public override Expression<Func<string, bool>> GetExtraRouteFilter(object shardingKey, ShardingOperatorEnum shardingOperator, string shardingPropertyName)
switch (shardingPropertyName)
case nameof(Order.OrderNo): return GetOrderNoRouteFilter(shardingKey, shardingOperator);
default: throw new NotImplementedException(shardingPropertyName);
/// <summary>
/// 订单编号的路由
/// </summary>
/// <param name="shardingKey"></param>
/// <param name="shardingOperator"></param>
/// <returns></returns>
private Expression<Func<string, bool>> GetOrderNoRouteFilter(object shardingKey,
ShardingOperatorEnum shardingOperator)
//将分表字段转成订单编号
var orderNo = shardingKey?.ToString() ?? string.Empty;
//判断订单编号是否是我们符合的格式
if (!CheckOrderNo(orderNo, out var orderTime))
//如果格式不一样就直接返回false那么本次查询因为是and链接的所以本次查询不会经过任何路由,可以有效的防止恶意攻击
return tail => false;
//当前时间的tail
var currentTail = TimeFormatToTail(orderTime);
//因为是按月分表所以获取下个月的时间判断id是否是在临界点创建的
var nextMonthFirstDay = ShardingCoreHelper.GetNextMonthFirstDay(DateTime.Now);
if (orderTime.AddSeconds(10) > nextMonthFirstDay)
var nextTail = TimeFormatToTail(nextMonthFirstDay);
return DoOrderNoFilter(shardingOperator, orderTime, currentTail, nextTail);
//因为是按月分表所以获取这个月月初的时间判断id是否是在临界点创建的
if (orderTime.AddSeconds(-10) < ShardingCoreHelper.GetCurrentMonthFirstDay(DateTime.Now))
//上个月tail
var previewTail = TimeFormatToTail(orderTime.AddSeconds(-10));
return DoOrderNoFilter(shardingOperator, orderTime, previewTail, currentTail);
return DoOrderNoFilter(shardingOperator, orderTime, currentTail, currentTail);
private Expression<Func<string, bool>> DoOrderNoFilter(ShardingOperatorEnum shardingOperator, DateTime shardingKey, string minTail, string maxTail)
switch (shardingOperator)
case ShardingOperatorEnum.GreaterThan:
case ShardingOperatorEnum.GreaterThanOrEqual:
return tail => String.Compare(tail, minTail, StringComparison.Ordinal) >= 0;
case ShardingOperatorEnum.LessThan:
var currentMonth = ShardingCoreHelper.GetCurrentMonthFirstDay(shardingKey);
//处于临界值 o=>o.time < [2021-01-01 00:00:00] 尾巴20210101不应该被返回
if (currentMonth == shardingKey)
return tail => String.Compare(tail, maxTail, StringComparison.Ordinal) < 0;
return tail => String.Compare(tail, maxTail, StringComparison.Ordinal) <= 0;
case ShardingOperatorEnum.LessThanOrEqual:
return tail => String.Compare(tail, maxTail, StringComparison.Ordinal) <= 0;
case ShardingOperatorEnum.Equal:
var isSame = minTail == maxTail;
if (isSame)
return tail => tail == minTail;
else
return tail => tail == minTail || tail == maxTail;
default:
return tail => true;
private bool CheckOrderNo(string orderNo, out DateTime orderTime)
//yyyyMMddHHmmss+new Random().Next(0,10000).ToString().PadLeft(4,\'0\')
if (orderNo.Length == 18)
if (DateTime.TryParseExact(orderNo.Substring(0, 14), "yyyyMMddHHmmss", CultureInfo.InvariantCulture,
DateTimeStyles.None, out var parseDateTime))
orderTime = parseDateTime;
return true;
orderTime = DateTime.MinValue;
return false;
这边我来讲解一下为什么用额外字段分片需要些这么多代码呢,其实是这样的因为你是用订单创建时间CreateTime来进行分片的那么CreateTime和OrderNo的赋值原理上说应该在系统里面是不可能实现同一时间赋值的肯定有先后关系可能是几微妙甚至几飞秒,但是为了消除这种差异这边采用了临界点兼容算法来实现，让我们来看下一下代码
var order=new Order()
//执行这边生成出来的id是2021-11-30 23:59:59.999.999
order.OrderNo=DateTime.Now.ToString("yyyyMMddHHmmss")+"xxx";
//business code //具体执行时间不确定,哪怕没有business code也没有办法保证两者生成的时间一致,当然如果你可以做到一致完全不需要这么复杂的编写
............
//执行这边生成出来的时间是2021-12-01 00:00:00.000.000
order.CreateTime=DateTime.Now;
当然系统里面采用了前后添加10秒是一个比较保守的估算你可以采用前后一秒甚至几百毫秒都是ok的,具体业务具体实现,因为大部分的创建时间可能是由框架在提交后才会生成而不是new Order的时候,当然也不排除这种情况,当然如果你只需要考虑equal一种情况可以只编写equal的判断而不需要全部情况都考虑
ShardingCore启动配置
ILoggerFactory efLogger = LoggerFactory.Create(builder =>
builder.AddFilter((category, level) => category == DbLoggerCategory.Database.Command.Name && level == LogLevel.Information).AddConsole();
);
var builder = WebApplication.CreateBuilder(args);
// Add services to the container.
builder.Services.AddControllers();
builder.Services.AddShardingDbContext<DefaultDbContext>((conStr,builder)=>builder
.UseSqlServer(conStr)
.UseLoggerFactory(efLogger)
)
.Begin(o =>
o.CreateShardingTableOnStart = true;
o.EnsureCreatedWithOutShardingTable = true;
).AddShardingTransaction((connection, builder) =>
builder.UseSqlServer(connection).UseLoggerFactory(efLogger);
).AddDefaultDataSource("ds0","Data Source=localhost;Initial Catalog=ShardingMultiProperties;Integrated Security=True;")//如果你是sqlserve只需要修改这边的链接字符串即可
.AddShardingTableRoute(op =>
op.AddShardingTableRoute<OrderVirtualRoute>();
)
.AddTableEnsureManager(sp=>new SqlServerTableEnsureManager<DefaultDbContext>())//告诉ShardingCore启动时有哪些表
.End();
var app = builder.Build();
// Configure the HTTP request pipeline.
app.Services.GetRequiredService<IShardingBootstrapper>().Start();
app.UseAuthorization();
app.MapControllers();
//额外添加一些种子数据
using (var serviceScope = app.Services.CreateScope())
var defaultDbContext = serviceScope.ServiceProvider.GetService<DefaultDbContext>();
if (!defaultDbContext.Set<Order>().Any())
var orders = new List<Order>(8);
var beginTime = new DateTime(2021, 9, 5);
for (int i = 0; i < 8; i++)
var orderNo = beginTime.ToString("yyyyMMddHHmmss") + i.ToString().PadLeft(4, \'0\');
orders.Add(new Order()
Id = Guid.NewGuid().ToString("n"),
CreateTime = beginTime,
Name = $"Order" + i,
OrderNo = orderNo
);
beginTime = beginTime.AddDays(1);
if (i % 2 == 1)
beginTime = beginTime.AddMonths(1);
defaultDbContext.AddRange(orders);
defaultDbContext.SaveChanges();
app.Run();
整个配置下来其实也就两个地方需要配置还是相对比较简单的,直接启动开始我们的测试模式
测试
默认配置下的测试
public async Task<IActionResult> Test1()
//订单名称全表扫描
Console.WriteLine("--------------Query Name Begin--------------");
var order1 = await _defaultDbContext.Set<Order>().Where(o=>o.Name=="Order3").FirstOrDefaultAsync();
Console.WriteLine("--------------Query Name End--------------");
//订单编号查询 精确定位
Console.WriteLine("--------------Query OrderNo Begin--------------");
var order2 = await _defaultDbContext.Set<Order>().Where(o=>o.OrderNo== "202110080000000003").FirstOrDefaultAsync();
Console.WriteLine("--------------Query OrderNo End--------------");
//创建时间查询 精确定位
Console.WriteLine("--------------Query OrderCreateTime Begin--------------");
var dateTime = new DateTime(2021,10,08);
var order4 = await _defaultDbContext.Set<Order>().Where(o=>o.CreateTime== dateTime).FirstOrDefaultAsync();
Console.WriteLine("--------------Query OrderCreateTime End--------------");
//订单编号in 精确定位
Console.WriteLine("--------------Query OrderNo Contains Begin--------------");
var orderNos = new string[] "202110080000000003", "202111090000000004" ;
var order5 = await _defaultDbContext.Set<Order>().Where(o=> orderNos.Contains(o.OrderNo)).ToListAsync();
Console.WriteLine("--------------Query OrderNo Contains End--------------");
//订单号和创建时间查询 精确定位 无路由结果 抛错或者返回default
Console.WriteLine("--------------Query OrderNo None Begin--------------");
var time = new DateTime(2021,11,1);
var order6 = await _defaultDbContext.Set<Order>().Where(o=> o.OrderNo== "202110080000000003"&&o.CreateTime> time).FirstOrDefaultAsync();
Console.WriteLine("--------------Query OrderNo None End--------------");
//非正确格式订单号 抛错或者返回default防止击穿数据库
Console.WriteLine("--------------Query OrderNo Not Check Begin--------------");
var order3 = await _defaultDbContext.Set<Order>().Where(o => o.OrderNo == "a02110080000000003").FirstOrDefaultAsync();
Console.WriteLine("--------------Query OrderNo Not Check End--------------");
return Ok();
测试结果
测试结果非常完美除了无法匹配路由的时候那么我们该如何设置呢
测试无路由返回默认值
builder.Services.AddShardingDbContext<DefaultDbContext>(...)
.Begin(o =>
....
o.ThrowIfQueryRouteNotMatch = false;//配置默认不抛出异常
)
我们再次来看下测试结果
为何我们测试是不经过数据库直接查询,原因就是在我们做各个属性分片交集的时候返回了空那么框架会选择抛出异常或者返回默认值两种选项,并且我们在编写路由的时候判断格式不正确返回 return tail => false;直接让所有的交集都是空所以不会进行一次无意义的数据库查询
总结
看到这边你应该已经看到了本框架的强大之处,本框架不但可以实现多字段分片还可以实现自定义分片,而不是单单按时间分片这么简单,我完全可以设置订单从2021年后的订单按月分片,2021年前的订单按年分片,对于sharding-core而言这简直轻而易举,但是据我所知.Net下目前除了我没有任何一款框架可以做到真正的全自动分片+多字段分片,所以我们在设计框架分片的时候尽可能的将有用的信息添加到一些无意义的字段上比如Id可以有效的解决很多在大数据下发生的问题,你可以简单理解为我加了一个索引并且附带了额外列,我加了一个id并且带了分表信息在里面,也可以完全设计出一款附带分库的属性到id里面使其可以支持分表分库
最后的最后
demo地址 https://github.com/xuejmnet/MultiShardingProperties
您都看到这边了确定不点个star或者赞吗,一款.Net不得不学的分库分表解决方案,简单理解为sharding-jdbc在.net中的实现并且支持更多特性和更优秀的数据聚合,拥有原生性能的97%,并且无业务侵入性,支持未分片的所有efcore原生查询
以上是关于javaer你还在手写分表分库?来看看这个框架怎么做的 干货满满的主要内容，如果未能解决你的问题，请参考以下文章