Clickhouse 内存达到限制

Posted 2022-12-13 福州司马懿

Memory limit (total) exceeded

localhost :) insert into `test`.`test_memory`  select * mysql('192.168.100.101:3306','test',"test_memory','root','<密码>');                                                  

Received exception from server (version 21.11.3):
Code: 241. DB::Exception: Received from localhost:9000. DB::Exception: Memory limit (total) exceeded: would use 2.46 GiB (attempt to allocate chunk of 6291456 bytes), maximum: 2.46 GiB. (MEMORY_LIMIT_EXCEEDED)

需要在 config.xml 或 users.xml 中增加 max_memory_usage值的大小

Too many partitions for single INSERT block

Received exception from server (version 21.11.4):
Code: 252. DB::Exception: Received from localhost:9000. DB::Exception: Too many partitions for single INSERT block (more than 100). The limit is controlled by 'max_partitions_per_insert_block' setting. Large number of partitions is a common misconception. It will lead to severe negative performance impact, including slow server startup, slow INSERT queries and slow SELECT queries. Recommended total number of partitions for a table is under 1000..10000. Please note, that partitioning is not intended to speed up SELECT queries (ORDER BY key is sufficient to make range queries fast). Partitions are intended for data manipulation (DROP PARTITION, etc).. (TOO_MANY_PARTS)

要增大 max_partitions_per_insert_block 的大小

Memory limit (for query) exceeded

SQL 错误 [1002]: ClickHouse exception, code: 1002, host: 114.115.160.23, port: 8123; Code: 241. DB::Exception: Memory limit (for query) exceeded: would use 9.32 GiB (attempt to allocate chunk of 4390912 bytes), maximum: 9.31 GiB: While executing MergeSortingTransform. (MEMORY_LIMIT_EXCEEDED) (version 21.11.4.14 (official build))

如果是group by内存不够，推荐配置上max_bytes_before_external_group_by参数，当使用内存到达该阈值，进行磁盘group by

如果是order by内存不够，推荐配置上max_bytes_before_external_sort参数，当使用内存到达该阈值，进行磁盘order by

如果是count distinct内存不够，推荐使用一些预估函数(如果业务场景允许)，这样不仅可以减少内存的使用同时还会提示查询速度

注意点

system库里面的表都是普通表，直接set只会在当前server生效

、常见参数配置推荐

1）max_concurrent_queries

最大并发处理的请求数(包含select,insert等)，默认值100，推荐150(不够再加)，在我们的集群中出现过”max concurrent queries”的问题。

2）max_bytes_before_external_sort

当order by已使用max_bytes_before_external_sort内存就进行溢写磁盘(基于磁盘排序)，如果不设置该值，那么当内存不够时直接抛错，设置了该值order by可以正常完成，但是速度相对存内存来说肯定要慢点(实测慢的非常多，无法接受)。

3）background_pool_size

后台线程池的大小，merge线程就是在该线程池中执行，当然该线程池不仅仅是给merge线程用的，默认值16，推荐32提升merge的速度(CPU允许的前提下)。

4）max_memory_usage

单个SQL在单台机器最大内存使用量，该值可以设置的比较大，这样可以提升集群查询的上限。

5）max_memory_usage_for_all_queries

单机最大的内存使用量可以设置略小于机器的物理内存(留一点内操作系统)。

6）max_bytes_before_external_group_by

在进行group by的时候，内存使用量已经达到了max_bytes_before_external_group_by的时候就进行写磁盘(基于磁盘的group by相对于基于磁盘的order by性能损耗要好很多的)，一般max_bytes_before_external_group_by设置为max_memory_usage / 2，原因是在clickhouse中聚合分两个阶段：

查询并且建立中间数据；

合并中间数据 写磁盘在第一个阶段，如果无须写磁盘，clickhouse在第一个和第二个阶段需要使用相同的内存。

这些内存参数强烈推荐配置上，增强集群的稳定性避免在使用过程中出现莫名其妙的异常。

将没有分区的表数据插入到含有分区键的新表报错

SQL 错误 [1002]: ClickHouse exception, code: 1002, host: 114.115.160.23, port: 8123; Code: 252. DB::Exception: Too many partitions for single INSERT block (more than 100). The limit is controlled by 'max_partitions_per_insert_block' setting. Large number of partitions is a common misconception. It will lead to severe negative performance impact, including slow server startup, slow INSERT queries and slow SELECT queries. Recommended total number of partitions for a table is under 1000..10000. Please note, that partitioning is not intended to speed up SELECT queries (ORDER BY key is sufficient to make range queries fast). Partitions are intended for data manipulation (DROP PARTITION, etc). (TOO_MANY_PARTS) (version 21.11.4.14 (official build))

SELECT * FROM system.settings WHERE name = 'max_partitions_per_insert_block' 默认值为100，即单次插入涉及的最大分区数为100。

将其改为0即可（修改users.xml，不用重启clickhouse会自动检测配置文件的修改并刷新到系统表中）

查询超时

SQL 错误 [159]: ClickHouse exception, code: 159, host: 114.115.160.23, port: 8123; Read timed out

这个可以在JDBC连接的URL里配置，驱动名为 socket_timeout，单位毫秒，默认30秒。当超过该时间客户端还未收到SQL服务器响应时，会自动产生一个 Read timed out，错误。但此时SQL服务器依旧在继续执行，而不会因此中断！！！