ClickHouse版本升级

Posted

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了ClickHouse版本升级相关的知识,希望对你有一定的参考价值。

参考技术A 以我们项目为例,整理下ClickHouse升级的相关注意事项,由于鄙人学识浅薄,提供的方式方法仅供大家参考,引起的任何问题,本人不承担任何责任。安全生产第一条:记得备份、记得备份、记得备份,重要的事情说三遍。

centos7.2

我们ClickHouse是通过RPM方式来安装的,包括如下三个文件:

我们目前没做分片,只有两个节点。可以理解为:1分片2副本。

1、备份配置文件

路径为:/etc/clickhouse-server,把这个文件夹下面都备份吧,正常安装新版本,clickhouse会自动将之前的配置文件config.xml备份,命名为:config.xml.rpmsave。不过小心使得万年船。

2、备份数据文件

数据文件的路径是在config.xml中进行配置的,我们配置指向一块单独的盘。

总而言之,言而总之,把整个数据文件备份吧。

1、查看目前安装版本

yum list installed | grep clickhouse

2、删除软件版本

yum remove -y clickhouse-common-static

yum remove -y clickhouse-server-common

rpm -ivh *.rpm

安装成功后,会在/etc/clickhouse-server下生产相应的配置文件。

按照旧的config.xml配置信息,更改config.xml。也可先直接覆盖,如果后续不成功,还是一条一条来修改,最好比对下。防止有小版本差异。

sudo systemctl start clickhouse-server

Apache Flink写入Clickhouse报错 code: 1002, ip:8123 failed to respond

目录

版本

ClickHouse server version 21.12.2
驱动版本:0.2.4

一、问题

1 问题描述

flink任务读取kafka消息,处理,JDBC方式写入到clickhouse。开始运行时可以写入个几万条,过30分钟左右报错。

Caused by: ru.yandex.clickhouse.except.ClickHouseUnknownException: ClickHouse exception, code: 1002, host: 192.168.66.66, port: 8123; 192.168.66.66:8123 failed to respond

2 日志

Cause: ru.yandex.clickhouse.except.ClickHouseUnknownException: ClickHouse exception, code: 1002, host: 192.168.66.66, port: 8123; 192.168.66.66:8123 failed to respond
	at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:153) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:145) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:76) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:87) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:145) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:86) ~[sale-1.0-SNAPSHOT.jar:?]
	at com.sun.proxy.$Proxy34.insert(Unknown Source) ~[?:?]
	at com.tanjiu.realtime.common.func.ClickhouseSink.invoke(ClickhouseSink.java:52) ~[sale-1.0-SNAPSHOT.jar:?]
	at com.tanjiu.realtime.common.func.ClickhouseSink.invoke(ClickhouseSink.java:18) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:56) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:53) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.api.operators.async.queue.StreamRecordQueueEntry.emitResult(StreamRecordQueueEntry.java:65) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.api.operators.async.queue.UnorderedStreamElementQueue$Segment.emitCompleted(UnorderedStreamElementQueue.java:263) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.api.operators.async.queue.UnorderedStreamElementQueue.emitCompletedElement(UnorderedStreamElementQueue.java:150) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.api.operators.async.AsyncWaitOperator.outputCompletedElement(AsyncWaitOperator.java:280) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.api.operators.async.AsyncWaitOperator.access$100(AsyncWaitOperator.java:78) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.api.operators.async.AsyncWaitOperator$ResultHandler.processResults(AsyncWaitOperator.java:352) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.api.operators.async.AsyncWaitOperator$ResultHandler.lambda$processInMailbox$0(AsyncWaitOperator.java:337) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:78) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:283) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:184) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:575) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:539) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547) ~[sale-1.0-SNAPSHOT.jar:?]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
Caused by: ru.yandex.clickhouse.except.ClickHouseUnknownException: ClickHouse exception, code: 1002, host: 192.168.66.66, port: 8123; 192.168.66.66:8123 failed to respond
	at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.getException(ClickHouseExceptionSpecifier.java:91) ~[sale-1.0-SNAPSHOT.jar:?]
	at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:55) ~[sale-1.0-SNAPSHOT.jar:?]
	at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:24) ~[sale-1.0-SNAPSHOT.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.getInputStream(ClickHouseStatementImpl.java:633) ~[sale-1.0-SNAPSHOT.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:117) ~[sale-1.0-SNAPSHOT.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:100) ~[sale-1.0-SNAPSHOT.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:95) ~[sale-1.0-SNAPSHOT.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:90) ~[sale-1.0-SNAPSHOT.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.execute(ClickHouseStatementImpl.java:226) ~[sale-1.0-SNAPSHOT.jar:?]
	at ru.yandex.clickhouse.ClickHousePreparedStatementImpl.execute(ClickHousePreparedStatementImpl.java:105) ~[sale-1.0-SNAPSHOT.jar:?]
	at com.alibaba.druid.pool.DruidPooledPreparedStatement.execute(DruidPooledPreparedStatement.java:497) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:64) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:63) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:325) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:89) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:151) ~[sale-1.0-SNAPSHOT.jar:?]
	... 36 more
Caused by: org.apache.http.NoHttpResponseException: 192.168.66.66:8123 failed to respond
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) ~[sale-1.0-SNAPSHOT.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.getInputStream(ClickHouseStatementImpl.java:614) ~[sale-1.0-SNAPSHOT.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:117) ~[sale-1.0-SNAPSHOT.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:100) ~[sale-1.0-SNAPSHOT.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:95) ~[sale-1.0-SNAPSHOT.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:90) ~[sale-1.0-SNAPSHOT.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.execute(ClickHouseStatementImpl.java:226) ~[sale-1.0-SNAPSHOT.jar:?]
	at ru.yandex.clickhouse.ClickHousePreparedStatementImpl.execute(ClickHousePreparedStatementImpl.java:105) ~[sale-1.0-SNAPSHOT.jar:?]
	at com.alibaba.druid.pool.DruidPooledPreparedStatement.execute(DruidPooledPreparedStatement.java:497) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:64) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:63) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:325) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:89) ~[sale-1.0-SNAPSHOT.jar:?]
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:151) ~[sale-1.0-SNAPSHOT.jar:?]
	... 36 more

二、解决

搜索发现有几个相关的已经是closed状态的bug,那应该是可以通过改配置或包的方式解决。

解决过程

1 升级驱动jar包版本(无效)

<!-- <dependency>
            <groupId>ru.yandex.clickhouse</groupId>
            <artifactId>clickhouse-jdbc</artifactId>
            <version>0.2.4</version>
</dependency>-->

<dependency>
            <groupId>ru.yandex.clickhouse</groupId>
            <artifactId>clickhouse-jdbc</artifactId>
            <version>0.3.1-patch</version>
</dependency>

改完后报错仍旧出现,看来和驱动包相关性不大

2 修改连接参数(无效)

这里我用druid连的clickhouse,尝试调整一下参数

	DruidDataSource druidDataSource = new DruidDataSource();
        
    //Setting up driver
    druidDataSource.setDriverClassName("ru.yandex.clickhouse.ClickHouseDriver");
    druidDataSource.setUrl(url);
    druidDataSource.setUsername(user);
    druidDataSource.setPassword(password);


    // 解决 failed to respond 问题
    druidDataSource.setInitialSize(1);
    druidDataSource.setMaxActive(2);
    druidDataSource.setMaxWait(6000);
    druidDataSource.setMinIdle(12);

    druidDataSource.setTimeBetweenEvictionRunsMillis(60000);
    druidDataSource.setValidationQuery("select 1");

    druidDataSource.setMinEvictableIdleTimeMillis(300000);
    druidDataSource.setMaxEvictableIdleTimeMillis(900000);

    druidDataSource.setTestWhileIdle(true);
    druidDataSource.setTestOnBorrow(false);
    druidDataSource.setTestOnReturn(false);

    druidDataSource.setKeepAlive(true);
    druidDataSource.setPoolPreparedStatements(true);
    druidDataSource.setMaxOpenPreparedStatements(20);

    druidDataSource.setAsyncInit(true);

依旧报错

3 修改服务端参数(有效)

# pwd
/etc/clickhouse-server
# ls
config.d  config.xml  metrika.xml  users.d  users.xml
# vim config.xml

查看连接超时时间的配置,发现配置是3

查看 官方文档 发现官方默认参数是10,这里修改为30

修改完目前已经正常运行了3个小时,未出现报错

4 修改获取连接方式(todo)

当前获取连接的流程

  1. 当前获取连接的时候是先去缓存里找;
  2. 如果缓存没有就新建一个连接加入到缓存中返回连接;
  3. 如果缓存存在连接,就直接从缓存中获取,返回;

找资料的时候偶然看到这位大佬的话

突然想到自己获取连接的方式,可能需要处理一下。或许可以尝试每次都获取新连接,用完后则立刻关闭,不再缓存。

参考资料

官方文档
Clickhouse错误问题汇总
ClickHouse到底该写本地表还是分布式表

以上是关于ClickHouse版本升级的主要内容,如果未能解决你的问题,请参考以下文章

Apache Flink写入Clickhouse报错 code: 1002, ip:8123 failed to respond

如何升级OpenSSl版本

Ubuntu升级版本

KindEdito从3.51升级到4.0版本流程怎么升级?

Mac升级node版本

跨版本顺利升级Ubuntu