WaterDrop 系列报错

Posted wuning

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了WaterDrop 系列报错相关的知识,希望对你有一定的参考价值。

1.使用WaterDrop 从kafka中消费数据,写入到ClickHouse

1.1 环境

SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179
clickhouse-1.1.54236-4.el7.x86_64
waterdrop-1.4.2

waterdrop 配置文件

spark {
  spark.streaming.batchDuration = 5
  spark.app.name = "Waterdrop"
  spark.executor.instances = 2
  spark.executor.cores = 1
  spark.executor.memory = "1g"
}

input {
  kafkaStream {
    topics = "waterdrop"
    consumer.bootstrap.servers = "cdh01:9092,cdh02:9092,cdh03:9092"
    consumer.group.id = "waterdrop_group"
    }
}

filter {
  split {
    fields = ["FlightDate", "Year"]
    delimiter = ","
  }
}

output {
  clickhouse {
    host = "cdh03:8123"
    clickhouse.socket_timeout = 50000
    database = "ontime"
    table = "ontime_test"
    fields = ["FlightDate", "Year"]
    username = "default"
    password = "default"
    bulk_size = 20000
  }
}

启动waterdrop 报错:

Exception in thread "main" java.lang.NoSuchFieldError: INSTANCE
 at org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:1023)
 at ru.yandex.clickhouse.util.ClickHouseHttpClientBuilder.buildClient(ClickHouseHttpClientBuilder.java:59)
 at ru.yandex.clickhouse.

原因:

  1. clickhouse 依赖于 clickhouse-jdbc2.0,clickhouse-jdbc2.0 又依赖于 httpclient 4.5.2
    参考地址:https://www.mvnjar.com/ru.yandex.clickhouse/clickhouse-jdbc/0.2/detail.html

  2. SPARK2-2.3.0.cloudera4-1.cdh5.13.3 没有 httpclient 4.5.2,只有 httpcore
    技术图片

解决方法:

  1. 下载httpclient 4.5.2

  2. httpclient 放入 Spark 的安装目录;如:/opt/cloudera/parcels/SPARK2-2.3.0.cloudera4-1.cdh5.13.3.p0.611179/lib/spark2/jars

以上是关于WaterDrop 系列报错的主要内容,如果未能解决你的问题,请参考以下文章

WaterDrop

waterdrop(1.5.1版本)增加bitmap类型导数的遇到的问题

clickhouse使用waterdrop将Hive中的数据导入ClickHouse

错误记录Flutter 混合开发获取 BinaryMessenger 报错 ( FlutterActivityAndFragmentDelegate.getFlutterEngine() )(代码片段

springcloud报错-------关于 hystrix 的异常 FallbackDefinitionException:fallback method wasn't found(代码片段

错误记录Android Studio 编译报错 ( Could not determine java version from ‘11.0.8‘. | Android Studio 降级 )(代码片段