waterdrop1.5.1

Posted 2021-11-22 码紮

waterdrop1.5.1将hive中orc格式的文件导入clickhouse中

diagnostics: User class threw exception: java.lang.ExceptionInInitializerError
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:84)
    at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:48)
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parseTableIdentifier(ParseDriver.scala:49)
    at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:409)
    at com.xinyan.blockchain.ods.DataLoadETH$.loadEth(DataLoadETH.scala:82)
    at com.xinyan.blockchain.ods.DataLoadETH$.main(DataLoadETH.scala:45)
    at com.xinyan.blockchain.ods.DataLoadETH.main(DataLoadETH.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:665)
Caused by: java.lang.UnsupportedOperationException: java.io.InvalidClassException: org.antlr.v4.runtime.atn.ATN; Could not deserialize ATN with UUID 59627784-3be5-417a-b9eb-8131a7286089 (expected aadb8d7e-aeef-4415-ad2b-8204d6cf042e or a legacy UUID).
    at org.antlr.v4.runtime.atn.ATNDeserializer.deserialize(ATNDeserializer.java:154)
    at org.apache.spark.sql.catalyst.parser.SqlBaseLexer.<clinit>(SqlBaseLexer.java:1175)

解决方案：

在batch.conf中 配置 spark.executor.extraClassPath = "/*/parcels/CDH/lib/spark/hive/hive-exec-2.1.1-cdh6.3.2.jar"即可