无法使用 Spark cassandra 连接器 1.5.0 连接 Cassandra 3.0

Posted 2023-04-18

【中文标题】无法使用 Spark cassandra 连接器 1.5.0 连接 Cassandra 3.0

【英文标题】：Unable to connect Cassandra 3.0 using Spark cassandra connector 1.5.0

【发布时间】：2016-04-27 13:28:52

【问题描述】：

问题 - 无法使用 Spark Cassandra 连接器 1.5.0 连接 Cassandra 3.0

背景 - 我尝试使用 Spark 1.5.0 连接 Cassandra 3.0 通过使用提供的 Spark cassandra 连接器 1.5.0 但我遇到错误 -

根据 DataStax Spark Cassandra 连接器document，它表示 Spark 连接器 1.5 可用于从 Spark 1.5.0/1.6.0 到 Cassandra 3.0。

您能否建议我在这里缺少任何步骤？

尝试过的方法

我试图从 Spark 流和核心 jar 中排除 Guava 依赖项

在“pom.xml”中添加了单独的Guava依赖

提前致谢。

---

16/04/26 09:45:07 WARN TaskSetManager: Lost task 4.0 in stage 1.0 (TID 16, ip-172-31-23-23.ec2.internal): java.lang.ExceptionInInitializerError
    at  com.datastax.spark.connector.cql.DefaultConnectionFactory$.clusterBuilder(CassandraConnectionFactory.scala:35)
    at com.datastax.spark.connector.cql.DefaultConnectionFactory$.createCluster(CassandraConnectionFactory.scala:87)
    at 
---
--
Caused by: java.lang.IllegalStateException: Detected Guava issue #1635 which indicates that a version of Guava less than 16.01 is in use.  This introduces codec resolution issues and potentially other incompatibility issues in the driver.  Please upgrade to Guava 16.01 or later.
    at com.datastax.driver.core.SanityChecks.checkGuava(SanityChecks.java:62)
    at com.datastax.driver.core.SanityChecks.check(SanityChecks.java:36)
    at com.datastax.driver.core.Cluster.<clinit>(Cluster.java:67)
    ... 23 more16/04/26 09:45:07 WARN TaskSetManager: Lost task 4.0 in stage 1.0 (TID 16, ip-172-31-23-23.ec2.internal): java.lang.ExceptionInInitializerError
    at com.datastax.spark.connector.cql.DefaultConnectionFactory$.clusterBuilder(CassandraConnectionFactory.scala:35)
    at com.datastax.spark.connector.cql.DefaultConnectionFactory$.createCluster(CassandraConnectionFactory.scala:87)
    at com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$createSession(CassandraConnector.scala:153)
    at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:148)
    at com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(CassandraConnector.scala:148)
    at com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefCountedCache.scala:31)

---

【参考方案1】：

连接器随附的 Guava 版本与 spark 随附的版本之间存在已知冲突。您是如何尝试遮蔽 Guava 库的？

尝试将其添加到您的 build.sbt 文件中：

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.google.**" -> "shadeio.@1").inAll
)