运行spark官方的graphx 示例 ComprehensiveExample.scala报错解决

Posted 2020-10-21 丹江湖畔养蜂子的赵大爹

运行spark官方的graphx 示例 ComprehensiveExample.scala报错解决

在Idea中，直接运行ComprehensiveExample.scala，报需要指定master异常。
修改源码：指定master为local模式，

 val spark = SparkSession
      .builder
      .appName(s"${this.getClass.getSimpleName}").master("local[2]")
      .getOrCreate()

继续运行，报如下错误：

Exception in thread "main" java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.mapred.FileInputFormat
    at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:312)
    at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:199)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
    at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
    at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2094)
    at org.apache.spark.rdd.RDD.count(RDD.scala:1158)
    at org.apache.spark.graphx.GraphLoader$.edgeListFile(GraphLoader.scala:94)
    at graphx.ComprehensiveExample$.main(ComprehensiveExample.scala:53)
    at graphx.ComprehensiveExample.main(ComprehensiveExample.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)

com.google.common.base.Stopwatch 在guava jar包中，查了相关资料，说可能是guava包的版本问题，
我的测试程序使用的版本是18.0，而spark官方源码使用的是14.0.1，spark 2.2.1 使用依赖的hadoop 版本是2.6.5。

        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>14.0.1</version>
        </dependency>

修改后重新运行spark任务，问题解决。