spark-submit 中的 scala.ScalaReflectionException

Posted 2023-03-24

【中文标题】spark-submit 中的 scala.ScalaReflectionException

【英文标题】：scala.ScalaReflectionException in spark-submit from command-line

【发布时间】：2017-05-06 02:48:08

【问题描述】：

无法在本地模式下进行 spark-submit。异常指出找不到该类。

请在下面找到命令和堆栈跟踪。

spark-submit --master local[4] --driver-class-path /home/rahul.midha/test/src/main/resources/conf/ --class com.test.main.TestMain test.jar [21.12.2016] [00:47:58] 错误 [主要] 异常： scala.ScalaReflectionException：JavaMirror 中的类 com.test.do.TestDetails，类型为 sun.misc.Launcher$AppClassLoader 的 sun.misc.Launcher$AppClassLoader@3e2f1b1a，类路径 [file:/home/rahul.midha/test/src/ main/resources/conf/，文件：/usr/local/spark/conf/，文件：/usr/local/spark/jars/joda-time-2.9.3.jar，文件：/usr/local/spark/jars /jcl-over-slf4j-1.7.16.jar，文件：/usr/local/spark/jars/slf4j-api-1.7.16.jar，文件：/usr/local/spark/jars/log4j-1.2.17 .jar，文件：/usr/local/spark/jars/javax.inject-1.jar，文件：/usr/local/spark/jars/metrics-graphite-3.1.2.jar，文件：/usr/local/ spark/jars/commons-beanutils-core-1.8.0.jar，文件：/usr/local/spark/jars/javax.annotation-api-1.2.jar，文件：/usr/local/spark/jars/slf4j- log4j12-1.7.16.jar，文件：/usr/local/spark/jars/commons-io-2.4.jar，文件：/usr/local/spark/jars/mail-1.4.7.jar，文件：/usr /local/spark/jars/derby-10.12.1.1.jar，文件：/usr/local/spark/jars/hadoop-common-2.6.4.jar，文件：/usr/local/spark/jars/commons-beanutils -1.7.0.jar，文件：/usr/local/spark/jars/parqu et-common-1.7.0.jar，文件：/usr/local/spark/jars/spire-macros_2.11-0.7.4.jar，文件：/usr/local/spark/jars/xbean-asm5-shaded- 4.4.jar，文件：/usr/local/spark/jars/spark-sql_2.11-2.0.1.jar，文件：/usr/local/spark/jars/commons-lang3-3.3.2.jar，文件： /usr/local/spark/jars/api-asn1-api-1.0.0-M20.jar，文件：/usr/local/spark/jars/ivy-2.4.0.jar，文件：/usr/local/spark /jars/commons-lang-2.6.jar，文件：/usr/local/spark/jars/pyrolite-4.9.jar，文件：/usr/local/spark/jars/super-csv-2.2.0.jar，文件：/usr/local/spark/jars/arpack_combined_all-0.1.jar，文件：/usr/local/spark/jars/datanucleus-core-3.2.10.jar，文件：/usr/local/spark/jars/parquet- hadoop-bundle-1.6.0.jar，文件：/usr/local/spark/jars/hadoop-yarn-server-web-proxy-2.6.4.jar，文件：/usr/local/spark/jars/pmml- model-1.2.15.jar，文件：/usr/local/spark/jars/scala-xml_2.11-1.0.2.jar，文件：/usr/local/spark/jars/curator-client-2.6.0。 jar，文件：/usr/local/spark/jars/spark-tags_2.11-2.0.1.jar，文件：/usr/local/spark/jars/hive-exec-1.2.1.spark2.jar，文件： /usr/local/spark/jars/avro-mapred-1 .7.7-hadoop2.jar，文件：/usr/local/spark/jars/avro-ipc-1.7.7.jar，文件：/usr/local/spark/jars/activation-1.1.1.jar，文件：/ usr/local/spark/jars/hadoop-mapreduce-client-app-2.6.4.jar，文件：/usr/local/spark/jars/api-util-1.0.0-M20.jar，文件：/usr/ local/spark/jars/jta-1.1.jar，文件：/usr/local/spark/jars/jetty-util-6.1.26.jar，文件：/usr/local/spark/jars/metrics-core-3.1。 2.jar，文件：/usr/local/spark/jars/hadoop-hdfs-2.6.4.jar，文件：/usr/local/spark/jars/spark-repl_2.11-2.0.1.jar，文件： /usr/local/spark/jars/javolution-5.5.1.jar，文件：/usr/local/spark/jars/json-20090211.jar，文件：/usr/local/spark/jars/datanucleus-api-jdo -3.2.6.jar，文件：/usr/local/spark/jars/spire_2.11-0.7.4.jar，文件：/usr/local/spark/jars/guice-servlet-3.0.jar，文件：/ usr/local/spark/jars/chill_2.11-0.8.0.jar，文件：/usr/local/spark/jars/avro-1.7.7.jar，文件：/usr/local/spark/jars/scala- parser-combinators_2.11-1.0.4.jar，文件：/usr/local/spark/jars/jersey-server-2.22.2.jar，文件：/usr/local/spark/jars/compress-lzf-1.0。 3.jar，文件：/usr/local/spark/jars/s triingtemplate-3.2.1.jar，文件：/usr/local/spark/jars/json4s-ast_2.11-3.2.11.jar，文件：/usr/local/spark/jars/paranamer-2.3.jar，文件： /usr/local/spark/jars/calcite-avatica-1.2.0-incubating.jar，文件：/usr/local/spark/jars/osgi-resource-locator-1.0.1.jar，文件：/usr/local /spark/jars/calcite-core-1.2.0-incubating.jar，文件：/usr/local/spark/jars/hadoop-auth-2.6.4.jar，文件：/usr/local/spark/jars/breeze_2 .11-0.11.2.jar，文件：/usr/local/spark/jars/spark-unsafe_2.11-2.0.1.jar，文件：/usr/local/spark/jars/spark-streaming_2.11-2.0 .1.jar，文件：/usr/local/spark/jars/spark-core_2.11-2.0.1.jar，文件：/usr/local/spark/jars/xercesImpl-2.9.1.jar，文件：/ usr/local/spark/jars/hadoop-mapreduce-client-core-2.6.4.jar，文件：/usr/local/spark/jars/java-xmlbuilder-1.0.jar，文件：/usr/local/spark/ jars/objenesis-2.1.jar，文件：/usr/local/spark/jars/hk2-locator-2.4.0-b34.jar，文件：/usr/local/spark/jars/hadoop-mapreduce-client-shuffle- 2.6.4.jar，文件：/usr/local/spark/jars/xmlenc-0.52.jar，文件：/usr/local/spark/jars/spark-yarn_2.11-2.0.1.jar，文件：/usr/local/spark/jars/parquet-encoding-1.7.0.jar，文件：/usr/local/spark/jars/json4s-core_2.11-3.2.11.jar，文件：/usr/local /spark/jars/aopalliance-repackaged-2.4.0-b34.jar，文件：/usr/local/spark/jars/snappy-0.2.jar，文件：/usr/local/spark/jars/antlr-2.7.7 .jar，文件：/usr/local/spark/jars/jackson-core-asl-1.9.13.jar，文件：/usr/local/spark/jars/hive-cli-1.2.1.spark2.jar，文件：/usr/local/spark/jars/hadoop-yarn-client-2.6.4.jar，文件：/usr/local/spark/jars/jersey-container-servlet-core-2.22.2.jar，文件：/ usr/local/spark/jars/scala-library-2.11.8.jar，文件：/usr/local/spark/jars/guice-3.0.jar，文件：/usr/local/spark/jars/spark-sketch_2。 11-2.0.1.jar，文件：/usr/local/spark/jars/chill-java-0.8.0.jar，文件：/usr/local/spark/jars/hadoop-yarn-server-common-2.6。 4.jar，文件：/usr/local/spark/jars/eigenbase-properties-1.1.5.jar，文件：/usr/local/spark/jars/parquet-hadoop-1.7.0.jar，文件：/usr /local/spark/jars/stax-api-1.0.1.jar，文件：/usr/local/spark/jars/netty-3.8.0.Final.jar，文件：/usr/local/spark/jars/jackson -模块缩放a_2.11-2.6.5.jar，文件：/usr/local/spark/jars/mesos-0.21.1-shaded-protobuf.jar，文件：/usr/local/spark/jars/httpclient-4.5.2。 jar，文件：/usr/local/spark/jars/hadoop-yarn-api-2.6.4.jar，文件：/usr/local/spark/jars/curator-framework-2.6.0.jar，文件：/usr /local/spark/jars/commons-net-2.2.jar，文件：/usr/local/spark/jars/parquet-jackson-1.7.0.jar，文件：/usr/local/spark/jars/htrace-core -3.0.4.jar，文件：/usr/local/spark/jars/jpam-1.1.jar，文件：/usr/local/spark/jars/jersey-media-jaxb-2.22.2.jar，文件：/ usr/local/spark/jars/spark-network-common_2.11-2.0.1.jar，文件：/usr/local/spark/jars/minlog-1.3.0.jar，文件：/usr/local/spark/ jars/jdo-api-3.0.1.jar，文件：/usr/local/spark/jars/snappy-java-1.1.2.6.jar，文件：/usr/local/spark/jars/spark-mllib_2.11- 2.0.1.jar，文件：/usr/local/spark/jars/javax.servlet-api-3.1.0.jar，文件：/usr/local/spark/jars/scala-reflect-2.11.8.jar，文件：/usr/local/spark/jars/jackson-jaxrs-1.9.13.jar，文件：/usr/local/spark/jars/hadoop-yarn-common-2.6.4.jar，文件：/usr/local /spark/jars/jets3t-0.9.3.jar，文件：/ usr/local/spark/jars/RoaringBitmap-0.5.11.jar，文件：/usr/local/spark/jars/jodd-core-3.5.2.jar，文件：/usr/local/spark/jars/apacheds- i18n-2.0.0-M15.jar，文件：/usr/local/spark/jars/metrics-jvm-3.1.2.jar，文件：/usr/local/spark/jars/ST4-4.0.4.jar，文件：/usr/local/spark/jars/hive-metastore-1.2.1.spark2.jar，文件：/usr/local/spark/jars/xz-1.0.jar，文件：/usr/local/spark/jars /scalap-2.11.8.jar，文件：/usr/local/spark/jars/guava-14.0.1.jar，文件：/usr/local/spark/jars/commons-logging-1.1.3.jar，文件：/usr/local/spark/jars/jersey-client-2.22.2.jar，文件：/usr/local/spark/jars/jackson-annotations-2.6.5.jar，文件：/usr/local/spark/ jars/spark-launcher_2.11-2.0.1.jar，文件：/usr/local/spark/jars/antlr4-runtime-4.5.3.jar，文件：/usr/local/spark/jars/datanucleus-rdbms- 3.2.9.jar，文件：/usr/local/spark/jars/hive-jdbc-1.2.1.spark2.jar，文件：/usr/local/spark/jars/py4j-0.10.3.jar，文件： /usr/local/spark/jars/spark-graphx_2.11-2.0.1.jar，文件：/usr/local/spark/jars/spark-hive_2.11-2.0.1.jar，文件：/usr/local /spark/jars/hk2-utils-2.4.0-b3 4.jar，文件：/usr/local/spark/jars/curator-recipes-2.6.0.jar，文件：/usr/local/spark/jars/core-1.1.2.jar，文件：/usr/local /spark/jars/parquet-generator-1.7.0.jar，文件：/usr/local/spark/jars/stream-2.7.0.jar，文件：/usr/local/spark/jars/mx4j-3.0.2 .jar，文件：/usr/local/spark/jars/netty-all-4.0.29.Final.jar，文件：/usr/local/spark/jars/validation-api-1.1.0.Final.jar，文件：/usr/local/spark/jars/janino-2.7.8.jar，文件：/usr/local/spark/jars/jersey-container-servlet-2.22.2.jar，文件：/usr/local/spark/ jars/univocity-parsers-2.1.1.jar，文件：/usr/local/spark/jars/gson-2.2.4.jar，文件：/usr/local/spark/jars/scala-compiler-2.11.8。 jar，文件：/usr/local/spark/jars/jackson-databind-2.6.5.jar，文件：/usr/local/spark/jars/jackson-module-paranamer-2.6.5.jar，文件：/usr /local/spark/jars/calcite-linq4j-1.2.0-incubating.jar，文件：/usr/local/spark/jars/json4s-jackson_2.11-3.2.11.jar，文件：/usr/local/spark /jars/hk2-api-2.4.0-b34.jar，文件：/usr/local/spark/jars/jsr305-1.3.9.jar，文件：/usr/local/spark/jars/jackson-mapper-asl -1.9.13.jar,fi文件：/usr/local/spark/jars/commons-codec-1.10.jar，文件：/usr/local/spark/jars/spark-network-shuffle_2.11-2.0.1.jar，文件：/usr/local /spark/jars/commons-collections-3.2.2.jar，文件：/usr/local/spark/jars/kryo-shaded-3.0.3.jar，文件：/usr/local/spark/jars/spark-hive -thriftserver_2.11-2.0.1.jar，文件：/usr/local/spark/jars/libfb303-0.9.2.jar，文件：/usr/local/spark/jars/hadoop-client-2.6.4.jar ,文件:/usr/local/spark/jars/antlr-runtime-3.4.jar,文件:/usr/local/spark/jars/httpcore-4.4.4.jar,文件:/usr/local/spark/jars/ parquet-column-1.7.0.jar，文件：/usr/local/spark/jars/commons-httpclient-3.1.jar，文件：/usr/local/spark/jars/hadoop-mapreduce-client-common-2.6。 4.jar，文件：/usr/local/spark/jars/bonecp-0.8.0.RELEASE.jar，文件：/usr/local/spark/jars/hive-beeline-1.2.1.spark2.jar，文件： /usr/local/spark/jars/zookeeper-3.4.6.jar，文件：/usr/local/spark/jars/jetty-6.1.26.jar，文件：/usr/local/spark/jars/pmml-schema -1.2.15.jar，文件：/usr/local/spark/jars/breeze-macros_2.11-0.11.2.jar，文件：/usr/local/spark/jars/jersey-guava-2.2 2.2.jar，文件：/usr/local/spark/jars/commons-configuration-1.6.jar，文件：/usr/local/spark/jars/javax.ws.rs-api-2.0.1.jar，文件： /usr/local/spark/jars/leveldbjni-all-1.8.jar，文件：/usr/local/spark/jars/jackson-xc-1.9.13.jar，文件：/usr/local/spark/jars/parquet -format-2.3.0-incubating.jar，文件：/usr/local/spark/jars/commons-compress-1.4.1.jar，文件：/usr/local/spark/jars/hadoop-annotations-2.6.4 .jar，文件：/usr/local/spark/jars/commons-dbcp-1.4.jar，文件：/usr/local/spark/jars/stax-api-1.0-2.jar，文件：/usr/local/ spark/jars/lz4-1.3.0.jar，文件：/usr/local/spark/jars/JavaEWAH-0.3.2.jar，文件：/usr/local/spark/jars/commons-cli-1.2.jar，文件：/usr/local/spark/jars/opencsv-2.3.jar，文件：/usr/local/spark/jars/aopalliance-1.0.jar，文件：/usr/local/spark/jars/javassist-3.18.1 -GA.jar，文件：/usr/local/spark/jars/apache-log4j-extras-1.2.17.jar，文件：/usr/local/spark/jars/javax.inject-2.4.0-b34.jar ,文件:/usr/local/spark/jars/apacheds-kerberos-codec-2.0.0-M15.jar,文件:/usr/local/spark/jars/oro-2.0.8.jar,文件:/usr/本地/火花/j ars/jline-2.12.1.jar，文件：/usr/local/spark/jars/commons-compiler-2.7.6.jar，文件：/usr/local/spark/jars/jersey-common-2.22.2。 jar，文件：/usr/local/spark/jars/metrics-json-3.1.2.jar，文件：/usr/local/spark/jars/commons-pool-1.5.4.jar，文件：/usr/local /spark/jars/jtransforms-2.4.0.jar，文件：/usr/local/spark/jars/jaxb-api-2.2.2.jar，文件：/usr/local/spark/jars/protobuf-java-2.5 .0.jar，文件：/usr/local/spark/jars/base64-2.3.8.jar，文件：/usr/local/spark/jars/libthrift-0.9.2.jar，文件：/usr/local/ spark/jars/hadoop-mapreduce-client-jobclient-2.6.4.jar，文件：/usr/local/spark/jars/jackson-core-2.6.5.jar，文件：/usr/local/spark/jars/ commons-math3-3.4.1.jar，文件：/usr/local/spark/jars/bcprov-jdk15on-1.51.jar，文件：/usr/local/spark/jars/spark-mllib-local_2.11-2.0。 1.jar，文件：/usr/local/spark/jars/spark-catalyst_2.11-2.0.1.jar，文件：/usr/local/spark/jars/commons-digester-1.8.jar，文件：/usr /local/spark/jars/jul-to-slf4j-1.7.16.jar,file:/etc/hadoop/conf.pseudo/] 和父是 sun.misc.Launcher$ExtClassLoader@69c67db of t ype 类 sun.misc.Launcher$ExtClassLoader 与类路径 [file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/dnsns.jar,file:/usr/lib/jvm/java- 7-openjdk-amd64/jre/lib/ext/localedata.jar，文件：/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/sunjce_provider.jar，文件：/usr/lib/ jvm/java-7-openjdk-amd64/jre/lib/ext/java-atk-wrapper.jar，文件：/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/libatk-wrapper .so，文件：/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/sunpkcs11.jar，文件：/usr/lib/jvm/java-7-openjdk-amd64/jre/lib /ext/zipfs.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/icedtea-sound.jar] 并且父级是具有引导类路径的原始类加载器 [/usr/lib/ jvm/java-7-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java- 7-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-7-openjdk- amd64/jre/lib/jce.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-7-openjdk -amd64/jre/lib/rhino.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre /classes] 未找到。 在 scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:123) 在 scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:22) 在 com.test.main.TestMain$$typecreator7$1.apply(TestMain.scala:50) 在 scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232) 在 scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232) 在 org.apache.spark.sql.SQLImplicits$$typecreator9$1.apply(SQLImplicits.scala:125) 在 scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232) 在 scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232) 在 org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:49) 在 org.apache.spark.sql.SQLImplicits.newProductSeqEncoder(SQLImplicits.scala:125) 在 com.test.main.TestMain$.main(TestMain.scala:50) 在 com.test.main.TestMain.main(TestMain.scala) 在 sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 在 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 在 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 在 java.lang.reflect.Method.invoke(Method.java:606) 在 org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736) 在 org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185) 在 org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) 在 org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) 在 org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

【参考方案1】：

找到了一个解决方案，您将参数添加为 --driver-class-path 并将其与您的程序集 jar 一起提供

例如：

spark-submit --master local --driver-class-path file:///home/test/test.jar --class com.test.TestMain test.jar

但是，这适用于 yarn 客户端和 spark 本地模式。找不到纱线集群模式的优雅解决方案。