Spack 2.4.5 和 Scala 2.11 使用哪个版本的 Sbt 和 Sbt 程序集？
Posted
技术标签:
【中文标题】Spack 2.4.5 和 Scala 2.11 使用哪个版本的 Sbt 和 Sbt 程序集？【英文标题】：Which version of Sbt and Sbt assembly to use for Spack 2.4.5 and Scala 2.11? 【发布时间】：2020-12-25 23:58:55 【问题描述】：目前，在 AWS EMR 集群中，我使用的是 Spark v2.4.5，它带有 Scala v2.11。所以在我的项目中，我想使用 Scala v2.11 和相应的SBT 和Sbt-Assembly 版本。但我得到一个或另一个版本与各种博客和 *** 答案上可用的所有排列冲突。
这是我抛出错误的依赖文件：
build.sbt
name := "user-view-services"
version := "0.1"
scalaVersion := "2.11.12"
autoScalaLibrary := false
// Major Libraries version
val sparkVersion = "2.4.5"
val kafkaVersion = "2.5.0"
val akkaVersion = "2.6.4"
val confluentRepo = "confluent-repo" at "https://packages.confluent.io/maven/"
lazy val commonDependencies = Seq(
"junit" % "junit" % "4.13" % Test,
"org.mockito" % "mockito-all" % "1.10.19" % Test,
"com.novocode" % "junit-interface" % "0.11" % Test,
"ch.qos.logback" % "logback-classic" % "1.2.3",
"com.typesafe" % "config" % "1.4.0",
"com.opencsv" % "opencsv" % "5.1",
"org.apache.avro" % "avro" % "1.9.2",
"commons-beanutils" % "commons-beanutils" % "1.9.4",
"commons-beanutils" % "commons-beanutils-bean-collections" % "1.8.3",
"org.drools" % "drools-compiler" % "7.41.0.Final",
"com.typesafe.scala-logging" % "scala-logging_2.11" % "3.5.0"
)
lazy val mongoDBDependencies = Seq(
"org.mongodb" % "mongodb-driver-sync" % "4.0.2"
)
lazy val akkaDependencies = Seq(
"com.typesafe.akka" %% "akka-actor-typed" % akkaVersion
)
lazy val kafkaDependencies = Seq(
"org.apache.kafka" % "kafka-clients" % kafkaVersion,
"org.apache.kafka" % "kafka-streams" % kafkaVersion,
"io.confluent" % "kafka-avro-serializer" % "5.4.1"
)
lazy val sparkDependencies = Seq(
"org.apache.spark" %% "spark-core" % sparkVersion,
"org.apache.spark" %% "spark-sql" % sparkVersion,
"org.apache.spark" %% "spark-streaming" % sparkVersion
)
lazy val rulemodule = (project in file("rule-module"))
.enablePlugins(JavaAppPackaging)
.settings(
name := "rule-module",
version := "1.0",
libraryDependencies ++= commonDependencies
)
/**
* to create uberjar - run command
* 1) sbt "sparkjobs / clean"
* 2) sbt "sparkjobs / assembly"
*/
lazy val sparkjobs = (project in file("user-enrich-batch"))
.dependsOn(rulemodule)
.settings(
name := "user-enrich-batch",
version := "1.0",
resolvers ++= Seq(
confluentRepo
),
libraryDependencies ++= commonDependencies,
libraryDependencies ++= sparkDependencies,
libraryDependencies ++= kafkaDependencies,
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.6.7",
assemblyMergeStrategy in assembly :=
case PathList("META-INF", xs @ _*) => MergeStrategy.discard
case x => MergeStrategy.first
,
mainClass in assembly := some("com.analytics.user.etl.ETLProcessor"),
assemblyJarName := "user-view-batch-assembly.jar"
)
lazy val userservice = (project in file("user-enrich-service"))
.dependsOn(rulemodule)
.enablePlugins(JavaAppPackaging)
.settings(
name := "user-enrich-service",
version := "1.0",
resolvers ++= Seq(
confluentRepo
),
libraryDependencies ++= commonDependencies,
libraryDependencies ++= kafkaDependencies,
libraryDependencies ++= akkaDependencies,
libraryDependencies ++= mongoDBDependencies
)
build.properties
sbt.version = 0.13.17
plugins.sbt
addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.3.4")
scalaVersion := "2.11.12"
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.6")
addSbtPlugin("com.cavorite" % "sbt-avro-1-9" % "1.1.7")
程序集.sbt
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.6")
错误：
[info] Loading global plugins from C:\Users\smishra5\.sbt\0.13\plugins
[info] Loading project definition from C:\dataanalytics\Recsys\project
[info] Updating file:/C:/dataanalytics/Recsys/project/recsys-build...
[info] Resolving com.eed3si9n#sbt-assembly;0.14.6 ...
[warn] module not found: com.eed3si9n#sbt-assembly;0.14.6
[warn] ==== typesafe-ivy-releases: tried
[warn] https://repo.typesafe.com/typesafe/ivy-releases/com.eed3si9n/sbt-assembly/scala_2.11/sbt_0.13/0.14.6/ivys/ivy.xml
[warn] ==== sbt-plugin-releases: tried
[warn] https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.eed3si9n/sbt-assembly/scala_2.11/sbt_0.13/0.14.6/ivys/ivy.xml
[warn] ==== local: tried
[warn] C:\Users\smishra5\.ivy2\local\com.eed3si9n\sbt-assembly\scala_2.11\sbt_0.13\0.14.6\ivys\ivy.xml
[warn] ==== public: tried
[warn] https://repo1.maven.org/maven2/com/eed3si9n/sbt-assembly_2.11_0.13/0.14.6/sbt-assembly-0.14.6.pom
[warn] ==== local-preloaded-ivy: tried
[warn] C:\Users\smishra5\.sbt\preloaded\com.eed3si9n\sbt-assembly\0.14.6\ivys\ivy.xml
[warn] ==== local-preloaded: tried
[warn] file:/C:/Users/smishra5/.sbt/preloaded/com/eed3si9n/sbt-assembly_2.11_0.13/0.14.6/sbt-assembly-0.14.6.pom
[info] Resolving com.typesafe.sbt#sbt-native-packager;1.3.4 ...
[warn] module not found: com.typesafe.sbt#sbt-native-packager;1.3.4
[warn] ==== typesafe-ivy-releases: tried
[warn] https://repo.typesafe.com/typesafe/ivy-releases/com.typesafe.sbt/sbt-native-packager/scala_2.11/sbt_0.13/1.3.4/ivys/ivy.xml
[warn] ==== sbt-plugin-releases: tried
[warn] https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.typesafe.sbt/sbt-native-packager/scala_2.11/sbt_0.13/1.3.4/ivys/ivy.xml
[warn] ==== local: tried
[warn] C:\Users\smishra5\.ivy2\local\com.typesafe.sbt\sbt-native-packager\scala_2.11\sbt_0.13\1.3.4\ivys\ivy.xml
[warn] ==== public: tried
[warn] https://repo1.maven.org/maven2/com/typesafe/sbt/sbt-native-packager_2.11_0.13/1.3.4/sbt-native-packager-1.3.4.pom
[warn] ==== local-preloaded-ivy: tried
[warn] C:\Users\smishra5\.sbt\preloaded\com.typesafe.sbt\sbt-native-packager\1.3.4\ivys\ivy.xml
[warn] ==== local-preloaded: tried
[warn] file:/C:/Users/smishra5/.sbt/preloaded/com/typesafe/sbt/sbt-native-packager_2.11_0.13/1.3.4/sbt-native-packager-1.3.4.pom
[info] Resolving com.cavorite#sbt-avro-1-9;1.1.7 ...
[warn] module not found: com.cavorite#sbt-avro-1-9;1.1.7
[warn] ==== typesafe-ivy-releases: tried
[warn] https://repo.typesafe.com/typesafe/ivy-releases/com.cavorite/sbt-avro-1-9/scala_2.11/sbt_0.13/1.1.7/ivys/ivy.xml
[warn] ==== sbt-plugin-releases: tried
[warn] https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.cavorite/sbt-avro-1-9/scala_2.11/sbt_0.13/1.1.7/ivys/ivy.xml
[warn] ==== local: tried
[warn] C:\Users\smishra5\.ivy2\local\com.cavorite\sbt-avro-1-9\scala_2.11\sbt_0.13\1.1.7\ivys\ivy.xml
[warn] ==== public: tried
[warn] https://repo1.maven.org/maven2/com/cavorite/sbt-avro-1-9_2.11_0.13/1.1.7/sbt-avro-1-9-1.1.7.pom
[warn] ==== local-preloaded-ivy: tried
[warn] C:\Users\smishra5\.sbt\preloaded\com.cavorite\sbt-avro-1-9\1.1.7\ivys\ivy.xml
[warn] ==== local-preloaded: tried
[warn] file:/C:/Users/smishra5/.sbt/preloaded/com/cavorite/sbt-avro-1-9_2.11_0.13/1.1.7/sbt-avro-1-9-1.1.7.pom
[info] Resolving jline#jline;2.14.3 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.eed3si9n#sbt-assembly;0.14.6: not found
[warn] :: com.typesafe.sbt#sbt-native-packager;1.3.4: not found
[warn] :: com.cavorite#sbt-avro-1-9;1.1.7: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Some unresolved dependencies have extra attributes. Check that these dependencies exist with the requested attributes.
[warn] com.eed3si9n:sbt-assembly:0.14.6 (scalaVersion=2.11, sbtVersion=0.13)
[warn] com.typesafe.sbt:sbt-native-packager:1.3.4 (scalaVersion=2.11, sbtVersion=0.13)
[warn] com.cavorite:sbt-avro-1-9:1.1.7 (scalaVersion=2.11, sbtVersion=0.13)
[warn]
[warn] Note: Unresolved dependencies path:
[warn] com.eed3si9n:sbt-assembly:0.14.6 (scalaVersion=2.11, sbtVersion=0.13) (C:\dataanalytics\Recsys\project\plugins.sbt#L5-6)
[warn] +- default:recsys-build:0.1-SNAPSHOT (scalaVersion=2.11, sbtVersion=0.13)
[warn] com.typesafe.sbt:sbt-native-packager:1.3.4 (scalaVersion=2.11, sbtVersion=0.13) (C:\dataanalytics\Recsys\project\plugins.sbt#L1-2)
[warn] +- default:recsys-build:0.1-SNAPSHOT (scalaVersion=2.11, sbtVersion=0.13)
[warn] com.cavorite:sbt-avro-1-9:1.1.7 (scalaVersion=2.11, sbtVersion=0.13) (C:\dataanalytics\Recsys\project\plugins.sbt#L7-8)
[warn] +- default:recsys-build:0.1-SNAPSHOT (scalaVersion=2.11, sbtVersion=0.13)
sbt.ResolveException: unresolved dependency: com.eed3si9n#sbt-assembly;0.14.6: not found
unresolved dependency: com.typesafe.sbt#sbt-native-packager;1.3.4: not found
unresolved dependency: com.cavorite#sbt-avro-1-9;1.1.7: not found
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:320)
at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:191)
at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:168)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:156)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:156)
at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:133)
at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:57)
at sbt.IvySbt$$anon$4.call(Ivy.scala:65)
at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:95)
at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:80)
at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:99)
at xsbt.boot.Using$.withResource(Using.scala:10)
at xsbt.boot.Using$.apply(Using.scala:9)
at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:60)
at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:50)
at xsbt.boot.Locks$.apply0(Locks.scala:31)
at xsbt.boot.Locks$.apply(Locks.scala:28)
at sbt.IvySbt.withDefaultLogger(Ivy.scala:65)
at sbt.IvySbt.withIvy(Ivy.scala:128)
at sbt.IvySbt.withIvy(Ivy.scala:125)
at sbt.IvySbt$Module.withModule(Ivy.scala:156)
at sbt.IvyActions$.updateEither(IvyActions.scala:168)
at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1555)
at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1551)
at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$122.apply(Defaults.scala:1586)
at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$122.apply(Defaults.scala:1584)
at sbt.Tracked$$anonfun$lastOutput$1.apply(Tracked.scala:37)
at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1589)
at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1583)
at sbt.Tracked$$anonfun$inputChanged$1.apply(Tracked.scala:60)
at sbt.Classpaths$.cachedUpdate(Defaults.scala:1606)
at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1533)
at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1485)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
at sbt.std.Transform$$anon$4.work(System.scala:63)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
at sbt.Execute.work(Execute.scala:237)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[error] (*:update) sbt.ResolveException: unresolved dependency: com.eed3si9n#sbt-assembly;0.14.6: not found
[error] unresolved dependency: com.typesafe.sbt#sbt-native-packager;1.3.4: not found
[error] unresolved dependency: com.cavorite#sbt-avro-1-9;1.1.7: not found
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore? q
参考：https://github.com/sbt/sbt-assembly/issues/311
【问题讨论】：
sbt 汇编插件版本似乎有问题，请看这个：https://github.com/sbt/sbt-assembly/issues/245 和这个https://***.com/questions/47788640/module-not-found-com-eed3si9nsbt-assembly0-14-3
这能回答你的问题吗？ module not found: com.eed3si9n#sbt-assembly;0.14.3
尝试将 sbt-assembly;0.14.6 和 0.14.0 与 sbt 版本 0.13.13 或 1.0.0 或 1.0.4 一起使用，但它在所有这些排列中都失败了，错误如下：未解决依赖项：com.eed3si9n#sbt-assembly；0.14.6：未找到 [错误] 未解决的依赖项：com.typesafe.sbt#sbt-native-packager；1.3.4：未找到 [错误] 未解决的依赖项：com.cavorite# sbt-avro-1-9;1.1.7: 未找到
【参考方案1】：
您似乎混淆了 SBT 使用的 Scala 版本和项目中使用的 Scala 版本。
如果您需要使用 Scala 2.11 构建项目，只需在 build.sbt 中指定即可
scalaVersion := "2.11.12"
您必须从 project 文件夹中的 plugins.sbt 中删除 scalaVersion := "2.11.12"，因为 SBT 是使用特定 Scala 版本预先构建的，您无法更改它。例如，0.13.17 是使用 Scala 2.10 构建的，而 SBT 1.x.x 是使用 Scala 2.12 构建的。
您可以随意使用任何您喜欢的 SBT 版本，但我建议使用最新的 1.3.13 和相应插件的最新版本。
【讨论】：
在将“sbt-assembly”版本更改为“0.14.10”并从plugins.sbt中删除scalaVersion之后工作想知道我想用ie构建我的项目的Scala版本之间是否有任何关系Scala 2.11.12 和我使用的 SBT 版本，即 Sbt 0.13.13 和 sbt-assembly 版本，即 0.14.10 @sunny_dev 正如您可能已经了解的那样，sbt 构建的 Scala 版本独立于您在项目中使用的版本。至于 SBT 插件，它们是为不同的 SBT 版本（和 Scala 版本）交叉构建的，您可以在 bintray 看到与 sbt 0.13 一起工作的 sbt-assembly 插件版本以上是关于Spack 2.4.5 和 Scala 2.11 使用哪个版本的 Sbt 和 Sbt 程序集？的主要内容，如果未能解决你的问题，请参考以下文章
大数据平台搭建：Hadoop-3.1.3+Hive-3.1.2+HBase-2.2.3+Zookeeper-3.5.7+Kafka_2.11-2.4.0+Spark-2.4.5