为什么在我的build.sbt中没有正确遮蔽番石榴?这里
Posted
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了为什么在我的build.sbt中没有正确遮蔽番石榴?这里相关的知识,希望对你有一定的参考价值。
tl;dr: Here's a repo containing the problem.
Cassandra和HDFS都在内部使用番石榴,但由于各种原因,它们都不会影响依赖性。因为番石榴的版本不是二进制兼容的,所以我在运行时找到了NoSuchMethodError
s。
我试图在我的build.sbt
中自己遮蔽番石榴:
val HadoopVersion = "2.6.0-cdh5.11.0"
// ...
val hadoopHdfs = "org.apache.hadoop" % "hadoop-hdfs" % HadoopVersion
val hadoopCommon = "org.apache.hadoop" % "hadoop-common" % HadoopVersion
val hadoopHdfsTest = "org.apache.hadoop" % "hadoop-hdfs" % HadoopVersion % "test" classifier "tests"
val hadoopCommonTest = "org.apache.hadoop" % "hadoop-common" % HadoopVersion % "test" classifier "tests"
val hadoopMiniDFSCluster = "org.apache.hadoop" % "hadoop-minicluster" % HadoopVersion % Test
// ...
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("com.google.common.**" -> "shade.com.google.common.@1").inLibrary(hadoopHdfs).inProject,
ShadeRule.rename("com.google.common.**" -> "shade.com.google.common.@1").inLibrary(hadoopCommon).inProject,
ShadeRule.rename("com.google.common.**" -> "shade.com.google.common.@1").inLibrary(hadoopHdfsTest).inProject,
ShadeRule.rename("com.google.common.**" -> "shade.com.google.common.@1").inLibrary(hadoopCommonTest).inProject,
ShadeRule.rename("com.google.common.**" -> "shade.com.google.common.@1").inLibrary(hadoopMiniDFSCluster).inProject
)
assemblyJarName in assembly := s"${name.value}-${version.value}.jar"
assemblyMergeStrategy in assembly := {
case PathList("META-INF", "MANIFEST.MF") => MergeStrategy.discard
case _ => MergeStrategy.first
}
但运行时异常仍然存在(哈 - 这是一个cassandra笑话,人们)。
具体的例外是
[info] HdfsEntitySpec *** ABORTED ***
[info] java.lang.NoSuchMethodError: com.google.common.base.Objects.toStringHelper(Ljava/lang/Object;)Lcom/google/common/base/Objects$ToStringHelper;
[info] at org.apache.hadoop.metrics2.lib.MetricsRegistry.toString(MetricsRegistry.java:406)
[info] at java.lang.String.valueOf(String.java:2994)
[info] at java.lang.StringBuilder.append(StringBuilder.java:131)
[info] at org.apache.hadoop.ipc.metrics.RetryCacheMetrics.<init>(RetryCacheMetrics.java:46)
[info] at org.apache.hadoop.ipc.metrics.RetryCacheMetrics.create(RetryCacheMetrics.java:53)
[info] at org.apache.hadoop.ipc.RetryCache.<init>(RetryCache.java:202)
[info] at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initRetryCache(FSNamesystem.java:1038)
[info] at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:949)
[info] at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:796)
[info] at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1040)
[info] ...
如何正确遮挡番石榴以阻止运行时错误?
答案
着色规则仅适用于构建胖罐的情况。在其他sbt任务期间不会应用它。
如果你想在hadoop依赖项中隐藏一些库,你可以创建一个只包含hadoop依赖项的新项目,遮蔽库,并发布一个带有所有着色hadoop依赖项的胖jar。
这不是一个完美的解决方案,因为新的hadoop jar中的所有依赖项对于使用它们的人来说都是“未知”,并且您需要手动处理冲突。
这是您在build.sbt
中需要的代码,用于发布胖的hadoop jar(使用您的代码和sbt程序集docs):
val HadoopVersion = "2.6.0-cdh5.11.0"
val hadoopHdfs = "org.apache.hadoop" % "hadoop-hdfs" % HadoopVersion
val hadoopCommon = "org.apache.hadoop" % "hadoop-common" % HadoopVersion
val hadoopHdfsTest = "org.apache.hadoop" % "hadoop-hdfs" % HadoopVersion classifier "tests"
val hadoopCommonTest = "org.apache.hadoop" % "hadoop-common" % HadoopVersion % classifier "tests"
val hadoopMiniDFSCluster = "org.apache.hadoop" % "hadoop-minicluster" % HadoopVersion
lazy val fatJar = project
.enablePlugins(AssemblyPlugin)
.settings(
libraryDependencies ++= Seq(
hadoopHdfs,
hadoopCommon,
hadoopHdfsTest,
hadoopCommonTest,
hadoopMiniDFSCluster
),
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("com.google.common.**" -> "shade.@0").inAll
),
assemblyMergeStrategy in assembly := {
case PathList("META-INF", "MANIFEST.MF") => MergeStrategy.discard
case _ => MergeStrategy.first
},
artifact in (Compile, assembly) := {
val art = (artifact in (Compile, assembly)).value
art.withClassifier(Some("assembly"))
},
addArtifact(artifact in (Compile, assembly), assembly),
crossPaths := false, // Do not append Scala versions to the generated artifacts
autoScalaLibrary := false, // This forbids including Scala related libraries into the dependency
skip in publish := true
)
lazy val shaded_hadoop = project
.settings(
name := "shaded-hadoop",
packageBin in Compile := (assembly in (fatJar, Compile)).value
)
我没有测试它,但这是它的要点。
我想指出我注意到的另一个问题,你的合并策略可能会给你带来问题,因为你想对某些文件应用不同的策略。看到默认策略here。
我建议使用这样的东西来保留原来的策略,不是deduplicate
assemblyMergeStrategy in assembly := {
entry: String => {
val strategy = (assemblyMergeStrategy in assembly).value(entry)
if (strategy == MergeStrategy.deduplicate) MergeStrategy.first
else strategy
}
}
以上是关于为什么在我的build.sbt中没有正确遮蔽番石榴?这里的主要内容,如果未能解决你的问题,请参考以下文章
sbt.build 用于多个项目和 playframework