使用 org.apache.hadoop/* 依赖项离线编译 sbt 的问题

Posted

技术标签:

【中文标题】使用 org.apache.hadoop/* 依赖项离线编译 sbt 的问题【英文标题】:Troubles with sbt compiling offline using org.apache.hadoop/* dependencies 【发布时间】:2015-07-04 20:58:18 【问题描述】:

在离线编译时遇到很多麻烦,因为 sbt 依赖于 org.apache.hadoop 包。

一个简单的 build.sbt:

name := "Test"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.hadoop" % "hadoop-yarn-api" % "2.2.0"

在线时工作正常,但离线运行时出现以下错误,而包存在于常春藤缓存中(在 ~/ivy2/cache/org.apache.hadoop/...下):

[info] Loading project definition from /home/martin/Dev/S/project
[info] Set current project to Test (in build file:/home/martin/Dev/S/)
[info] Updating file:/home/martin/Dev/S/s...
[info] Resolving org.apache.hadoop#hadoop-yarn-api;2.2.0 ...
[warn] Host repo1.maven.org not found. url=https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-api/2.2.0/hadoop-yarn-api-2.2.0.pom
[info] You probably access the destination server through a proxy server that is not well configured.
[warn]  module not found: org.apache.hadoop#hadoop-yarn-api;2.2.0
[warn] ==== local: tried
[warn]   /home/martin/.ivy2/local/org.apache.hadoop/hadoop-yarn-api/2.2.0/ivys/ivy.xml
[warn] ==== public: tried
[warn]   https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-api/2.2.0/hadoop-yarn-api-2.2.0.pom
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: org.apache.hadoop#hadoop-yarn-api;2.2.0: not found
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn] 
[warn]  Note: Unresolved dependencies path:
[warn]      org.apache.hadoop:hadoop-yarn-api:2.2.0 (/home/martin/Dev/S/build.sbt#L15-16)
[warn]        +- test:test_2.10:1.0
sbt.ResolveException: unresolved dependency: org.apache.hadoop#hadoop-yarn-api;2.2.0: not found
    ...
[error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.hadoop#hadoop-yarn-api;2.2.0: not found
[error] Total time: 3 s, completed Apr 26, 2015 2:46:58 PM

添加以下解析器没有帮助:

resolvers += Resolver.file("Local repo", file(System.getProperty("user.home") + "/.ivy2/cache")) (Resolver.ivyStylePatterns)

只是增加了

[warn] ==== Local repo: tried
[warn]   /home/martin/.ivy2/cache/org.apache.hadoop/hadoop-yarn-api/2.2.0/ivys/ivy.xml

文件存在但命名为 ivy-2.2.0.xml,而不是 2.2.0/ivys/ivy.xml

所以我尝试添加

resolvers += Resolver.file("Local repo 2", file(System.getProperty("user.home") + "/.ivy2/cache")) ( Patterns("[organisation]/[module]/[artifact]-[revision].[ext]") )

强制命名约定,但它会在下面查看

[warn] ==== Local repo 2: tried
[warn]   /home/martin/.ivy2/cache/org/apache/hadoop/hadoop-yarn-api/ivy-2.2.0.xml

即使根据 sbt doc [organisation] 应该是 org.apache.hadoop 而不是 org/apache/hadoop

所以最后作为最后的手段,我添加了一个丑陋的

resolvers += Resolver.file("Local hadoop cache", file(System.getProperty("user.home") + "/.ivy2/cache")) ( Patterns("org.apache.hadoop/[module]/[artifact]-[revision].[ext]") )

它在那里找到了一些东西,但仍然不满意:

[info] Loading project definition from /home/martin/Dev/S/project
[info] Set current project to Test (in build file:/home/martin/Dev/S/)
[info] Updating file:/home/martin/Dev/S/s...
[info] Resolving org.apache.hadoop#hadoop-yarn-api;2.2.0 ...
[warn] Host repo1.maven.org not found. url=https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-yarn-api/2.2.0/hadoop-yarn-api-2.2.0.pom
[info] You probably access the destination server through a proxy server that is not well configured.
[warn] xml parsing: ivy-2.2.0.xml.original:18:69: schema_reference.4: Failed to read schema document 'http://maven.apache.org/xsd/maven-4.0.0.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.
[warn] xml parsing: ivy-2.2.0.xml.original:19:11: schema_reference.4: Failed to read schema document 'http://maven.apache.org/xsd/maven-4.0.0.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.
[warn] xml parsing: ivy-2.2.0.xml.original:20:17: schema_reference.4: Failed to read schema document 'http://maven.apache.org/xsd/maven-4.0.0.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.
[warn] xml parsing: ivy-2.2.0.xml.original:21:14: schema_reference.4: Failed to read schema document 'http://maven.apache.org/xsd/maven-4.0.0.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.
[warn] xml parsing: ivy-2.2.0.xml.original:22:14: schema_reference.4: Failed to read schema document 'http://maven.apache.org/xsd/maven-4.0.0.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.
[warn] xml parsing: ivy-2.2.0.xml.original:24:17: schema_reference.4: Failed to read schema document 'http://maven.apache.org/xsd/maven-4.0.0.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.
[warn] xml parsing: ivy-2.2.0.xml.original:25:12: schema_reference.4: Failed to read schema document 'http://maven.apache.org/xsd/maven-4.0.0.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.
...

我不知道下一步该尝试什么。如果我将 .ivy2/cache/ 目录复制到离线机器,那么对于我尝试过的任何其他依赖项,离线构建都可以正常工作。导致此问题的只是一堆 org.apache.hadoop 依赖项。 .ivy2/cache/org.apache.hadoop 下的结构和文件看起来与其他运行良好的依赖项中的相同。

添加

offline := true

也没有用。

使用 sbt 0.13.7

谢谢!

【问题讨论】:

【参考方案1】:

似乎是一个 sbt 错误。我在使用 sbt 0.13.8 时遇到了同样的问题,但他们改用 0.13.9-RC3 并清理了 ~/.ivy2

【讨论】:

谢谢!虽然我无法再访问导致 OP 的原始项目,但我只是复制了它并验证了您的解决方案。 OP 与 sbt 0.13.7 一起使用,但我可以确认 0.13.8 仍然存在问题,并且已使用 0.13.9-RC3 修复。我不需要清理 ~/.ivy2 目录,但我需要清理项目的本地 target/ 目录才能使离线构建工作。在清理在线和离线工作之后,如果不清理它,只有在线构建工作。仍在使用旧的 ~/.ivy2 不确定project/target 目录,可能是我忘记删除了。很高兴它有帮助。 顺便说一句,0.13.9 版本现已正式发布,并已推送到他们的软件包存储库。再次感谢 我遇到了类似的问题,但使用的是 0.13.12。只需删除有问题的缓存库,例如~/.ivy2/cache/org.apache.hadoop 成功了

以上是关于使用 org.apache.hadoop/* 依赖项离线编译 sbt 的问题的主要内容,如果未能解决你的问题,请参考以下文章

maven package org.apache.hadoop.io does not exist是啥依赖没dao

将 Intellij 与 SBT 一起使用时如何更改提供的依赖项?

flume出现org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme “hdfs“

将 S3(法兰克福)与 Spark 结合使用

Java零基础入门 26:java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hdfs.DFSUt

tried to access methos com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop...(代码