如何在 java 中使用 spark 2.0.0 预览版

Posted 2023-04-17

【中文标题】如何在 java 中使用 spark 2.0.0 预览版

【英文标题】：how to use spark 2.0.0 preview in java

【发布时间】：2016-06-06 12:04:55

【问题描述】：

我想从 spark 2.0.0 预览版导入库， 但现在 Maven 存储库中没有任何 spark 2。 我在here 中使用了建议的解决方案。 但它没有用： 错误是：

    The POM for org.apache.spark:spark-core_2.2.11:jar:2.0.0-SNAPSHOT is missing, no dependency information available
The POM for org.apache.spark:spark-mllib_2.11:jar:2.0.0-SNAPSHOT is missing, no dependency information available
The POM for org.apache.spark:spark-sql_2.2.11:jar:2.0.0-SNAPSHOT is missing, no dependency information available
------------------------------------------------------------------------
BUILD FAILURE

The following artifacts could not be resolved: org.apache.spark:spark-core_2.2.11:jar:2.0.0-SNAPSHOT, org.apache.spark:spark-mllib_2.11:jar:2.0.0-SNAPSHOT, org.apache.spark:spark-sql_2.2.11:jar:2.0.0-SNAPSHOT: Could not find artifact org.apache.spark:spark-core_2.2.11:jar:2.0.0-SNAPSHOT -> [Help 1]

我的 pom 文件是这样的：

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.2.11</artifactId>
            <version>2.0.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-mllib_2.11</artifactId>
            <version>2.0.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.2.11</artifactId>
            <version>2.0.0-SNAPSHOT</version>
        </dependency>

【问题讨论】：

你的 pom.xml 看起来怎么样？

【参考方案1】：

将此添加到您的 pom.xml：

<repositories>
    <repository>
        <id>asf-snapshot</id>
        <url>http://repository.apache.org/snapshots</url>
        <snapshots>
            <enabled>true</enabled>
        </snapshots>
    </repository>
</repositories>

【参考方案2】：

2.0.0-preview 可在Apache Spark Core 2.11 获得。您可以按如下方式修改您的 pom.xml：

<properties>
    <spark.version>2.0.0-preview</spark.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>$spark.version</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>$spark.version</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-yarn_2.11</artifactId>
        <version>$spark.version</version>
    </dependency>