Spark 类型不匹配：无法从 DataFrame 转换为 Dataset<Row>

Posted 2023-04-15

【中文标题】Spark 类型不匹配：无法从 DataFrame 转换为 Dataset<Row>

【英文标题】：Spark Type mismatch: cannot convert from DataFrame to Dataset<Row>

【发布时间】：2017-02-04 13:06:49

【问题描述】：

我收到一个奇怪的错误提示

Type mismatch: cannot convert from DataFrame to Dataset<Row>

当我尝试实现来自here 的示例代码时。

这是给我错误的行。

    Dataset<Row> verDF = spark.createDataFrame(uList, User.class);

我还查看了 Spark here 的文档，其中给出了相同的示例。但我不确定为什么它在我的情况下不起作用。

这是我的imports：

import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.graphx.*;
import org.apache.spark.graphx.lib.*;
import org.apache.spark.rdd.RDD;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.storage.StorageLevel;
import org.graphframes.GraphFrame;

import scala.Tuple2;
import scala.collection.Iterator;
import scala.collection.immutable.Map;
import scala.collection.immutable.Seq;

以下是相关的依赖：

    <repositories>
        <repository>
            <id>cloudera</id>
            <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
        </repository>
        <repository>
            <id>SparkPackagesRepo</id>
            <url>http://dl.bintray.com/spark-packages/maven</url>
        </repository>
    </repositories>

    <dependencies>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>$hadoop.version</version>
        </dependency>
    <dependency>
        <groupId>graphframes</groupId>
        <artifactId>graphframes</artifactId>
        <version>0.2.0-spark2.0-s_2.11</version>
    </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.10</artifactId>
            <version>$spark.version</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.10</artifactId>
            <version>$spark.version</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-mllib_2.10 -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-mllib_2.10</artifactId>
            <version>1.3.0</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-graphx_2.10 -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-graphx_2.10</artifactId>
            <version>2.1.0</version>
        </dependency>




    </dependencies>

【参考方案1】：

解决了这个问题。我使用了以下依赖项并使用SparkSession 实例来创建数据框。

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>2.0.0-cloudera1-SNAPSHOT</version>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.11</artifactId>
    <version>2.0.0-cloudera1-SNAPSHOT</version>
</dependency>