从 pyspark 中的多行文件中读取 JSON 文件

Posted 2023-04-18

【中文标题】从 pyspark 中的多行文件中读取 JSON 文件

【英文标题】：Read JSON files from multiple line file in pyspark

【发布时间】：2020-10-06 09:04:07

【问题描述】：

我正在学习 python 中的 spark。我有一个 JSON 文件如下：

  
   "_class":"fdsfsdfsdfds",
   "n":"test ccorona",
   "fn":"ccorona",
   "ln":"ccorona",
   "un":"ccorona",
   "p":"line",
   "g":"ccorona",
   "l":"ccorona",
   "e":"ccorona",     
   ,

只有这段代码：

df = spark.read.json("1.json")
df.show()

当我运行节目时，我得到了这个消息

pyspark.sql.utils.AnalysisException: Since Spark 2.3, the queries from raw JSON/CSV files are disallowed when the
referenced columns only include the internal corrupt record column
(named _corrupt_record by default). For example:
spark.read.schema(schema).json(file).filter($"_corrupt_record".isNotNull).count()
and spark.read.schema(schema).json(file).select("_corrupt_record").show().
Instead, you can cache or save the parsed results and then send the same query.
For example, val df = spark.read.schema(schema).json(file).cache() and then
df.filter($"_corrupt_record".isNotNull).count().;

还有

df.printSchema()
root
 |-- _corrupt_record: string (nullable = true)

还有这个

df.select("fn")

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/spark/python/pyspark/sql/dataframe.py", line 1421, in select
    jdf = self._jdf.select(self._jcols(*cols))
  File "/opt/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py", line 1304, in __call__
  File "/opt/spark/python/pyspark/sql/utils.py", line 134, in deco
    raise_from(converted)
  File "<string>", line 3, in raise_from
pyspark.sql.utils.AnalysisException: cannot resolve '`fn`' given input columns: [_corrupt_record];;
'Project ['fn]
+- Relation[_corrupt_record#168] json

如何从多行读取 Spark 中的 JSON sqlContext？

【参考方案1】：

这就是你阅读多行 JSON 的方式：

df = spark.read.option("multiLine", "true").option("mode", "PERMISSIVE").json("1.json")

【讨论】：

嘿，谢谢，但是 df = spark.read.option("multiLine", true).option("mode", "PERMISSIVE").json("1.json") Traceback (最近最后调用）：文件“”，第 1 行，在 NameError: name 'true' is not defined

现在试试，“true”不在引号中。