PySpark - Parquet - 调用 None.None 时发生错误

Posted 2023-04-15

【中文标题】PySpark - Parquet - 调用 None.None 时发生错误

【英文标题】：PySpark - Parquet - An error occurred while calling None.None

【发布时间】：2017-07-21 11:27:38

【问题描述】：

我有一个 hdf5 文件，其中包含大约 200 个组。因此，对于每个组，我编写了一个代码，将它们转换为镶木地板文件：

fastparquet.write(dir_name + '/' + metricname + '.parq', groupDataFrame)

在我尝试阅读它们之后，它工作得很好，除了一些我收到错误消息的指标：（我使用 HDFVIEW 来查看这个指标的值是否有问题，但我什么也没看到，只有整数值）

ERROR:root:Exception while sending command.
Traceback (most recent call last):
  File "C:\prog\spark-2.1.0-bin-hadoop2.6\python\lib\py4j-0.10.4-src.zip\py4j\java_gateway.py", line 883, in send_command
    response = connection.send_command(command)
  File "C:\prog\spark-2.1.0-bin-hadoop2.6\python\lib\py4j-0.10.4-src.zip\py4j\java_gateway.py", line 1040, in send_command
    "Error while receiving", e, proto.ERROR_ON_RECEIVE)
Py4JNetworkError: Error while receiving
---------------------------------------------------------------------------
Py4JError                                 Traceback (most recent call last)
C:\Users\uids2739\AppData\Local\Continuum\Anaconda2\lib\site-packages\IPython\core\interactiveshell.pyc in run_code(self, code_obj, result)
   2896             if result is not None:
   2897                 result.error_in_exec = sys.exc_info()[1]
-> 2898             self.showtraceback()
   2899         else:
   2900             outflag = 0

C:\Users\uids2739\AppData\Local\Continuum\Anaconda2\lib\site-packages\IPython\core\interactiveshell.pyc in showtraceback(self, exc_tuple, filename, tb_offset, exception_only)
   1824                                             value, tb, tb_offset=tb_offset)
   1825 
-> 1826                     self._showtraceback(etype, value, stb)
   1827                     if self.call_pdb:
   1828                         # drop into debugger

C:\Users\uids2739\AppData\Local\Continuum\Anaconda2\lib\site-packages\ipykernel\zmqshell.pyc in _showtraceback(self, etype, evalue, stb)
    509             u'traceback' : stb,
    510             u'ename' : unicode_type(etype.__name__),
--> 511             u'evalue' : py3compat.safe_unicode(evalue),
    512         
    513 

C:\Users\uids2739\AppData\Local\Continuum\Anaconda2\lib\site-packages\ipython_genutils\py3compat.pyc in safe_unicode(e)
     63     """
     64     try:
---> 65         return unicode_type(e)
     66     except UnicodeError:
     67         pass

C:\prog\spark-2.1.0-bin-hadoop2.6\python\lib\py4j-0.10.4-src.zip\py4j\protocol.py in __str__(self)
    419         gateway_client = self.java_exception._gateway_client
    420         answer = gateway_client.send_command(self.exception_cmd)
--> 421         return_value = get_return_value(answer, gateway_client, None, None)
    422         return "0: 1".format(self.errmsg, return_value)

C:\prog\spark-2.1.0-bin-hadoop2.6\python\lib\py4j-0.10.4-src.zip\py4j\protocol.py in get_return_value(answer, gateway_client, target_id, name)
    325             raise Py4JError(
    326                 "An error occurred while calling 012".
--> 327                 format(target_id, ".", name))
    328     else:
    329         type = answer[1]

Py4JError: An error occurred while calling None.None

感谢您的帮助。

【参考方案1】：

所以这是类型的问题，因为镶木地板不支持所有。 我以这样的 sql 形式发出读取请求：

df = spark.sql("SELECT * FROM parquet.`file.parq`")

给我错误信息：

AnalysisException: u'Job aborted due to stage failure: Task 0 in stage 46.0 failed 1 times, most recent failure: Lost task 0.0 in stage 46.0 (TID 46, localhost, executor driver): org.apache.spark.sql.AnalysisException: Parquet type not supported: INT32

所以我在创建这个 parquet 文件时将类型 int32 更改为 float，问题就解决了。