为啥 Spark 应用程序失败并显示“IOException: (null) entry in command string: null chmod 0644”？ [复制]

Posted 2023-04-17

【中文标题】为啥 Spark 应用程序失败并显示“IOException: (null) entry in command string: null chmod 0644”？ [复制]

【英文标题】：Why does Spark application fail with "IOException: (null) entry in command string: null chmod 0644"? [duplicate]为什么 Spark 应用程序失败并显示“IOException: (null) entry in command string: null chmod 0644”？ [复制]

【发布时间】：2017-12-28 16:04:17

【问题描述】：

我正在尝试使用下面的 JAVA 将数据集结果写入单个 CSV

dataset.write().mode(SaveMode.Overwrite).option("header",true).csv("C:\\tmp\\csvs");

但是超时，文件没有被写入。

抛出org.apache.spark.SparkException: Job aborted.

错误：

org.apache.spark.SparkException: Job aborted due to stage failure:

Task 0 in stage 13.0 failed 1 times, most recent failure: Lost task 0.0 in stage 13.0 (TID 16, localhost): java.io.IOException: (null) entry in command string: null chmod 0644 C:\tmp\12333333testSpark\_temporary\0\_temporary\attempt_201712282255_0013_m_000000_0\part-r-00000-229fd1b6-ffb9-4ba1-9dc9-89dfdbd0be43.csv
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:770)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:225)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:132)
at org.apache.spark.sql.execution.datasources.csv.CsvOutputWriter.<init>(CSVRelation.scala:200)
at org.apache.spark.sql.execution.datasources.csv.CSVOutputWriterFactory.newInstance(CSVRelation.scala:170)
at org.apache.spark.sql.execution.datasources.BaseWriterContainer.newOutputWriter(WriterContainer.scala:131)
at org.apache.spark.sql.execution.datasources.DefaultWriterContainer.writeRows(WriterContainer.scala:247)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(InsertIntoHadoopFsRelationCommand.scala:143)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(InsertIntoHadoopFsRelationCommand.scala:143)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

【问题讨论】：

看看这个。 ***.com/questions/40764807/…

正在创建输出文件夹，但不是文件。它显示相同的错误。 :(

【参考方案1】：

您可能希望缩小范围以修复以下异常：

java.io.IOException: (null) entry in command string: null chmod 0644

尝试将HADOOP_HOME 设置为bin\winutils.exe 的子目录，如SO question 中所述。如果这没有帮助，另一个SO link 报告了一个解决方法。

【讨论】：

这确实有效。谢谢，但是你能说出为什么会出现这个问题吗？

显然这个问题与 Hadoop bin tarball 中的 missing binary file for Windows 有关。

但是我在配置了 spark 的 EMR 集群中也遇到了同样的错误。完全相同的错误。那里我不能复制这个winutils。我可以在那里做什么？？？

在这种情况下，我建议您提交AWS support request 来解决问题。