Spark is not running in local mode, therefore the checkpoint directory must not be on the local……(代码
Posted Alex Hub
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Spark is not running in local mode, therefore the checkpoint directory must not be on the local……(代码相关的知识，希望对你有一定的参考价值。
Spark执行的时候报错：
WARN spark.SparkContext: Spark is not running in local mode, therefore the checkpoint directory must not be on the local filesystem. Directory 'file:///home/checkpointData' appears to be on the local filesystem.
简单翻译下：Spark没有在本地模式下运行，因此检查点目录不能在本地文件系统上。
也就是说，Spark应用跑在集群模式下，checkpoint directory是不可以设置在本地文件系统的。
在HDFS上创建一个目录：
hdfs dfs -mkdir sparkCheckpoint
hdfs dfs -chmod 777 sparkCheckpoint
hdfs dfs -ls
然后在spark里面配置一下就可以了。
sc.setCheckpointDir("hdfs://ns1/tmp/username/sparkCheckpoint")
以上是关于Spark is not running in local mode, therefore the checkpoint directory must not be on the local……(代码的主要内容，如果未能解决你的问题，请参考以下文章
NOT IN vs. NOT EXISTS vs. LEFT JOIN / IS NULL: SQL Server
NOT IN vs. NOT EXISTS vs. LEFT JOIN / IS NULL: SQL Server
解决shell命令"** is not in the sudoers file..."错误
eclipse Archive for required library in project cannot be read or is not a valid ZIP file
Exception in thread “main“ java.lang.UnsupportedOperationException: Schema for type Any is not suppo