Spark细节汇总
Posted Laurence
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Spark细节汇总相关的知识，希望对你有一定的参考价值。
启动带内置Hudi的Spark-Shell
spark-shell \\
--jars /usr/lib/hudi/hudi-spark-bundle.jar \\
--conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" \\
--conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog" \\
--conf "spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension"
启动带内置Hudi的Spark-SQL
spark-sql \\
--jars /usr/lib/hudi/hudi-spark-bundle.jar \\
--conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" \\
--conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog" \\
--conf "spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension"
启动带特定Hudi版本的Spark-Shell
spark-shell \\
--packages org.apache.hudi:hudi-spark3.3-bundle_2.12:0.13.0 \\
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \\
--conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog' \\
--conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension'
启动带特定Hudi版本的Spark-SQL
以：Spark 3.3 + Scala 2.12 为例：
spark-sql \\
--packages org.apache.hudi:hudi-spark3.3-bundle_2.12:0.13.0 \\
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \\
--conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' \\
--conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog'
打印Spark所有配置
cat | spark-shell <<EOF
spark.sparkContext.getConf.getAll.foreach(
conf => println(conf._1 +" = "+ conf._2)
)
EOF
以上是关于Spark细节汇总的主要内容，如果未能解决你的问题，请参考以下文章