[Spark]What's the difference between spark.sql.shuffle.partitions and spark.default.parallelism?
Posted 深圳私塾
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了[Spark]What's the difference between spark.sql.shuffle.partitions and spark.default.parallelism?相关的知识，希望对你有一定的参考价值。
From the answer here,
spark.sql.shuffle.partitions configures the number of partitions that are used when shuffling data for joins or aggregations.
spark.default.parallelism is the default number of partitions in RDDs returned by transformations like join, reduceByKey, and parallelize when not set explicitly by the user. Note that spark.default.parallelism seems to only be working for raw RDD and is ignored when working with dataframes.
If the task you are performing is not a join or aggregation and you are working with dataframes then setting these will not have any effect. You could, however, set the number of partitions yourself by calling df.repartition(numOfPartitions) (don‘t forget to assign it to a new val) in your code.
以上是关于[Spark]What's the difference between spark.sql.shuffle.partitions and spark.default.parallelism?的主要内容，如果未能解决你的问题，请参考以下文章
What's the difference between all the Selection Segues?
What's the IO Instruction Restart