Spark成长之路-Correlation

Posted 2022-12-12 Q博士

spark ml
源码

spark准备彻底支持DataFrame特性，所以重新了ml的api，原先的以RDD为基础的api都放在了mllib中，但是都是维护阶段，推荐使用ml下的api。

相关性

有2种相关性，皮尔森积矩相关系数和斯皮尔曼等级相关，具体原理请自行搜索，主要是判断两个向量的关联性。

样例

import org.apache.spark.ml.linalg.Matrix, Vectors
import org.apache.spark.ml.stat.Correlation
import org.apache.spark.sql.Row, SparkSession


object CorrelationExample 
  def main(args: Array[String]): Unit = 
    val spark = SparkSession.builder.appName("CorrelationExample").getOrCreate()
    spark.sparkContext.setLogLevel("WARN")
    val data = spark.sparkContext.makeRDD(Seq(
      Vectors.sparse(4, Seq((0, 1.0), (3, -2.0))),
      Vectors.dense(4.0, 5.0, 0.0, 3.0),
      Vectors.dense(6.0, 7.0, 0.0, 8.0),
      Vectors.sparse(4, Seq((0, 9.0), (3, 1.0)))
    ))
    import spark.implicits._
    val df = data.map(Tuple1.apply).toDF("features")
    val Row(coeff1: Matrix) = Correlation.corr(df, "features").head
    println("Pearson correlation matrix:\\n" + coeff1.toString)
    val Row(coeff2: Matrix) = Correlation.corr(df, "features", "spearman").head
    println("Spearman correlation matrix:\\n" + coeff2.toString)
  

执行结果

Pearson correlation matrix:
1.0                   0.055641488407465814  NaN  0.4004714203168137  
0.055641488407465814  1.0                   NaN  0.9135958615342522  
NaN                   NaN                   1.0  NaN                 
0.4004714203168137    0.9135958615342522    NaN  1.0                 

Spearman correlation matrix:
1.0                  0.10540925533894532  NaN  0.40000000000000174  
0.10540925533894532  1.0                  NaN  0.9486832980505141   
NaN                  NaN                  1.0  NaN                  
0.40000000000000174  0.9486832980505141   NaN  1.0  

每一行都有四个数，代表当前第几个向量与Seq中的4个向量的相关性，比如皮尔森的第一行结果1.0 0.055641488407465814 NaN 0.4004714203168137与自己的相关性是1.0，与第二个相关性为0.055641488407465814,与第三个无法计算相关性，与第四个相关性0.055641488407465814。