spark1.统计句子中特定内容

Posted 小毛驴

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了spark1.统计句子中特定内容相关的知识，希望对你有一定的参考价值。

    val logFile = "./README.md"  // Should be some file on your server.
    val conf = new SparkConf().setAppName("Simple Application").setMaster("local")
    val sc = new SparkContext(conf)
    val logData = sc.textFile(logFile, 2).cache()
//    val numAs = logData.filter(line => line.contains("h")).count()
//    val numBs = logData.filter(line => line.contains("j")).count()
    var  params = List("h","j","c","w");

    var searchAnylisay =   params.map(item => logData.filter(line => line.contains(item)).count() )

    println("searchAnylisay length  : %s,".format(searchAnylisay.length))

    searchAnylisay.foreach( x => println(x))

以上是关于spark1.统计句子中特定内容的主要内容，如果未能解决你的问题，请参考以下文章