Spark MLlib速成宝典模型篇04朴素贝叶斯Naive Bayes(Python版)
Posted 黎明程序员
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Spark MLlib速成宝典模型篇04朴素贝叶斯Naive Bayes(Python版)相关的知识,希望对你有一定的参考价值。
目录
朴素贝叶斯原理
朴素贝叶斯代码(Spark Python)
朴素贝叶斯原理 |
详见博文:http://www.cnblogs.com/itmorn/p/7905975.html
朴素贝叶斯代码(Spark Python) |
代码里数据:https://pan.baidu.com/s/1jHWKG4I 密码:acq1
# -*-coding=utf-8 -*- from pyspark import SparkConf, SparkContext sc = SparkContext(\'local\') from pyspark.mllib.classification import NaiveBayes, NaiveBayesModel from pyspark.mllib.util import MLUtils # Load and parse the data file. data = MLUtils.loadLibSVMFile(sc, "data/mllib/sample_libsvm_data.txt") \'\'\' 每一行使用以下格式表示一个标记的稀疏特征向量 label index1:value1 index2:value2 ... tempFile.write(b"+1 1:1.0 3:2.0 5:3.0\\\\n-1\\\\n-1 2:4.0 4:5.0 6:6.0") >>> tempFile.flush() >>> examples = MLUtils.loadLibSVMFile(sc, tempFile.name).collect() >>> tempFile.close() >>> examples[0] LabeledPoint(1.0, (6,[0,2,4],[1.0,2.0,3.0])) >>> examples[1] LabeledPoint(-1.0, (6,[],[])) >>> examples[2] LabeledPoint(-1.0, (6,[1,3,5],[4.0,5.0,6.0])) \'\'\' # Split data approximately into training (60%) and test (40%) 将数据集按照6:4的比例分成训练集和测试集 training, test = data.randomSplit([0.6, 0.4]) # Train a naive Bayes model. 训练朴素贝叶斯模型 model = NaiveBayes.train(training, 1.0) # Make prediction and test accuracy. 预测和测试准确率 predictionAndLabel = test.map(lambda p: (model.predict(p.features), p.label)) accuracy = 1.0 * predictionAndLabel.filter(lambda pl: pl[0] == pl[1]).count() / test.count() print(\'model accuracy {}\'.format(accuracy)) #1 # Save and load model 保存和加载模型 output_dir = \'myNaiveBayesModel\' model.save(sc, output_dir) sameModel = NaiveBayesModel.load(sc, output_dir) predictionAndLabel = test.map(lambda p: (sameModel.predict(p.features), p.label)) accuracy = 1.0 * predictionAndLabel.filter(lambda pl: pl[0] == pl[1]).count() / test.count() print(\'sameModel accuracy {}\'.format(accuracy)) #1
以上是关于Spark MLlib速成宝典模型篇04朴素贝叶斯Naive Bayes(Python版)的主要内容,如果未能解决你的问题,请参考以下文章
Spark MLlib 源码学习---朴素贝叶斯模型(Naive Bayes)
Spark MLlib 源码学习---朴素贝叶斯模型(Naive Bayes)
Spark MLlib 源码学习---朴素贝叶斯模型(Naive Bayes)
Spark MLlib 源码学习---朴素贝叶斯模型(Naive Bayes)