无效的语法错误：使用 Python 和 Spark 构建决策树、流失预测

Posted 2023-03-12

【中文标题】无效的语法错误：使用 Python 和 Spark 构建决策树、流失预测

【英文标题】：Invalid syntax error: Building decision tree with Python and Spark, Churn prediction

【发布时间】：2017-05-11 19:01:23

【问题描述】：

正如标题所示：我正在研究用于预测客户流失的决策树模型。我是数据科学、python 和 spark 的初学者。

结合讲座和在线文档中的所有示例，我设法提出了一个决策树模型。唯一的问题是，错误计算函数给了我一个语法错误。

基本上，我用于模型的数据如下所示：

[LabeledPoint(0.0, [1031.0,947.0,0.333333333333,10.9933333333,10.3,12.0,1.33333333333,10.0133333333,83.6666666667,5.86,55.69,0.596666666667,10.3333333333,0.666666666667,0.0,0.0,0.0,0.666666666667,23.3333333333,2.88333333333,25.0,0.666666666667,0.0,0.0,0.0,0.666666666667,135.333333333,4.44,0.06,0.333333333333,16.3333333333,0.98,0.333333333333,57.6666666667,3.46,0.333333333333,0.0,0.0,0.333333333333,14.0,0.0,0.0,0.0,0.0,0.0,0.0,1307.0,5.66666666667,22.0166666667,130.48,0.0,65.3333333333,0.0,287.333333333,34.0,113.666666667,0.0,0.0,0.0,1.0,1.0,0.0,1.0]),
LabeledPoint(0.0, [4231.0,951.0,1.33333333333,27.5466666667,6.45,22.0,1.0,12.0133333333,46.3333333333,6.45,47.15,1.32333333333,8.81,1.33333333333,0.0,0.0,0.0,1.33333333333,31.6666666667,6.4,42.6566666667,1.33333333333,0.0,0.0,0.0,1.33333333333,0.666666666667,0.0,0.0,57.0,0.0,0.0,57.0,0.0,0.0,57.0,0.0,0.0,57.0,10.6666666667,0.0,0.0,0.0,0.0,0.0,0.0,1307.0,4.0,32.0266666667,156.966666667,0.0,145.43,0.0,1.66666666667,0.0,0.333333333333,0.0,0.0,0.0,1.0,1.0,0.0,1.0]),
LabeledPoint(0.0, [5231.0,523.0,0.666666666667,14.62,1.1,1307.0,0.0,0.0,14.3333333333,1.1,7.57333333333,0.726666666667,4.84,0.666666666667,0.0,0.0,0.0,0.666666666667,8.33333333333,0.323333333333,2.15666666667,0.666666666667,0.0,0.0,0.0,0.666666666667,0.0,0.0,0.0,1307.0,0.0,0.0,1307.0,0.0,0.0,1307.0,0.0,0.0,1307.0,8.33333333333,0.0,0.0,0.0,0.0,0.0,0.0,1307.0,0.0,0.0,47.33,0.0,10.3566666667,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0,0.0]),
LabeledPoint(0.0, [6031.0,741.0,7.0,5.38666666667,2.13,58.0,0.333333333333,4.0,21.3333333333,1.35333333333,11.2966666667,0.48,3.2,8.33333333333,0.666666666667,0.0,0.0,8.33333333333,11.3333333333,0.453333333333,3.03,8.33333333333,1.0,0.0133333333333,0.166666666667,8.33333333333,2.33333333333,0.776666666667,0.363333333333,23.0,1.33333333333,0.08,23.0,0.0,0.0,23.0,0.333333333333,0.03,23.0,9.33333333333,0.666666666667,1.33333333333,0.0,0.0,0.0,0.0,1307.0,1.33333333333,16.0,61.25,3.31666666667,10.94,3.65,11.3333333333,7.0,0.0,1.0,0.0,0.0,1.0,1.0,0.0,1.0]),
LabeledPoint(0.0, [8831.0,840.0,5.33333333333,2.21,2.76,35.6666666667,0.666666666667,4.0,66.3333333333,2.76,17.7466666667,0.283333333333,1.20666666667,5.33333333333,0.0,0.0,0.0,5.33333333333,42.6666666667,2.43333333333,16.2166666667,5.33333333333,1.0,0.0,0.0,5.33333333333,1.0,0.0,0.0,23.0,0.0,0.0,23.0,0.666666666667,0.0,23.0,0.0,0.0,23.0,6.33333333333,0.0,1.33333333333,0.0,0.0,0.0,0.0,1307.0,1.66666666667,10.0,62.6333333333,0.0,56.7833333333,0.0,4.33333333333,0.666666666667,2.0,0.0,0.0,0.0,1.0,1.0,0.0,1.0])]

然后我将 Spark 文档中提供的代码用于决策树：

# Split the data into training and test sets (30% held out for testing)
(trainingData, testData) = data.randomSplit([0.7, 0.3])

# Train a DecisionTree model.
#  Empty categoricalFeaturesInfo indicates all features are continuous.
model = DecisionTree.trainClassifier(trainingData, numClasses=2, categoricalFeaturesInfo=,
                                 impurity='gini', maxDepth=5, maxBins=32)

# Evaluate model on test instances and compute test error
predictions = model.predict(testData.map(lambda x: x.features))
labelsAndPredictions = testData.map(lambda lp: lp.label).zip(predictions)
testErr = labelsAndPredictions.filter(lambda (v, p): v != p).count() / float(testData.count())
print('Test Error = ' + str(testErr))
print('Learned classification tree model:')
print(model.toDebugString())

还有

testErr = labelsAndPredictions.filter(lambda (v, p): v != p).count() / float(testData.count())

给我的错误是：

  File "<ipython-input-70-e37b435ea51d>", line 1
testErr = labelsAndPredictions.filter(lambda (v, p): v != p).count() / float(testData.count())
                                             ^
SyntaxError: invalid syntax

如果有人想查看整个代码，我把它放在here in my Dropbox

我不知道，为什么它会给我这个错误。似乎这条线对其他人来说效果很好。所以我担心它可能与模型创建之前的步骤有关..

非常感谢您的帮助！

【问题讨论】：

labelsAndPredictions = ...; testErr = ... 没有出现语法错误。

【参考方案1】：

lambda 表达式在有多个参数时不需要括号，所以 lambda (v, p): 应该是 lambda v, p:

lambda (x,y): 是仅在 Python 2 中有效的语法。在 Python 3 中，它是 SyntaxError。

【讨论】：

a = lambda (x,v): x*v print a((2,2)) 返回 4

但无论如何你已经澄清了你的观点。 when having multiple arguments

@MYGz 仅在 Python 2 中。在 Python 3 中是 SyntaxError。

哦。我在python2.7

【参考方案2】：

lambda (v, p) 仅适用于 Python 2.7 及以下版本的有效 Python 语法。您可能正在使用 Python 3，其中tuple parameter packing is no longer allowed。

我相信兼容 3.X 的版本应该是这样的：

testErr = labelsAndPredictions.filter(lambda seq: seq[0] != seq[1]).count() / float(testData.count())

【讨论】：

是的，这行得通，我很惊讶这与我的数据技巧无关。谢谢你，节日快乐！ :)