sklearn管道适合:AttributeError:未找到下限
Posted
技术标签:
【中文标题】sklearn管道适合:AttributeError:未找到下限【英文标题】:sklearn pipeline fit: AttributeError: lower not found 【发布时间】:2017-03-24 20:43:46 【问题描述】:我想在 sklearn 中使用管道,如下所示:
corpus = load_files('corpus/train')
stop_words = [x for x in open('stopwords.txt', 'r').read().split('\n')] # Uppercase!
countvec = CountVectorizer(stop_words=stop_words, ngram_range=(1, 2))
X_train, X_test, y_train, y_test = train_test_split(corpus.data, corpus.target, test_size=0.9,
random_state=0)
x_train_counts = countvec.fit_transform(X_train)
x_test_counts = countvec.transform(X_test)
k_fold = KFold(n=len(corpus.data), n_folds=6)
confusion = np.array([[0, 0], [0, 0]])
pipeline = Pipeline([
('vectorizer', CountVectorizer(stop_words=stop_words, ngram_range=(1, 2))),
('classifier', MultinomialNB()) ])
for train_indices, test_indices in k_fold:
pipeline.fit(x_train_counts, y_train)
predictions = pipeline.predict(x_test_counts)
但是,我收到此错误:
AttributeError: lower not found
我看过这篇文章:
AttributeError: lower not found; using a Pipeline with a CountVectorizer in scikit-learn
但我将字节列表传递给矢量化器,所以这不应该是问题。
编辑
corpus = load_files('corpus')
stop_words = [x for x in open('stopwords.txt', 'r').read().split('\n')]
X_train, X_test, y_train, y_test = train_test_split(corpus.data, corpus.target, test_size=0.5,
random_state=0)
k_fold = KFold(n=len(corpus.data), n_folds=6)
confusion = np.array([[0, 0], [0, 0]])
pipeline = Pipeline([
('vectorizer', CountVectorizer(stop_words=stop_words, ngram_range=(1, 2))),
('classifier', MultinomialNB())])
for train_indices, test_indices in k_fold:
pipeline.fit(X_train[train_indices], y_train[train_indices])
predictions = pipeline.predict(X_test[test_indices])
现在我得到了错误:
TypeError: only integer arrays with one element can be converted to an index
第二次编辑
corpus = load_files('corpus')
stop_words = [y for x in open('stopwords.txt', 'r').read().split('\n') for y in (x, x.title())]
k_fold = KFold(n=len(corpus.data), n_folds=6)
confusion = np.array([[0, 0], [0, 0]])
pipeline = Pipeline([
('vectorizer', CountVectorizer(stop_words=stop_words, ngram_range=(1, 2))),
('classifier', MultinomialNB())])
for train_indices, test_indices in k_fold:
pipeline.fit(corpus.data, corpus.target)
【问题讨论】:
等等,你为什么要进行训练,测试拆分然后kfold?当你没有足够的测试时,通常使用折叠。此外,您正在折叠 corpus.data,这可能与您的 X_train 不同,后者将缩短 50%。 @Ale:好吧,我想我误解了 kfold。我认为这是一种打乱测试和训练数据的方法。第二次编辑没有抛出错误,但我现在如何使用 predict() 函数? 那是您在 kfold 中使用索引来提取训练和测试数据的地方,请参阅我编辑的答案。或者不要使用 kfold 并在 X_train 上进行训练并在 X_test 上进行测试 @Ale 非常感谢!!我想我现在明白了:D 【参考方案1】:您没有正确使用管道。你不需要传递向量化的数据,想法是管道对数据进行向量化。
# This is done by the pipeline
# x_train_counts = countvec.fit_transform(X_train)
# x_test_counts = countvec.transform(X_test)
k_fold = KFold(n=len(corpus.data), n_folds=6)
confusion = np.array([[0, 0], [0, 0]])
pipeline = Pipeline([
('vectorizer', CountVectorizer(stop_words=stop_words, ngram_range=(1, 2))),
('classifier', MultinomialNB()) ])
# also you are not using the indices...
for train_indices, test_indices in k_fold:
pipeline.fit(corpus.data[train_indices], corpus.target[train_indices])
predictions = pipeline.predict(corpus.data[test_indices])
【讨论】:
感谢您的回复!我更改了代码,但现在出现了新错误!你能再看看吗? @user3813234 应该对 corpus.data 和 corpus.target 进行折叠以上是关于sklearn管道适合:AttributeError:未找到下限的主要内容,如果未能解决你的问题,请参考以下文章
Keras Sklearn Tuner 模块“sklearn”没有属性“管道”
sklearn:在 RandomizedSearchCV 中使用管道?
使用 KNeighborsClassifier 的 SKlearn 管道