如何在管道中重新采样文本(不平衡组)?

Posted

技术标签:

【中文标题】如何在管道中重新采样文本(不平衡组)?【英文标题】:How to resample text (imbalanced groups) in a pipeline? 【发布时间】:2019-06-04 16:43:15 【问题描述】:

我正在尝试使用 MultinomialNB 进行一些文本分类,但我遇到了问题,因为我的数据不平衡。 (为简单起见,以下是一些示例数据。实际上,我的数据要大得多。)我正在尝试使用过采样对数据进行重新采样,理想情况下我希望将其构建到此管道中。

下面的管道在没有过度采样的情况下工作正常,但同样,在现实生活中我的数据需要它。这是非常不平衡的。

使用当前代码,我不断收到错误消息:“TypeError:所有中间步骤都应该是转换器并实现拟合和转换。”

如何将 RandomOverSampler 构建到此管道中?

data = [['round red fruit that is sweet','apple'],['long yellow fruit with a peel','banana'],
    ['round green fruit that is soft and sweet','pear'], ['red fruit that is common', 'apple'],
    ['tiny fruits that grow in bunches','grapes'],['purple fruits', 'grapes'], ['yellow and long', 'banana'],
    ['round, small, green', 'grapes'], ['can be red, green, or purple', 'grapes'], ['tiny fruits', 'grapes'], 
    ['small fruits', 'grapes']]

df = pd.DataFrame(data,columns=['Description','Type'])  

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 0)
text_clf = Pipeline([('vect', CountVectorizer()),
                    ('tfidf', TfidfTransformer()), 
                    ('RUS', RandomOverSampler()),
                    ('clf', MultinomialNB())])
text_clf = text_clf.fit(X_train, y_train)
y_pred = text_clf.predict(X_test)

print('Score:',text_clf.score(X_test, y_test))

【问题讨论】:

【参考方案1】:

您应该使用在imblearn 包中实现的管道,而不是来自sklearn 的管道。例如,这段代码运行良好:

import pandas as pd

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.naive_bayes import MultinomialNB

from imblearn.over_sampling import RandomOverSampler
from imblearn.pipeline import Pipeline


data = [['round red fruit that is sweet','apple'],['long yellow fruit with a peel','banana'],
    ['round green fruit that is soft and sweet','pear'], ['red fruit that is common', 'apple'],
    ['tiny fruits that grow in bunches','grapes'],['purple fruits', 'grapes'], ['yellow and long', 'banana'],
    ['round, small, green', 'grapes'], ['can be red, green, or purple', 'grapes'], ['tiny fruits', 'grapes'],
    ['small fruits', 'grapes']]

df = pd.DataFrame(data, columns=['Description','Type'])

X_train, X_test, y_train, y_test = train_test_split(df['Description'],
    df['Type'], random_state=0)

text_clf = Pipeline([('vect', CountVectorizer()),
                    ('tfidf', TfidfTransformer()),
                    ('RUS', RandomOverSampler()),
                    ('clf', MultinomialNB())])
text_clf = text_clf.fit(X_train, y_train)
y_pred = text_clf.predict(X_test)

print('Score:',text_clf.score(X_test, y_test))

【讨论】:

以上是关于如何在管道中重新采样文本(不平衡组)?的主要内容,如果未能解决你的问题,请参考以下文章

如何在不过度采样的情况下平衡数据集

[一起面试AI]NO.10 什么是数据不平衡问题,应该如何解决

使用 Imblearn 管道和 GridSearchCV 进行交叉验证

如何在交叉验证和 GridSearchCV 中实现 SMOTE

解决正负样本数据不平衡

如何对不平衡的多类数据集进行欠采样? (Python)