如何使用 sklearn Pipeline 和 FeatureUnion 选择多个（数字和文本）列进行文本分类？

Posted 2023-03-13

【中文标题】如何使用 sklearn Pipeline 和 FeatureUnion 选择多个（数字和文本）列进行文本分类？

【英文标题】：How to select multiple (numerical & text) columns using sklearn Pipeline & FeatureUnion for text classification?

【发布时间】：2018-12-21 23:54:33

【问题描述】：

我开发了一个用于多标签分类的文本模型。 OneVsRestClassifier LinearSVC 模型使用 sklearns Pipeline 和 FeatureUnion 进行模型准备。

主要输入特征包括一个名为 response 的文本列以及 5 个名为 t1_prob - t5_prob 的主题概率（从以前的 LDA 主题模型生成），以预测 5 个可能的标签。用于生成TfidfVectorizer 的管道中还有其他功能创建步骤。

我最终用ItemSelector 调用每一列，并分别在这些主题概率列上执行 ArrayCaster（请参阅下面的代码以了解函数定义）5 次。有没有更好的方法来使用FeatureUnion 来选择管道中的多个列？ （所以我不必做 5 次）

我想知道是否需要复制topic1_feature -topic5_feature 代码或者是否可以以更简洁的方式选择多个列？

我输入的数据是 Pandas 数据框：

id response label_1 label_2 label3  label_4 label_5     t1_prob t2_prob t3_prob t4_prob t5_prob
1   Text from response...   0.0 0.0 0.0 0.0 0.0 0.0     0.0625  0.0625  0.1875  0.0625  0.1250
2   Text to model with...   0.0 0.0 0.0 0.0 0.0 0.0     0.1333  0.1333  0.0667  0.0667  0.0667  
3   Text to work with ...   0.0 0.0 0.0 0.0 0.0 0.0     0.1111  0.0938  0.0393  0.0198  0.2759  
4   Free text comments ...  0.0 0.0 1.0 1.0 0.0 0.0     0.2162  0.1104  0.0341  0.0847  0.0559  

x_train 是response 和 5 个主题概率列（t1_prob、t2_prob、t3_prob、t4_prob、t5_prob）。

y_train 是 5 个 label 列，我将其称为 .values 以返回 DataFrame 的 numpy 表示。 (label_1, label_2, label3, label_4, label_5)

示例数据帧：

import pandas as pd
column_headers = ["id", "response", 
                  "label_1", "label_2", "label3", "label_4", "label_5",
                  "t1_prob", "t2_prob", "t3_prob", "t4_prob", "t5_prob"]

input_data = [
    [1, "Text from response",0.0,0.0,1.0,0.0,0.0,0.0625,0.0625,0.1875,0.0625,0.1250],
    [2, "Text to model with",0.0,0.0,0.0,0.0,0.0,0.1333,0.1333,0.0667,0.0667,0.0667],
    [3, "Text to work with",0.0,0.0,0.0,0.0,0.0,0.1111,0.0938,0.0393,0.0198,0.2759],
    [4, "Free text comments",0.0,0.0,1.0,1.0,1.0,0.2162,0.1104,0.0341,0.0847,0.0559]
    ]

df = pd.DataFrame(input_data, columns = column_headers)
df = df.set_index('id')
df

我认为我的实现有点绕，因为 FeatureUnion 在组合它们时只会处理二维数组，所以像 DataFrame 这样的任何其他类型对我来说都是有问题的。然而，这个例子是可行的——我只是在寻找改进它的方法，让它更干燥。

from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.base import BaseEstimator, TransformerMixin

class ItemSelector(BaseEstimator, TransformerMixin):
    def __init__(self, column):
        self.column = column

    def fit(self, X, y=None):
        return self

    def transform(self, X, y=None):
        return X[self.column]

class ArrayCaster(BaseEstimator, TransformerMixin):
    def fit(self, x, y=None):
        return self

    def transform(self, data):
        return np.transpose(np.matrix(data))


def basic_text_model(trainX, testX, trainY, testY, classLabels, plotPath):
    '''OneVsRestClassifier for multi-label prediction''' 
pipeline = Pipeline([
    ('features', FeatureUnion([
            ('topic1_feature', Pipeline([
                ('selector', ItemSelector(column='t1_prob')),
                ('caster', ArrayCaster())
            ])),
            ('topic2_feature', Pipeline([
                ('selector', ItemSelector(column='t2_prob')),
                ('caster', ArrayCaster())
            ])),
            ('topic3_feature', Pipeline([
                ('selector', ItemSelector(column='t3_prob')),
                ('caster', ArrayCaster())
            ])),
            ('topic4_feature', Pipeline([
                ('selector', ItemSelector(column='t4_prob')),
                ('caster', ArrayCaster())
            ])),
            ('topic5_feature', Pipeline([
                ('selector', ItemSelector(column='t5_prob')),
                ('caster', ArrayCaster())
            ])),
           ('word_features', Pipeline([
                    ('vect', CountVectorizer(analyzer="word", stop_words='english')), 
                    ('tfidf', TfidfTransformer(use_idf = True)),
            ])),
     ])),
    ('clf', OneVsRestClassifier(svm.LinearSVC(random_state=random_state))) 
])

# Fit the model
pipeline.fit(trainX, trainY)
predicted = pipeline.predict(testX)

我将 ArrayCaster 纳入流程源于此answer。

【参考方案1】：

我使用受@Marcus V 对此question 的解决方案启发的FunctionTransformer 找到了这个问题的答案。修改后的管道更加简洁。

from sklearn.preprocessing import FunctionTransformer

get_numeric_data = FunctionTransformer(lambda x: x[['t1_prob', 't2_prob', 't3_prob', 't4_prob', 't5_prob']], validate=False)

pipeline = Pipeline(
    [
        (
            "features",
            FeatureUnion(
                [
                    ("numeric_features", Pipeline([("selector", get_numeric_data)])),
                    (
                        "word_features",
                        Pipeline(
                            [
                                ("vect", CountVectorizer(analyzer="word", stop_words="english")),
                                ("tfidf", TfidfTransformer(use_idf=True)),
                            ]
                        ),
                    ),
                ]
            ),
        ),
        ("clf", OneVsRestClassifier(svm.LinearSVC(random_state=10))),
    ]
)

【讨论】：

考虑通过黑色运行格式化，除非我已经知道层次结构应该是什么，否则这有点难以理解

我已将结果重新格式化为黑色格式。谢谢你的建议。 :)

发现了这个列转换器 - scikit-learn.org/stable/auto_examples/compose/… - 另一种可能的解决方案