如何将 sklearn 管道转换为普通代码？

Posted 2023-02-23

【中文标题】如何将 sklearn 管道转换为普通代码？

【英文标题】：How to transform sklearn pipeline to normal code?

【发布时间】：2020-01-08 11:26:25

【问题描述】：

我有这个来自tutorial 的 sklearn 代码：

pipe = Pipeline([("cleaner", predictors()),
                 ('vectorizer', bow_vector),
                 ('classifier', classifier)])

我想把它转换成普通代码，像这样：

X_train = predictors.fit_transform(X_train)
X_train = bow_vector.fit_transform(X_train)
classifier.fit(X_train)

但我经常出错。快速阅读文档并没有帮助

更新

我的确切代码是

from sklearn.feature_extraction.text import CountVectorizer,TfidfVectorizer
df = pd.read_excel('data.xlsx')
from sklearn.model_selection import train_test_split

X = df['X'] 
ylabels = df['y']

X_train, X_test, y_train, y_test = train_test_split(X, ylabels, test_size=0.3, random_state=42)

标点符号列表

punctuations = string.punctuation

自然语言处理引擎

nlp = spacy.load('en')

停用词列表

stop_words = spacy.lang.en.stop_words.STOP_WORDS

加载英文分词器、标注器、解析器、NER 和词向量

parser = English()

分词器

def spacy_tokenizer(sentence):
    # Creating an token object
    mytokens = parser(sentence)

    # Lemmatizing each token and converting each token into lowercase
    mytokens = [ word.lemma_.lower().strip() if word.lemma_ != "-PRON-" else word.lower_ for word in mytokens ]

    # Removing stop words
    mytokens = [ word for word in mytokens if word not in stop_words and word not in punctuations ]

    # return preprocessed list of tokens
    return mytokens

管道的第一个元素

class predictors(TransformerMixin):
    def transform(self, X, **transform_params):
        # Cleaning Text
        return [clean_text(text) for text in X]

    def fit(self, X, y=None, **fit_params):
        return self

    def get_params(self, deep=True):
        return 

清除文本的基本功能

def clean_text(text):
    # Removing spaces and converting text into lowercase
    return text.strip().lower()

【问题讨论】：

嗯，有什么错误？你的确切代码是什么？我现在需要你的变量的数据类型

您应该能够在pipe 上调用所有这些方法，例如X_train = pipe.fit_transform(X_train)。如果您只想使用它，则无需“拆卸”管道。 sklearn.pipeline.Pipeline文档

相关：Python - What is exactly sklearn.pipeline.Pipeline?

我知道我可以一次完成管道中的所有操作，但我想深入挖掘并逐步了解 X_train 会发生什么，为此我想先执行第一个操作，然后查看X_train，然后进行第二次操作，以此类推

@sokolov0 的目标是发布minimal reproducible example，而不是整个代码

【参考方案1】：

我解决了我的问题。

tfidf_vector = TfidfVectorizer(tokenizer = spacy_tokenizer)

cleaner = predictors()

X_train_cleaned = cleaner.transform(X_train)

X_train_transformed = tfidf_vector.fit_transform(X_train_cleaned)

classifier = LogisticRegression(solver='lbfgs')

classifier.fit(X_train_transformed, y_train)

cleaner = predictors()

X_test_cleaned = cleaner.transform(X_test)

X_test_transformed = tfidf_vector.transform(X_test_cleaned)