问：捕获管道变压器阵列的尺寸

Posted 2023-03-12

【中文标题】问：捕获管道变压器阵列的尺寸

【英文标题】：Q: Capture dimensions of transformer array for pipeline

【发布时间】：2018-04-26 20:51:29

【问题描述】：

我想使用keras sklearn wrapper 创建一个sklearn 管道。我正在尝试使用aclimdb, aka large movie dataset 进行情绪分类任务，我已将其转换为两列的熊猫数据框，一列用于评论（字符串），一列用于标签（整数）。

> df.head(4)
                                              review  sentiment
0  "Lifeforce" is a truly bizarre adaptation of t...          1
1  I ordered this movie on the Internet as it is ...          0
2  he was my hero for all time until he went alon...          0
3  This is a 'sleeper'. It defines Nicholas Cage....          1

我有一个管道，它使用CountVectorizer 对评论进行标记，使用TfidfTransformer 应用tfidf 转换，然后使用KerasClassifier 和下面的model 函数拟合二元分类模型：

X_train = df.loc[1:25000, "review"]
y_train = df.loc[1:25000, 'sentiment'].values
X_test = df.loc[25000:, "review"]
y_test = df.loc[25000:, 'sentiment'].values


np.random.seed(123) # for reproducibility

def model():
    model = models.Sequential([
        layers.Dense(16, input_shape = (10**4,), activation='relu'),
        layers.Dropout(0.5),
        layers.Dense(16, activation='relu'),
        layers.Dropout(0.5),
        layers.Dense(1, activation='sigmoid')
    ])
    model.compile(loss='binary_crossentropy', optimizer='rmsprop', 
                  metrics=['accuracy'])
    return model


early_stopping = callbacks.EarlyStopping(monitor='val_loss', patience=1, verbose=0, mode='auto')

pipe = pipeline.Pipeline([
    ('vect', CountVectorizer(max_features=10**4)),
    ('tfidf', TfidfTransformer()),
    ('nn', KerasClassifier(build_fn=model, 
                           nb_epoch=10, batch_size=128,
                           validation_split=0.2, callbacks=[early_stopping]))
])

为了完成这项工作，我必须为 keras 模型指定 input_shape，这意味着我必须修复 CountVectorizer 的 max_features 的值。我不想这样做。

有没有办法从前一个管道阶段获得输出的维度，在这种情况下，TfidfTransformer 并将其传递给KerasClassifier？即，像这样：

def model(input_df):
    model = models.Sequential([
        layers.Dense(16, input_shape = input_df.shape, activation='relu'),
        layers.Dropout(0.5),
        layers.Dense(16, activation='relu'),
        layers.Dropout(0.5),
        layers.Dense(1, activation='sigmoid')
    ])
    model.compile(loss='binary_crossentropy', optimizer='rmsprop', 
                  metrics=['accuracy'])
    return model
​
​
early_stopping = callbacks.EarlyStopping(monitor='val_loss', patience=1, verbose=0, mode='auto')
​
pipe = pipeline.Pipeline([
#    ('vect', CountVectorizer(max_features=10**4)),
#    ('tfidf', TfidfTransformer()),
    ('tfidf', TfidfVectorizer(max_features=10**4)),
    ('nn', KerasClassifier(build_fn=model(input_df=tfidf), 
                           nb_epoch=10, batch_size=128,
                           validation_split=0.2, callbacks=[early_stopping]))
])
​
## train network pipeline
​
pipe.fit(X_train.values, y_train)
​
-------------------------------------------------------------------
NameError                         Traceback (most recent call last)
<ipython-input-6-21be14eb185d> in <module>()
     19 #    ('tfidf', TfidfTransformer()),
     20     ('tfidf', TfidfVectorizer(max_features=10**4)),
---> 21     ('nn', KerasClassifier(build_fn=model(input_df=tfidf), 
     22                            nb_epoch=10, batch_size=128,
     23                            validation_split=0.2, callbacks=[early_stopping]))

NameError: name 'tfidf' is not defined

我可以将管道分成两个步骤，然后保存两个转换器的输出数据帧，这样我可以轻松捕获形状，但我宁愿一次性完成。

系统信息：

print(platform.platform())
print("Python", sys.version)
print("NumPy", np.__version__)
print("SciPy", scipy.__version__)
print("Scikit-Learn", sklearn.__version__)
print("Keras Backend", os.getenv("KERAS_BACKEND")) # doesn't work with tf https://github.com/fchollet/keras/issues/4984
​
Linux-4.4.0-91-generic-x86_64-with-debian-stretch-sid
Python 3.5.3 |Anaconda custom (64-bit)| (default, Mar  6 2017, 11:58:13) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
NumPy 1.13.3
SciPy 0.19.1
Scikit-Learn 0.19.0
Keras Backend cntk

谢谢！

【参考方案1】：

为了解决这个问题，你必须：

从模型中移除 input_shape

为 sklearnpipeline 定义一个自定义的 ArrayTransformer

在 tfidf/counter 和 keras 模型之间插入这个新的转换器

在您的代码中：

def model():
    model = models.Sequential([
       layers.Dense(16, activation='relu'),
       layers.Dropout(0.5),
       layers.Dense(16, activation='relu'),
       layers.Dropout(0.5),
       layers.Dense(1, activation='sigmoid')
    ])
    model.compile(loss='binary_crossentropy', optimizer='rmsprop', 
                  metrics=['accuracy'])
    return model


class ArrayTransformer():

   def transform(self, X, **transform_params):
       return X.toarray()

   def fit(self, X, y=None, **fit_params):
       return self


early_stopping = callbacks.EarlyStopping(monitor='val_loss', patience=1, verbose=0, 
mode='auto')

pipe = pipeline.Pipeline([
     ('tfidf', TfidfVectorizer(max_features=XXX)),
     ('transformer', ArrayTransformer()),
     ('nn', KerasClassifier(build_fn=model, 
                       nb_epoch=10, batch_size=128,
                       validation_split=0.2, callbacks=[early_stopping]))
   ])

pipe.fit(X_train.values, y_train)

通过这种方式，您还可以将 tfidf/counter 与 GridSearchCV 结合并调整 min_df、max_features...