评分“roc_auc”值不适用于gridsearchCV应用RandomForestclassifer

Posted 2023-03-12

【中文标题】评分“roc_auc”值不适用于gridsearchCV应用RandomForestclassifer

【英文标题】：scoring "roc_auc" value is not working with gridsearchCV appling RandomForestclassifer

【发布时间】：2018-10-22 09:25:24

【问题描述】：

使用 gridsearchCV 执行此操作时，我不断收到此错误，评分值为 'roc_auc'('f1', 'precision','recall' 工作正常)

# Construct a pipeline
pipe = Pipeline([
('reduce_dim',PCA()),
('rf',RandomForestClassifier(min_samples_leaf=5,random_state=123))
])

N_FEATURES_OPTIONS = [2]  # for PCA [2, 4, 8]

# these below param is for RandomForestClassifier
N_ESTIMATORS = [10,50]  # 10,50,100
MAX_DEPTH = [5,6]  # 5,6,7,8,9
MIN_SAMPLE_LEAF = 5

param_grid = [
    
        'reduce_dim': [PCA()],
        'reduce_dim__n_components': N_FEATURES_OPTIONS,
        'rf__n_estimators' : N_ESTIMATORS,
        'rf__max_depth': MAX_DEPTH
    ,
    
        'reduce_dim': [SelectKBest(f_classif)],
        'reduce_dim__k': N_FEATURES_OPTIONS,
        'rf__n_estimators' : N_ESTIMATORS,
        'rf__max_depth': MAX_DEPTH
    ,
]

grid = GridSearchCV(pipe, param_grid= param_grid, cv =10,n_jobs=1,scoring = 'roc_auc')
grid.fit(X_train_s,y_train_s)

我得到了这个错误

AttributeError                            Traceback (most recent call last)
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/metrics/scorer.py in __call__(self, clf, X, y, sample_weight)
    186             try:
--> 187                 y_pred = clf.decision_function(X)
    188 

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/utils/metaestimators.py in __get__(self, obj, type)
    108                 else:
--> 109                     getattr(delegate, self.attribute_name)
    110                     break

AttributeError: 'RandomForestClassifier' object has no attribute 'decision_function'

During handling of the above exception, another exception occurred:

IndexError                                Traceback (most recent call last)
<ipython-input-16-86491f3b6aa7> in <module>()
----> 1 grid.fit(X_train_s,y_train_s)

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/model_selection/_search.py in fit(self, X, y, groups, **fit_params)
    637                                   error_score=self.error_score)
    638           for parameters, (train, test) in product(candidate_params,
--> 639                                                    cv.split(X, y, groups)))
    640 
    641         # if one choose to see train score, "out" will contain train score info

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py in __call__(self, iterable)
    777             # was dispatched. In particular this covers the edge
    778             # case of Parallel used with an exhausted iterator.
--> 779             while self.dispatch_one_batch(iterator):
    780                 self._iterating = True
    781             else:

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py in dispatch_one_batch(self, iterator)
    623                 return False
    624             else:
--> 625                 self._dispatch(tasks)
    626                 return True
    627 

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py in _dispatch(self, batch)
    586         dispatch_timestamp = time.time()
    587         cb = BatchCompletionCallBack(dispatch_timestamp, len(batch), self)
--> 588         job = self._backend.apply_async(batch, callback=cb)
    589         self._jobs.append(job)
    590 

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/externals/joblib/_parallel_backends.py in apply_async(self, func, callback)
    109     def apply_async(self, func, callback=None):
    110         """Schedule a func to be run"""
--> 111         result = ImmediateResult(func)
    112         if callback:
    113             callback(result)

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/externals/joblib/_parallel_backends.py in __init__(self, batch)
    330         # Don't delay the application, to avoid keeping the input
    331         # arguments in memory
--> 332         self.results = batch()
    333 
    334     def get(self):

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py in __call__(self)
    129 
    130     def __call__(self):
--> 131         return [func(*args, **kwargs) for func, args, kwargs in self.items]
    132 
    133     def __len__(self):

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py in <listcomp>(.0)
    129 
    130     def __call__(self):
--> 131         return [func(*args, **kwargs) for func, args, kwargs in self.items]
    132 
    133     def __len__(self):

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/model_selection/_validation.py in _fit_and_score(estimator, X, y, scorer, train, test, verbose, parameters, fit_params, return_train_score, return_parameters, return_n_test_samples, return_times, error_score)
    486         fit_time = time.time() - start_time
    487         # _score will return dict if is_multimetric is True
--> 488         test_scores = _score(estimator, X_test, y_test, scorer, is_multimetric)
    489         score_time = time.time() - start_time - fit_time
    490         if return_train_score:

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/model_selection/_validation.py in _score(estimator, X_test, y_test, scorer, is_multimetric)
    521     """
    522     if is_multimetric:
--> 523         return _multimetric_score(estimator, X_test, y_test, scorer)
    524     else:
    525         if y_test is None:

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/model_selection/_validation.py in _multimetric_score(estimator, X_test, y_test, scorers)
    551             score = scorer(estimator, X_test)
    552         else:
--> 553             score = scorer(estimator, X_test, y_test)
    554 
    555         if hasattr(score, 'item'):

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/sklearn/metrics/scorer.py in __call__(self, clf, X, y, sample_weight)
    195 
    196                 if y_type == "binary":
--> 197                     y_pred = y_pred[:, 1]
    198                 elif isinstance(y_pred, list):
    199                     y_pred = np.vstack([p[:, -1] for p in y_pred]).T

IndexError: index 1 is out of bounds for axis 1 with size 1

我已经查找了这个错误，并在这里发现了一些与 Kerasclassifier 类似的问题。但我不知道如何解决它

Keras Wrappers for Scikit Learn - AUC scorer is not working

谁能给我解释一下怎么回事？？？

【问题讨论】：

我的示例X_train_s,y_train_s 的代码没有出现任何错误。确保您拥有最新版本的 scikit-learn (0.19.1)，并显示代码中的导入语句。您是否收到任何弃用警告？

使用 sklearn 版本“0.18.2”，代码可以正常工作。使用：import sklearn，然后sklearn.__version__ 查看版本

我使用的版本是 0.19.1，它是最新的。 'RandomForestClassifier' 对象没有属性 'decision_function' 对我来说没有意义，因为我使用评分 = 'precision' 'recall' 运行这段代码很好。

请贴出完整的代码和一些示例数据来重现错误。

【参考方案1】：

错误可能是因为某些原因：

如果您只有一个目标类：它会失败 如果您有 >=3 个目标类：如果失败。 也许您有 2 个类，在 CV 的一个折叠中，测试标签仅来自一个类。

sklearn 计算 AUC 指标时，它必须有 2 个类，因为获取 AUC 的方法只需要两个类（计算所有阈值的 tpr 和 fpr）。 错误示例：

grid.fit(np.random.rand(100,2), np.random.randint(1, size=100)) #one class labels
grid.fit(np.random.rand(100,2), np.random.randint(3, size=100)) #3 class labels
#BOTH Throws same error when computing AUC

不应出错但可能发生的示例取决于 CV 的折叠：

grid.fit(np.random.rand(100,2), np.random.randint(2, size=100)) #two class labels
#This shouldnt throw an error 

解决方案

如果您有 2 个以上的类：您必须手动计算（或者可能有一些库，但我不知道），1 vs all，其中您计算 auc 有 2 个类（一个类 vs 所有其他），或 All vs All AUC（成对 AUC，您计算一个类，而 ALL 是一次一个类的单个类，然后计算平均值）。 如果您有 2 个类： grid = GridSearchCV(pipe, param_grid= param_grid, cv = StratifiedKFold(), n_jobs=1, scoring = 'roc_auc')