解决Input contains NaN，infinity

Posted 2021-07-28 dlage

项目场景：

https://blog.csdn.net/zphuangtang/article/details/117262848

Input contains NaN, infinity or a value too large for dtype('float64').

昨天处理好的数据，今天进行实验的时候出错了

---

问题描述：

使用imblearn进行随机上采样时发生错误

from imblearn.over_sampling import RandomOverSampler
import pandas as pd
import os
ros = RandomOverSampler(random_state=0)
train_root = '/data/file/classification_data/2012-2019/sum_droped/'
# pandas DataFrame
train_data = pd.read_csv(os.path.join(train_root, 'train_sum_2017.csv'), index_col=0)
train_label = pd.read_csv(os.path.join(train_root, 'label_sum_2017.csv'), index_col=0)

train_data_re, train_label_re = ros.fit_resample(train_data, train_label)

出错在最后一句：

train_data_re, train_label_re = ros.fit_resample(train_data, train_label)

完整错误：

ValueError                                Traceback (most recent call last)<ipython-input-58-a2ca54a0cd7f> in <module>()
      1 
----> 2 train_data_re, train_label_re = ros.fit_resample(train_data, train_label)
      3 
~/anaconda3/envs/dataAna/lib/python3.6/site-packages/imblearn/base.py in fit_resample(self, X, y)
     73             The corresponding label of `X_resampled`.
     74         """
---> 75         check_classification_targets(y)
     76         arrays_transformer = ArraysTransformer(X, y)
     77         X, y, binarize_y = self._check_X_y(X, y)
~/anaconda3/envs/dataAna/lib/python3.6/site-packages/sklearn/utils/multiclass.py in check_classification_targets(y)
    178     y : array-like
    179     """
--> 180     y_type = type_of_target(y)
    181     if y_type not in ['binary', 'multiclass', 'multiclass-multioutput',
    182                       'multilabel-indicator', 'multilabel-sequences']:
~/anaconda3/envs/dataAna/lib/python3.6/site-packages/sklearn/utils/multiclass.py in type_of_target(y)
    301     if y.dtype.kind == 'f' and np.any(y != y.astype(int)):
    302         # [.1, .2, 3] or [[.1, .2, 3]] or [[1., .2]] and not [1., 2., 3.]
--> 303         _assert_all_finite(y)
    304         return 'continuous' + suffix
    305 
~/anaconda3/envs/dataAna/lib/python3.6/site-packages/sklearn/utils/validation.py in _assert_all_finite(X, allow_nan, msg_dtype)
    104                     msg_err.format
    105                     (type_err,
--> 106                      msg_dtype if msg_dtype is not None else X.dtype)
    107             )
    108     # for object dtype data, we only check for NaNs (GH-13254)
ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

---

原因分析：

随机上采样的数据类型对数据类型有要求，这里我去查看我的数据类型

print(train_data.dtypes)

0.1    float64
1      float64
2      float64
3      float64
4      float64
        ...   
443    float64
444    float64
445    float64
446    float64
447    float64

应该都是float64数据类型， 中间还有些看不到。

print((train_data.dtypes != 'float64').sum())

输出20

应该是中间有一些数据的类型不是float64类型。

现在要重新对数据进行排查

---

解决方案：

对数据处理时将数据类型修改一下