对两列字符串数据执行一次热编码

Posted 2023-03-12

【中文标题】对两列字符串数据执行一次热编码

【英文标题】：Performing one hot encoding on two columns of string data

【发布时间】：2020-12-27 21:48:20

【问题描述】：

我正在尝试预测“Full_Time_Home_Goals”

我的代码是：

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import mean_absolute_error
from sklearn.ensemble import RandomForestRegressor
import os
import xlrd
import datetime
import numpy as np

# Set option to display all the rows and columns in the dataset. If there are more rows, adjust number accordingly.
pd.set_option('display.max_rows', 5000)
pd.set_option('display.max_columns', 500)
pd.set_option('display.width', 1000)

# Pandas needs you to define the column as date before its imported and then call the column and define as a date
# hence this step.
date_col = ['Date']
df = pd.read_csv(
    r'C:\Users\harsh\Documents\My Dream\Desktop\Machine Learning\Attempt1\Historical Data\Concat_Cleaned.csv'
    , parse_dates=date_col, skiprows=0, low_memory=False)

# Converting/defining the columns
# Before you define column types, you need to fill all NaN with a value. We will be reconverting them later
df = df.fillna(101)
# Defining column types
convert_dict = 'League_Division': str,
                'HomeTeam': str,
                'AwayTeam': str,
                'Full_Time_Home_Goals': int,
                'Full_Time_Away_Goals': int,
                'Full_Time_Result': str,
                'Half_Time_Home_Goals': int,
                'Half_Time_Away_Goals': int,
                'Half_Time_Result': str,
                'Attendance': int,
                'Referee': str,
                'Home_Team_Shots': int,
                'Away_Team_Shots': int,
                'Home_Team_Shots_on_Target': int,
                'Away_Team_Shots_on_Target': int,
                'Home_Team_Hit_Woodwork': int,
                'Away_Team_Hit_Woodwork': int,
                'Home_Team_Corners': int,
                'Away_Team_Corners': int,
                'Home_Team_Fouls': int,
                'Away_Team_Fouls': int,
                'Home_Offsides': int,
                'Away_Offsides': int,
                'Home_Team_Yellow_Cards': int,
                'Away_Team_Yellow_Cards': int,
                'Home_Team_Red_Cards': int,
                'Away_Team_Red_Cards': int,
                'Home_Team_Bookings_Points': float,
                'Away_Team_Bookings_Points': float,
                

df = df.astype(convert_dict)

# Reverting the replace values step to get original dataframe and with the defined filetypes
df = df.replace('101', np.NAN, regex=True)
df = df.replace(101, np.NAN, regex=True)

# Exploration
print(df.dtypes)
print(df)

# Clean dataset by dropping null rows
data = df.dropna(axis=0)

# Column that you want to predict = y
y = df.Full_Time_Home_Goals

# Columns that are inputted into the model to make predictions (dependants), Cannot be column y
features = ['HomeTeam', 'AwayTeam', 'Full_Time_Away_Goals', 'Full_Time_Result']
# Create X
X = df[features]

# Split into validation and training data
train_X, val_X, train_y, val_y = train_test_split(X, y, random_state=1)

# Specify Model
soccer_model = DecisionTreeRegressor(random_state=1)

# Fit Model
soccer_model.fit(train_X, train_y)

我遇到了与模型拟合的错误

# Fit Model
soccer_model.fit(train_X, train_y)

给我一​​个错误：

ValueError：无法将字符串转换为浮点数：“Nott'm Forest”

我该如何解决这个问题并运行模型以获得输出？我尝试了几个例子，但我无法进步。

您可以对示例 concat_cleaned 文件 here 进行罚款

【问题讨论】：

机器学习模型无法处理字符串数据。您将各种列指定为字符串。

请注意，在错误之后出现的任何代码都与问题无关（从未执行）并且应该不在此处发布，因为它只会造成不必要的混乱（已编辑）。

【参考方案1】：

您必须将分类数据转换为数值数据。为此，您可以使用 OneHotEncoder：

import os
import xlrd
import datetime
import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.preprocessing import OneHotEncoder

# Set option to display all the rows and columns in the dataset. If there are more rows, adjust number accordingly.
pd.set_option('display.max_rows', 5000)
pd.set_option('display.max_columns', 500)
pd.set_option('display.width', 1000)

# Pandas needs you to define the column as date before its imported and then call the column and define as a date
# hence this step.
date_col = ['Date']
df = pd.read_csv(
    r'Concat_Cleaned_Example.csv'
    , parse_dates=date_col, skiprows=0, low_memory=False)

# Converting/defining the columns
# Before you define column types, you need to fill all NaN with a value. We will be reconverting them later
df = df.fillna(101)
# Defining column types
convert_dict = 'League_Division': str,
                'HomeTeam': str,
                'AwayTeam': str,
                'Full_Time_Home_Goals': int,
                'Full_Time_Away_Goals': int,
                'Full_Time_Result': str,
                'Half_Time_Home_Goals': int,
                'Half_Time_Away_Goals': int,
                'Half_Time_Result': str,
                'Attendance': int,
                'Referee': str,
                'Home_Team_Shots': int,
                'Away_Team_Shots': int,
                'Home_Team_Shots_on_Target': int,
                'Away_Team_Shots_on_Target': int,
                'Home_Team_Hit_Woodwork': int,
                'Away_Team_Hit_Woodwork': int,
                'Home_Team_Corners': int,
                'Away_Team_Corners': int,
                'Home_Team_Fouls': int,
                'Away_Team_Fouls': int,
                'Home_Offsides': int,
                'Away_Offsides': int,
                'Home_Team_Yellow_Cards': int,
                'Away_Team_Yellow_Cards': int,
                'Home_Team_Red_Cards': int,
                'Away_Team_Red_Cards': int,
                'Home_Team_Bookings_Points': float,
                'Away_Team_Bookings_Points': float,
                

df = df.astype(convert_dict)

# Reverting the replace values step to get original dataframe and with the defined filetypes
df = df.replace('101', np.NAN, regex=True)
df = df.replace(101, np.NAN, regex=True)

# Clean dataset by dropping null rows
data = df.dropna(axis=0)

# Column that you want to predict = y
y = df.Full_Time_Home_Goals

# Columns that are inputted into the model to make predictions (dependants), Cannot be column y
features = ['HomeTeam', 'AwayTeam', 'Full_Time_Away_Goals', 'Full_Time_Result']
# Create X
X = df[features]

# Split into validation and training data
train_X, val_X, train_y, val_y = train_test_split(X, y, random_state=1)

# Specify Model
soccer_model = DecisionTreeRegressor(random_state=1)

# Define and train OneHotEncoder to transform numerical data to a numeric array
enc = OneHotEncoder(handle_unknown='ignore')
enc.fit(train_X)

transformed_train_X = enc.transform(train_X)

# Fit Model
soccer_model.fit(transformed_train_X, train_y)

这样您的数据，例如 (Man United,Newcastle,0,H) 将被编码为

(0, 14) 1.0
(0, 35) 1.0
(0, 43) 1.0
(0, 50) 1.0

您可以查看任何数据点以验证其是否正确编码，方法是：

entry_id = 1

print(transformed_train_X[entry_id])

for i in range(0,transformed_train_X[0].shape[1]):
  if(transformed_train_X[entry_id,i]==1.0):
    print(enc.get_feature_names()[i])

输出：

  (0, 14)   1.0
  (0, 35)   1.0
  (0, 43)   1.0
  (0, 50)   1.0
x0_Man United
x1_Newcastle
x2_0
x3_H

【讨论】：

那太好了，Kim，当我继续进行验证并计算 MAE 时，我收到一个字符串错误，将字符串转换为浮点数。 # Make validation predictions and calculate mean absolute error val_predictions = soccer_model.predict(val_X) val_mae = mean_absolute_error(val_predictions, val_y) print("Validation MAE when not specifying max_leaf_nodes : :,.0f".format(val_mae)) Error: ValueError: could not convert string to float: 'Wolves' 我要怎么做呢？

这是个新问题吧？您能否打开一个新问题并在此处添加带有错误消息的代码？它更容易阅读和理解，其他人也可以帮助您解决这个问题。