熊猫如何在时间序列数据上“get_dummies”

Posted 2023-03-12

【中文标题】熊猫如何在时间序列数据上“get_dummies”

【英文标题】：pandas how to 'get_dummies' on time series data

【发布时间】：2020-07-22 06:41:48

【问题描述】：

如果我有一些时间序列数据：（弥补一些）

import numpy as np
import pandas as pd
np.random.seed(11)

rows,cols = 50000,2
data = np.random.rand(rows,cols) 
tidx = pd.date_range('2019-01-01', periods=rows, freq='H') 
df = pd.DataFrame(data, columns=['Temperature','Value'], index=tidx)

如何使用get_dummies？光看Pandas documentation，不知道能不能申请我怎么做一个热编码。

例如，我知道如何制作一个虚拟变量来表示每周时间变量的唯一方法是一种非常笨重的冗余代码方法。有人可以给我建议如何更好地做到这一点吗？

#create dummy variables
df['month'] = df.index.month
df['year'] = df.index.year
df['day_of_week'] = df.index.dayofweek
df['hour'] = df.index.strftime('%H').astype('int')

df['hour_0'] = np.where(df['hour'].isin([0]), 1, 0)
df['hour_1'] = np.where(df['hour'].isin([1]), 1, 0)
df['hour_2'] = np.where(df['hour'].isin([2]), 1, 0)
df['hour_3'] = np.where(df['hour'].isin([3]), 1, 0)
df['hour_4'] = np.where(df['hour'].isin([4]), 1, 0)
df['hour_5'] = np.where(df['hour'].isin([5]), 1, 0)
df['hour_6'] = np.where(df['hour'].isin([6]), 1, 0)
df['hour_7'] = np.where(df['hour'].isin([7]), 1, 0)
df['hour_8'] = np.where(df['hour'].isin([8]), 1, 0)
df['hour_9'] = np.where(df['hour'].isin([9]), 1, 0)
df['hour_10'] = np.where(df['hour'].isin([10]), 1, 0)
df['hour_11'] = np.where(df['hour'].isin([11]), 1, 0)
df['hour_12'] = np.where(df['hour'].isin([12]), 1, 0)
df['hour_13'] = np.where(df['hour'].isin([13]), 1, 0)
df['hour_14'] = np.where(df['hour'].isin([14]), 1, 0)
df['hour_15'] = np.where(df['hour'].isin([15]), 1, 0)
df['hour_16'] = np.where(df['hour'].isin([16]), 1, 0)
df['hour_17'] = np.where(df['hour'].isin([17]), 1, 0)
df['hour_18'] = np.where(df['hour'].isin([18]), 1, 0)
df['hour_19'] = np.where(df['hour'].isin([19]), 1, 0)
df['hour_20'] = np.where(df['hour'].isin([20]), 1, 0)
df['hour_21'] = np.where(df['hour'].isin([21]), 1, 0)
df['hour_22'] = np.where(df['hour'].isin([22]), 1, 0)
df['hour_23'] = np.where(df['hour'].isin([23]), 1, 0)

df['monday'] = np.where(df['day_of_week'].isin([0]), 1, 0)
df['tuesday'] = np.where(df['day_of_week'].isin([1]), 1, 0)
df['wednesday'] = np.where(df['day_of_week'].isin([2]), 1, 0)
df['thursday'] = np.where(df['day_of_week'].isin([3]), 1, 0)
df['friday'] = np.where(df['day_of_week'].isin([4]), 1, 0)
df['saturday'] = np.where(df['day_of_week'].isin([5]), 1, 0)
df['sunday'] = np.where(df['day_of_week'].isin([6]), 1, 0)

df['january'] = np.where(df['month'].isin([1]), 1, 0)
df['february'] = np.where(df['month'].isin([2]), 1, 0)
df['march'] = np.where(df['month'].isin([3]), 1, 0)
df['april'] = np.where(df['month'].isin([4]), 1, 0)
df['may'] = np.where(df['month'].isin([5]), 1, 0)
df['june'] = np.where(df['month'].isin([6]), 1, 0)
df['july'] = np.where(df['month'].isin([7]), 1, 0)
df['august'] = np.where(df['month'].isin([8]), 1, 0)
df['september'] = np.where(df['month'].isin([9]), 1, 0)
df['october'] = np.where(df['month'].isin([10]), 1, 0)
df['november'] = np.where(df['month'].isin([11]), 1, 0)
df['december'] = np.where(df['month'].isin([12]), 1, 0)

df['year19'] = np.where(df['year'].isin([2019]), 1, 0)
df['year20'] = np.where(df['year'].isin([2020]), 1, 0)
df['year21'] = np.where(df['year'].isin([2021]), 1, 0)
df['year22'] = np.where(df['year'].isin([2022]), 1, 0)
df['year23'] = np.where(df['year'].isin([2023]), 1, 0)
df['year24'] = np.where(df['year'].isin([2024]), 1, 0)

然后我正在用 ML 算法试验的最终数据框将是：

df2 = df[['Temperature', 'Value', 
            'hour_0' , 'hour_1' , 'hour_2' , 'hour_3' , 'hour_4' , 'hour_5' , 'hour_6' ,
            'hour_7' , 'hour_8' , 'hour_9' , 'hour_10' , 'hour_11' , 'hour_12' , 'hour_13' , 
            'hour_14' , 'hour_15' , 'hour_16' , 'hour_17' , 'hour_18' , 'hour_19' , 'hour_20' , 
            'hour_21' , 'hour_22' , 'hour_23' , 
            'monday' , 'tuesday' , 'wednesday' , 'thursday' , 'friday' , 'saturday' , 'sunday' , 
            'january' , 'february' , 'march' , 'april' , 'may' , 'june' , 'july' , 'august' , 
            'september' , 'october' , 'november' , 'december' , 
            'year19' , 'year20' , 'year21' , 'year22' , 'year23' , 'year24']]

编辑 更新的代码尝试

import numpy as np
import pandas as pd
np.random.seed(11)

rows,cols = 50000,2
data = np.random.rand(rows,cols) 
tidx = pd.date_range('2019-01-01', periods=rows, freq='H') 
df = pd.DataFrame(data, columns=['Temperature','Value'], index=tidx)

df['hour'] = df.index.strftime('%H').astype('int')
df['day_of_week'] = df.index.dayofweek
df['month'] = df.index.month
df['year'] = df.index.year

hour_dummies = pd.get_dummies(df['hour'], prefix='hour')

day_mapping = 0: 'monday', 1: 'tuesday', 2: 'wednesday', 3: 'thursday', 4: 'friday', 5: 'saturday', 6: 'sunday'
day_dummies = pd.get_dummies(df['day_of_week'].map(day_mapping))

month_mapping = 0: 'jan', 1: 'feb', 2: 'mar', 3: 'apr', 4: 'may', 5: 'jun', 6: 'jul',
                 7: 'aug', 8: 'sep', 9: 'oct', 10: 'nov', 11: 'dec'
month_dummies = pd.get_dummies(df['month'].map(month_mapping))

year_mapping = 0: 'year_2019', 1: 'year_2020', 2: 'year_2021', 3: 'year_2022', 4: 'year_2023', 5: 'year_2024'
year_dummies = pd.get_dummies(df['year'].map(year_mapping))

df = df.join(hour_dummies)
df = df.join(day_dummies)
df = df.join(month_dummies)
df = df.join(year_dummies)

【参考方案1】：

您可以从时间索引中提取相应的信息，然后使用pd.get_dummies。例如

# day name
day_names = pd.get_dummies(df.index.day_name())

# hours
hours = pd.get_dummies(df.index.hour, prefix='hour')

# months
months = pd.get_dummies(df.index.month_name())

# year
years = pd.get_dummies(df.index.year, prefix='year')

然后concat：

df = pd.concat((df, hours, day_names), axis=1)

【讨论】：

当我运行代码并查看它的所有 NaN 时，由于某种原因时间戳是 1970

【参考方案2】：

您以问题的名义准确描述了解决方案：

>>> pd.get_dummies(df['hour'], prefix='hour')
                     hour_0  hour_1  hour_2  hour_3  hour_4  hour_5  hour_6  hour_7  hour_8  ...  hour_15  hour_16  hour_17  hour_18  hour_19  hour_20  hour_21  hour_22  hour_23
2019-01-01 00:00:00       1       0       0       0       0       0       0       0       0  ...        0        0        0        0        0        0        0        0        0
2019-01-01 01:00:00       0       1       0       0       0       0       0       0       0  ...        0        0        0        0        0        0        0        0        0
2019-01-01 02:00:00       0       0       1       0       0       0       0       0       0  ...        0        0        0        0        0        0        0        0        0
2019-01-01 03:00:00       0       0       0       1       0       0       0       0       0  ...        0        0        0        0        0        0        0        0        0
2019-01-01 04:00:00       0       0       0       0       1       0       0       0       0  ...        0        0        0        0        0        0        0        0        0
...                     ...     ...     ...     ...     ...     ...     ...     ...     ...  ...      ...      ...      ...      ...      ...      ...      ...      ...      ...
2024-09-14 03:00:00       0       0       0       1       0       0       0       0       0  ...        0        0        0        0        0        0        0        0        0
2024-09-14 04:00:00       0       0       0       0       1       0       0       0       0  ...        0        0        0        0        0        0        0        0        0
2024-09-14 05:00:00       0       0       0       0       0       1       0       0       0  ...        0        0        0        0        0        0        0        0        0
2024-09-14 06:00:00       0       0       0       0       0       0       1       0       0  ...        0        0        0        0        0        0        0        0        0
2024-09-14 07:00:00       0       0       0       0       0       0       0       1       0  ...        0        0        0        0        0        0        0        0        0

对于星期/月份，使用pandas.Series.map 将值替换为“可读”对应的值，然后调用pandas.get_dummies：

day_mapping = 0: 'monday', 1: 'tuesday', 2: 'wednesday', 3: 'thursday', 4: 'friday', 5: 'saturday', 6: 'sunday'
df = df.join(pd.get_dummies(df['day_of_week'].map(day_mapping)))

【讨论】：

我为我的答案创建了一个编辑部分，试图让代码工作。它看起来仍然像一些冗余编码......而且我也无法让year_dummies 正常工作，你会吗还能给我一些提示吗？

@HenryHub 在年份的情况下，您可以使用与小时相同的方法，但使用另一个前缀而不是映射，它应该可以按照您的意愿工作。那么对于月份，'jan' 的值不是 0 而是 1，所以你应该给每个值加 1 来映射：1:'jan', 2: 'feb', etc..