时间序列数据中 LSTM 训练测试拆分中的问题
Posted
技术标签:
【中文标题】时间序列数据中 LSTM 训练测试拆分中的问题【英文标题】：Problem in LSTM train-test split in time series data 【发布时间】：2022-01-21 18:45:36 【问题描述】：我正在尝试使用我的 csv 文件制作训练集和测试集来训练 LSTM。 csv 文件如下所示：
datetime invno inkw outkw Total wind_spd temp pres ts
2021-12-01 00:00:00 1 0.0 0.0 0.0 4.6 -0.9 1007.7 1638284400.0
2021-12-01 00:00:00 4 0.0 0.0 0.0, 4.6 -0.9 1007.7 1638284400.0
2021-12-01 00:00:00 2 0.0 0.0 0.0, 4.6 -0.9 1007.7 1638284400.0
2021-12-01 00:00:00 3 0.0 0.0 0.0, 4.6 -0.9 1007.7 1638284400.0
2021-12-01 00:00:00 5 0.0 0.0 0.0, 4.6 -0.9 1007.7 1638284400.0
2021-12-01 01:00:00 1 0.0 0.0 0.0, 9.8 -1.3 1007.7 1638288000.0
2021-12-01 01:00:00 4 0.0 0.0 0.0, 9.8 -1.3 1007.7 1638288000.0
.......... ........ . ... .... ... ... .... ... ......
.......... ........ . ... .... ... ... .... ... ......
2021-12-10 17:00:00 2 0.06735057830810548 0.087 23.9 2.3 -1.2 1007.6 163828800.0
2021-12-10 17:00:00 3 0.03403729248046875 0.091 24.1 2.3 -1.2 1007.6 163828800.0
2021-12-10 17:00:00 4 0.08401119232177734 0.09 24.3 2.3 -1.2 1007.6 163828800.0
2021-12-10 17:00:00 5 0.08356260681152344 0.087 24.6 2.3 -1.2 1007.6 163828800.0
制作训练集和测试集后的数据集形状：
(1170, 9)
Training shape: (930, 30, 8)
Testing shape: (185, 30, 8)
这是我的代码：
import os
import pandas as pd
import numpy as np
from sklearn.preprocessing import MinMaxScaler
#from sklearn.externals import joblib
import joblib
import seaborn as sns
sns.set(color_codes=True)
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
from numpy.random import seed
#from tensorflow import set_random_seed
import tensorflow
tensorflow.random.set_seed
import tensorflow as tf
#tf.logging.set_verbosity(tf.logging.ERROR)
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Input, Dropout, Dense, LSTM, TimeDistributed, RepeatVector
from tensorflow.keras.models import Model
from tensorflow.keras import regularizers
import plotly.graph_objects as go
dataset = pd.read_csv('./data/combined.csv')
print(dataset.shape)
dataset.fillna(0, inplace=True)
dataset = dataset.set_index('datetime')
train = dataset[:'2021-12-08 23:00:00']
test = dataset['2021-12-08 23:00:00':]
scaler = StandardScaler()
scaler = scaler.fit(train)
train = scaler.transform(train)
test = scaler.transform(test)
TIME_STEPS=30
def create_sequences(X, y, time_steps=TIME_STEPS):
Xs, ys = [], []
for i in range(len(X)-time_steps):
Xs.append(X.iloc[i:(i+time_steps)].values)
ys.append(y.iloc[i+time_steps])
return np.array(Xs), np.array(ys)
X_train, y_train = create_sequences(train, train)
X_test, y_test = create_sequences(test, test)
print(f'Training shape: X_train.shape')
print(f'Testing shape: X_test.shape')
model = Sequential()
model.add(LSTM(128, input_shape=(X_train.shape[1], X_train.shape[2])))
model.add(Dropout(rate=0.2))
model.add(RepeatVector(X_train.shape[1]))
model.add(LSTM(128, return_sequences=True))
model.add(Dropout(rate=0.2))
model.add(TimeDistributed(Dense(X_train.shape[2])))
model.compile(optimizer='adam', loss='mae')
model.summary()
history = model.fit(X_train, y_train, epochs=100, batch_size=16, validation_split=0.1 , shuffle=False)
每当我运行此代码时，都会出现以下错误：
Traceback (most recent call last):
File "/Users/sudip/Desktop/workspace/local_work/LSTM_api/test-1.py", line 58, in <module>
X_train, y_train = create_sequences(train, train)
File "/Users/sudip/Desktop/workspace/local_work/LSTM_api/test-1.py", line 53, in create_sequences
Xs.append(X.iloc[i:(i+time_steps)].values)
AttributeError: 'numpy.ndarray' object has no attribute 'iloc'
删除iloc 和values 后出现以下错误：
Epoch 1/100
Traceback (most recent call last):
File "/Users/sudip/Desktop/workspace/local_work/LSTM_api/test-1.py", line 77, in <module>
history = model.fit(X_train, y_train, epochs=100, batch_size=16, validation_split=0.1 , shuffle=False)
File "/Users/sudip/Desktop/workspace/env/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/Users/sudip/Desktop/workspace/env/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 58, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [16,30,8] vs. [16,8]
[[node gradient_tape/mean_absolute_error/sub/BroadcastGradientArgs
(defined at /Users/sudip/Desktop/workspace/env/lib/python3.9/site-packages/keras/optimizer_v2/optimizer_v2.py:464)
]] [Op:__inference_train_function_5593]
Errors may have originated from an input operation.
Input Source operations connected to node gradient_tape/mean_absolute_error/sub/BroadcastGradientArgs:
我认为错误来自输入形状。我可以得到一些帮助来解决这个问题吗？
如何根据日期和时间从时间序列数据中拆分训练和测试？
【问题讨论】：
StandardScaler变换方法的输出很可能是一个numpy ndarray 可能是。还有其他选择可以摆脱这个问题吗？如果我在时间序列数据中有多个特征，通常我会遇到这个问题。还是想不通。 数据形状有问题。你的网络的输入形状和输出形状是一样的，但是 X_train 和 y_train 的形状不一样 是的，我意识到了。有什么想法可以解决吗？这将是一个很大的帮助。谢谢 【参考方案1】：数据形状有问题。您的网络的输入形状和输出形状相同，但 X_train 和 y_train 的形状不同。 一个可以完成这项工作的简单模型：
model = Sequential()
model.add(LSTM(128, input_shape=(X_train.shape[1], X_train.shape[2])))
model.add(Dense(y_train.shape[1]))
model.compile(optimizer='adam', loss='mae')
model.summary()
【讨论】：
嗨易北河，所以删除 iloc 和值不会影响对吧？如果我包含 iloc，则会引发错误。正如我们之前所讨论的，transform 方法已经为我们提供了 np 数组。 我用了你的想法，训练顺利，如果我必须添加更多层怎么办？我必须在每一层之后添加 y_train 行吗？ 你可以玩模型，添加额外的层，看看它是否能提高性能。 y_train 线只在最后一层需要，保证输出的形状和y一样 非常感谢您的回答和时间。我还有一个问题。当我尝试使用“model.add(RepeatVector(X_train.shape[1]))”时，它再次引发形状问题。它不应该在所有情况下都有效吗？以上是关于时间序列数据中 LSTM 训练测试拆分中的问题的主要内容，如果未能解决你的问题，请参考以下文章