LSTM 编码器-解码器推理模型
Posted
技术标签:
【中文标题】LSTM 编码器-解码器推理模型【英文标题】:LSTM Encoder-Decoder Inference Model 【发布时间】:2019-09-11 23:14:54 【问题描述】:很多基于 LSTM 的 seq2seq 编解码器架构教程(例如英法翻译),将模型定义如下:
encoder_inputs = Input(shape=(None,))
en_x= Embedding(num_encoder_tokens, embedding_size)(encoder_inputs)
# Encoder lstm
encoder = LSTM(50, return_state=True)
encoder_outputs, state_h, state_c = encoder(en_x)
# We discard `encoder_outputs` and only keep the states.
encoder_states = [state_h, state_c]
# Set up the decoder, using `encoder_states` as initial state.
decoder_inputs = Input(shape=(None,))
# french word embeddings
dex= Embedding(num_decoder_tokens, embedding_size)
final_dex= dex(decoder_inputs)
# decoder lstm
decoder_lstm = LSTM(50, return_sequences=True, return_state=True)
decoder_outputs, _, _ = decoder_lstm(final_dex,
initial_state=encoder_states)
decoder_dense = Dense(num_decoder_tokens, activation='softmax')
decoder_outputs = decoder_dense(decoder_outputs)
# While training, model takes eng and french words and outputs #translated french word
fullmodel = Model([encoder_inputs, decoder_inputs], decoder_outputs)
# rmsprop is preferred for nlp tasks
fullmodel.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['acc'])
fullmodel.fit([encoder_input_data, decoder_input_data], decoder_target_data,
batch_size=128,
epochs=100,
validation_split=0.20)
然后对于预测,他们定义推理模型如下:
# define the encoder model
encoder_model = Model(encoder_inputs, encoder_states)
encoder_model.summary()
# Redefine the decoder model with decoder will be getting below inputs from encoder while in prediction
decoder_state_input_h = Input(shape=(50,))
decoder_state_input_c = Input(shape=(50,))
decoder_states_inputs = [decoder_state_input_h, decoder_state_input_c]
final_dex2= dex(decoder_inputs)
decoder_outputs2, state_h2, state_c2 = decoder_lstm(final_dex2, initial_state=decoder_states_inputs)
decoder_states2 = [state_h2, state_c2]
decoder_outputs2 = decoder_dense(decoder_outputs2)
# sampling model will take encoder states and decoder_input(seed initially) and output the predictions(french word index) We dont care about decoder_states2
decoder_model = Model(
[decoder_inputs] + decoder_states_inputs,
[decoder_outputs2] + decoder_states2)
然后预测使用:
# Reverse-lookup token index to decode sequences back to
# something readable.
reverse_input_char_index = dict(
(i, char) for char, i in input_token_index.items())
reverse_target_char_index = dict(
(i, char) for char, i in target_token_index.items())
def decode_sequence(input_seq):
# Encode the input as state vectors.
states_value = encoder_model.predict(input_seq)
# Generate empty target sequence of length 1.
target_seq = np.zeros((1,1))
# Populate the first character of target sequence with the start character.
target_seq[0, 0] = target_token_index['START_']
# Sampling loop for a batch of sequences
# (to simplify, here we assume a batch of size 1).
stop_condition = False
decoded_sentence = ''
while not stop_condition:
output_tokens, h, c = decoder_model.predict(
[target_seq] + states_value)
# Sample a token
sampled_token_index = np.argmax(output_tokens[0, -1, :])
sampled_char = reverse_target_char_index[sampled_token_index]
decoded_sentence += ' '+sampled_char
# Exit condition: either hit max length
# or find stop character.
if (sampled_char == '_END' or
len(decoded_sentence) > 52):
stop_condition = True
# Update the target sequence (of length 1).
target_seq = np.zeros((1,1))
target_seq[0, 0] = sampled_token_index
# Update states
states_value = [h, c]
return decoded_sentence
我的问题是,他们训练了名为“fullmodel”的模型以获得最佳权重……在预测部分,他们使用了名称为 (encoder_model & decoder_model) 的推理模型……所以他们没有使用任何来自“完整模型”的权重?!
我不明白他们如何从经过训练的模型中受益!
【问题讨论】:
【参考方案1】:诀窍是一切都在同一个变量范围内,所以变量被重用了。
【讨论】:
我从磁盘恢复了一个训练有素的 LSTM 模型,然后尝试从中构建推理模型,但我就是做不到。这甚至可能吗?正如你所说,我不得不重新训练整个事情并重新使用训练中的变量。【参考方案2】:如果您仔细注意到,训练后的层权重会被重复使用。 例如,在创建decoder_model时,我们使用decoder_lstm层,它被定义为完整模型的一部分, decoder_outputs2, state_h2, state_c2 = decoder_lstm(final_dex2, initial_state=decoder_states_inputs),
编码器模型也使用之前定义的encoder_inputs 和encoder_states 层。 编码器模型=模型(编码器输入,编码器状态)
由于编码器-解码器模型的架构,我们需要执行这些实现技巧。 此外,正如 keras 文档所提到的,使用函数式 API,可以很容易地重用经过训练的模型:您可以通过在张量上调用任何模型来将其视为一个层。 请注意,通过调用模型,您不仅可以重用模型的架构,还可以重用其权重。更多详情请参考 - https://keras.io/getting-started/functional-api-guide/#all-models-are-callable-just-like-layers
【讨论】:
以上是关于LSTM 编码器-解码器推理模型的主要内容,如果未能解决你的问题,请参考以下文章