动手学pytorch-注意力机制和Seq2Seq模型
Posted 54hys
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了动手学pytorch-注意力机制和Seq2Seq模型相关的知识,希望对你有一定的参考价值。
注意力机制和Seq2Seq模型
1. 基本概念
Attention 是一种通用的带权池化方法,输入由两部分构成:询问(query)和键值对(key-value pairs)。(??_??∈?^{??_??}, ??_??∈?^{??_??}). Query (??∈?^{??_??}) , attention layer得到输出与value的维度一致 (??∈?^{??_??}). 对于一个query来说,attention layer 会与每一个key计算注意力分数并进行权重的归一化,输出的向量(o)则是value的加权求和,而每个key计算的权重与value一一对应。
为了计算输出,我们首先假设有一个函数(alpha) 用于计算query和key的相似性,然后可以计算所有的 attention scores (a_1, ldots, a_n) by
[ a_i = alpha(mathbf q, mathbf k_i). ]
我们使用 softmax函数 获得注意力权重:
[ b_1, ldots, b_n = extrm{softmax}(a_1, ldots, a_n). ]
最终的输出就是value的加权求和:
[ mathbf o = sum_{i=1}^n b_i mathbf v_i. ]
不同的attetion layer的区别在于score函数的选择,下面主要讨论两个常用的注意层 Dot-product Attention 和 Multilayer Perceptron Attention
Softmax屏蔽
在深入研究实现之前,首先介绍softmax操作符的一个屏蔽操作,主要目的是屏蔽无关信息。
def SequenceMask(X, X_len,value=-1e6):
maxlen = X.size(1)
#print(X.size(),torch.arange((maxlen),dtype=torch.float)[None, :],'
',X_len[:, None] )
mask = torch.arange((maxlen),dtype=torch.float)[None, :] >= X_len[:, None]
#print(mask)
X[mask]=value
return X
def masked_softmax(X, valid_length):
# X: 3-D tensor, valid_length: 1-D or 2-D tensor
softmax = nn.Softmax(dim=-1)
if valid_length is None:
return softmax(X)
else:
shape = X.shape
if valid_length.dim() == 1:
try:
valid_length = torch.FloatTensor(valid_length.numpy().repeat(shape[1], axis=0))#[2,2,3,3]
except:
valid_length = torch.FloatTensor(valid_length.cpu().numpy().repeat(shape[1], axis=0))#[2,2,3,3]
else:
valid_length = valid_length.reshape((-1,))
# fill masked elements with a large negative, whose exp is 0
X = SequenceMask(X.reshape((-1, shape[-1])), valid_length)
return softmax(X).reshape(shape)
masked_softmax(torch.rand((2,2,4),dtype=torch.float), torch.FloatTensor([2,3]))
tensor([[[0.4047, 0.5953, 0.0000, 0.0000],
[0.4454, 0.5546, 0.0000, 0.0000]],
[[0.3397, 0.3389, 0.3213, 0.0000],
[0.3526, 0.3318, 0.3156, 0.0000]]])
超出2维矩阵的乘法
(X) 和 (Y) 是维度分别为((b,n,m)) 和((b, m, k))的张量,进行 (b) 次二维矩阵乘法后得到 (Z), 维度为 ((b, n, k))。
[ Z[i,:,:] = dot(X[i,:,:], Y[i,:,:])qquad for i= 1,…,n . ]
torch.bmm(torch.ones((2,1,3), dtype = torch.float), torch.ones((2,3,2), dtype = torch.float))
tensor([[[3., 3.]],
[[3., 3.]]])
2. 两种常用的attention层
2.1点积注意力
The dot product 假设query和keys有相同的维度, 即 $forall i, ??,??_?? ∈ ?_?? $. 通过计算query和key转置的乘积来计算attention score,通常还会除去 (sqrt{d}) 减少计算出来的score对维度??的依赖性,如下
[ ??(??,??)=???,???/ sqrt{d} ]
假设 $ ??∈?^{??×??}$ 有 (m) 个query,(??∈?^{??×??}) 有 (n) 个keys. 我们可以通过矩阵运算的方式计算所有 (mn) 个score:
[ ??(??,??)=????^??/sqrt{d} ]
下面来实现这个层,它支持一批查询和键值对。此外,它支持作为正则化随机删除一些注意力权重.
# Save to the d2l package.
class DotProductAttention(nn.Module):
def __init__(self, dropout, **kwargs):
super(DotProductAttention, self).__init__(**kwargs)
self.dropout = nn.Dropout(dropout)
# query: (batch_size, #queries, d)
# key: (batch_size, #kv_pairs, d)
# value: (batch_size, #kv_pairs, dim_v)
# valid_length: either (batch_size, ) or (batch_size, xx)
def forward(self, query, key, value, valid_length=None):
d = query.shape[-1]
# set transpose_b=True to swap the last two dimensions of key
scores = torch.bmm(query, key.transpose(1,2)) / math.sqrt(d)
attention_weights = self.dropout(masked_softmax(scores, valid_length))
print("attention_weight
",attention_weights)
return torch.bmm(attention_weights, value)
测试
创建两个batch,每个batch有一个query和10个key-values对。通过valid_length指定,对于第一批,只关注前2个键-值对,而对于第二批,我们将检查前6个键-值对。尽管这两个批处理具有相同的查询和键值对,但获得的输出是不同的。
atten = DotProductAttention(dropout=0)
keys = torch.ones((2,10,2),dtype=torch.float)
values = torch.arange((40), dtype=torch.float).view(1,10,4).repeat(2,1,1)
print(values.shape)
atten(torch.ones((2,1,2),dtype=torch.float), keys, values, torch.FloatTensor([2, 6]))
torch.Size([2, 10, 4])
attention_weight
tensor([[[0.5000, 0.5000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
0.0000, 0.0000]],
[[0.1667, 0.1667, 0.1667, 0.1667, 0.1667, 0.1667, 0.0000, 0.0000,
0.0000, 0.0000]]])
tensor([[[ 2.0000, 3.0000, 4.0000, 5.0000]],
[[10.0000, 11.0000, 12.0000, 13.0000]]])
2.2 多层感知机注意力
在多层感知器中,我们首先将 query and keys 投影到 (?^?) .为了更具体,我们将可以学习的参数做如下映射
(??_??∈?^{?×??_??}) , (??_??∈?^{?×??_??}) , and (??∈?^h) . 将score函数定义
[
??(??,??)=??^??tanh(??_????+??_????)
]
.
然后将key 和 value 在特征的维度上合并(concatenate),然后送至 a single hidden layer perceptron 这层中 hidden layer 为 ? and 输出的size为 1 .隐层激活函数为tanh,无偏置.
# Save to the d2l package.
class MLPAttention(nn.Module):
def __init__(self, units,ipt_dim,dropout, **kwargs):
super(MLPAttention, self).__init__(**kwargs)
# Use flatten=True to keep query's and key's 3-D shapes.
self.W_k = nn.Linear(ipt_dim, units, bias=False)
self.W_q = nn.Linear(ipt_dim, units, bias=False)
self.v = nn.Linear(units, 1, bias=False)
self.dropout = nn.Dropout(dropout)
def forward(self, query, key, value, valid_length):
query, key = self.W_k(query), self.W_q(key)
#print("size",query.size(),key.size())
# expand query to (batch_size, #querys, 1, units), and key to
# (batch_size, 1, #kv_pairs, units). Then plus them with broadcast.
features = query.unsqueeze(2) + key.unsqueeze(1)
#print("features:",features.size()) #--------------开启
scores = self.v(features).squeeze(-1)
attention_weights = self.dropout(masked_softmax(scores, valid_length))
return torch.bmm(attention_weights, value)
测试
尽管MLPAttention包含一个额外的MLP模型,但如果给定相同的输入和相同的键,我们将获得与DotProductAttention相同的输出
atten = MLPAttention(ipt_dim=2,units = 8, dropout=0)
atten(torch.ones((2,1,2), dtype = torch.float), keys, values, torch.FloatTensor([2, 6]))
torch.Size([2, 1, 10])
tensor([[[ 2.0000, 3.0000, 4.0000, 5.0000]],
[[10.0000, 11.0000, 12.0000, 13.0000]]], grad_fn=<BmmBackward>)
3. 带注意力机制的Seq2seq模型
解码器
在解码的每个时间步,使用解码器的最后一个RNN层的输出作为注意层的query。然后,将注意力模型的输出与输入嵌入向量连接起来,输入到RNN层。虽然RNN层隐藏状态也包含来自解码器的历史信息,但是attention model的输出显式地选择了enc_valid_len以内的编码器输出,这样attention机制就会尽可能排除其他不相关的信息。
class Seq2SeqAttentionDecoder(d2l.Decoder):
def __init__(self, vocab_size, embed_size, num_hiddens, num_layers,
dropout=0, **kwargs):
super(Seq2SeqAttentionDecoder, self).__init__(**kwargs)
self.attention_cell = MLPAttention(num_hiddens,num_hiddens, dropout)
self.embedding = nn.Embedding(vocab_size, embed_size)
self.rnn = nn.LSTM(embed_size+ num_hiddens,num_hiddens, num_layers, dropout=dropout)
self.dense = nn.Linear(num_hiddens,vocab_size)
def init_state(self, enc_outputs, enc_valid_len, *args):
outputs, hidden_state = enc_outputs
# print("first:",outputs.size(),hidden_state[0].size(),hidden_state[1].size())
# Transpose outputs to (batch_size, seq_len, hidden_size)
return (outputs.permute(1,0,-1), hidden_state, enc_valid_len)
#outputs.swapaxes(0, 1)
def forward(self, X, state):
enc_outputs, hidden_state, enc_valid_len = state
#("X.size",X.size())
X = self.embedding(X).transpose(0,1)
# print("Xembeding.size2",X.size())
outputs = []
for l, x in enumerate(X):
# print(f"
{l}-th token")
# print("x.first.size()",x.size())
# query shape: (batch_size, 1, hidden_size)
# select hidden state of the last rnn layer as query
query = hidden_state[0][-1].unsqueeze(1) # np.expand_dims(hidden_state[0][-1], axis=1)
# context has same shape as query
# print("query enc_outputs, enc_outputs:
",query.size(), enc_outputs.size(), enc_outputs.size())
context = self.attention_cell(query, enc_outputs, enc_outputs, enc_valid_len)
# Concatenate on the feature dimension
# print("context.size:",context.size())
x = torch.cat((context, x.unsqueeze(1)), dim=-1)
# Reshape x to (1, batch_size, embed_size+hidden_size)
# print("rnn",x.size(), len(hidden_state))
out, hidden_state = self.rnn(x.transpose(0,1), hidden_state)
outputs.append(out)
outputs = self.dense(torch.cat(outputs, dim=0))
return outputs.transpose(0, 1), [enc_outputs, hidden_state,
enc_valid_len]
encoder = Seq2SeqEncoder(vocab_size=10, embed_size=8,
num_hiddens=16, num_layers=2)
# encoder.initialize()
decoder = Seq2SeqAttentionDecoder(vocab_size=10, embed_size=8,
num_hiddens=16, num_layers=2)
X = torch.zeros((4, 7),dtype=torch.long)
print("batch size=4
seq_length=7
hidden dim=16
num_layers=2
")
print('encoder output size:', encoder(X)[0].size())
print('encoder hidden size:', encoder(X)[1][0].size())
print('encoder memory size:', encoder(X)[1][1].size())
state = decoder.init_state(encoder(X), None)
out, state = decoder(X, state)
out.shape, len(state), state[0].shape, len(state[1]), state[1][0].shape
batch size=4
seq_length=7
hidden dim=16
num_layers=2
encoder output size: torch.Size([7, 4, 16])
encoder hidden size: torch.Size([2, 4, 16])
encoder memory size: torch.Size([2, 4, 16])
(torch.Size([4, 7, 10]), 3, torch.Size([4, 7, 16]), 2, torch.Size([2, 4, 16]))
4. 实验
以上是关于动手学pytorch-注意力机制和Seq2Seq模型的主要内容,如果未能解决你的问题,请参考以下文章
Attention使用注意力机制的seq2seq 动手学深度学习v2
Pytorch系列教程-使用Seq2Seq网络和注意力机制进行机器翻译
PyTorch-16 seq2seq translation 使用序列到序列的网络和注意机制进行翻译