使用 fastT5 将 T5 模型导出到 onnx 时,得到“RuntimeError:形状 [5, 8, 1, 2] 的输出与广播形状 [5, 8, 2, 2] 不匹配”
Posted
技术标签:
【中文标题】使用 fastT5 将 T5 模型导出到 onnx 时,得到“RuntimeError:形状 [5, 8, 1, 2] 的输出与广播形状 [5, 8, 2, 2] 不匹配”【英文标题】:while exporting T5 model to onnx using fastT5 getting "RuntimeError:output with shape [5, 8, 1, 2] doesn't match the broadcast shape [5, 8, 2, 2]" 【发布时间】:2021-06-16 00:13:20 【问题描述】:我正在尝试使用 fastT5 库将 T5 模型转换为 onnx,但是 运行以下代码时出错
from fastT5 import export_and_get_onnx_model
from transformers import AutoTokenizer
model_name = 't5-small'
model = export_and_get_onnx_model(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
t_input = "translate English to French: The universe is a dark forest."
token = tokenizer(t_input, return_tensors='pt')
tokens = model.generate(input_ids=token['input_ids'],
attention_mask=token['attention_mask'],
num_beams=2)
output = tokenizer.decode(tokens.squeeze(), skip_special_tokens=True)
print(output)
错误:
/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py:244: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if causal_mask.shape[1] < attention_mask.shape[1]:
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-16-80094b7c4f6f> in <module>()
7 input_names=decoder_input_names,
8 output_names=decoder_output_names,
----> 9 dynamic_axes=dyn_axis_params,
10 )
24 frames
/usr/local/lib/python3.7/dist-packages/transformers/models/t5/modeling_t5.py in forward(self, hidden_states, mask, key_value_states, position_bias, past_key_value, layer_head_mask, query_length, use_cache, output_attentions)
497 position_bias = position_bias + mask # (batch_size, n_heads, seq_length, key_length)
498
--> 499 scores += position_bias
500 attn_weights = F.softmax(scores.float(), dim=-1).type_as(
501 scores
RuntimeError: output with shape [5, 8, 1, 2] doesn't match the broadcast shape [5, 8, 2, 2]
有人可以帮我解决这个问题吗? 谢谢。
【问题讨论】:
【参考方案1】:我检查了存储库,它看起来像这里报告的一个已知问题:https://github.com/Ki6an/fastT5/issues/1
该库的开发人员已在此处发布了解决方案并创建了一个笔记本文件:https://colab.research.google.com/drive/1HuH1Ui3pCBS22hW4djIOyUBP5UW93705?usp=sharing
解决办法是修改modeling_t5.py文件,第494行:
# Define this at line 426:
int_seq_length = int(seq_length)
# Change this at line 494:
position_bias = position_bias[:, :, -seq_length:, :]
position_bias = position_bias[:, :, -int_seq_length:, :] # Updated version
如果您不想自己修改文件,则需要等到this pull request 合并到 Transformers 库中。
【讨论】:
哦,我忘了检查它的问题,谢谢你的帮助。通常合并一个 PR 需要多长时间? 不太确定。这一切都取决于 Transformers 库的维护者是否要合并它、发布时间表、alpha/beta/stable 版本、将稳定的二进制文件上传到 pip/conda 等。自己更新文件可能会更容易和更快。以上是关于使用 fastT5 将 T5 模型导出到 onnx 时,得到“RuntimeError:形状 [5, 8, 1, 2] 的输出与广播形状 [5, 8, 2, 2] 不匹配”的主要内容,如果未能解决你的问题,请参考以下文章
PyTorch 1.0 中文官方教程:使用ONNX将模型从PyTorch传输到Caffe2和移动端