错误 Azure Synapse Notebook：ImportError: tokenizers>=0.10.1,<0.11 是此模块正常运行所必需的，但发现 tokenizers==0

Posted 2023-03-25

【中文标题】错误 Azure Synapse Notebook：ImportError: tokenizers>=0.10.1,<0.11 是此模块正常运行所必需的，但发现 tokenizers==0.9.2

【英文标题】：Error Azure Synapse Notebook: ImportError: tokenizers>=0.10.1,<0.11 is required for a normal functioning of this module, but found tokenizers==0.9.2

【发布时间】：2021-11-20 19:31:19

【问题描述】：

我正在尝试在 Azure synapse Notebook 中执行以下命令。

from transformers import AutoTokenizer, AutoModelForMaskedLM 
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModelForMaskedLM.from_pretrained("bert-base-uncased")

但是，我收到以下错误。

ImportError: tokenizers>=0.10.1,<0.11 is required for a normal functioning of this module, but found tokenizers==0.9.2.
Try: pip install transformers -U or pip install -e '.[dev]' if you're working with git master
Traceback (most recent call last):

  File "/home/trusted-service-user/cluster-env/env/lib/python3.6/site-packages/transformers/__init__.py", line 43, in <module>
    from . import dependency_versions_check

  File "/home/trusted-service-user/cluster-env/env/lib/python3.6/site-packages/transformers/dependency_versions_check.py", line 41, in <module>
    require_version_core(deps[pkg])

  File "/home/trusted-service-user/cluster-env/env/lib/python3.6/site-packages/transformers/utils/versions.py", line 120, in require_version_core
    return require_version(requirement, hint)

  File "/home/trusted-service-user/cluster-env/env/lib/python3.6/site-packages/transformers/utils/versions.py", line 114, in require_version
    _compare_versions(op, got_ver, want_ver, requirement, pkg, hint)

  File "/home/trusted-service-user/cluster-env/env/lib/python3.6/site-packages/transformers/utils/versions.py", line 50, in _compare_versions
    f"requirement is required for a normal functioning of this module, but found pkg==got_ver.hint"

ImportError: tokenizers>=0.10.1,<0.11 is required for a normal functioning of this module, but found tokenizers==0.9.2.
Try: pip install transformers -U or pip install -e '.[dev]' if you're working with git master

即使我升级了transformer和tokenizer版本，上面的错误仍然存​​在，并通过以下代码验证：

import pkg_resources
for d in pkg_resources.working_set:
    print(d)

上述代码的示例输出：

transformers 4.10.3
tokenizers 0.10.3

有什么想法或建议吗？提前致谢。

【参考方案1】：

ImportError: tokenizers>=0.10.1,<0.11 is required for a normal functioning of this module, but found tokenizers==0.9.2

这个错误可能是因为你的环境有不同的版本，你可以试试这个强制安装：

pip install -I transformers --no-cache-dir --force-reinstall

您可以在Github 中参考类似的问题。

【讨论】：

我尝试通过上传“requirements.txt”文件来升级它。甚至确认了升级包的输出。但是它仍然给我这个错误。重申一下，我在 Azure Synapse Notebook 中而不是在本地桌面上执行此操作。提前感谢您的调查。