pyinstaller打包torch出现OSError: could not get source codepkg_resources.DistributionNotFound

Posted 2022-02-08 Icy Hunter

记下主要怕以后自己忘记

首先打包先建一个虚拟环境，就装上打包的python需要的包，这样打包起来错误可能会少点，而且速度也会快些，大小也可能小些，如何创建可以参考我的另一篇博客https://blog.csdn.net/qq_52785473/article/details/122543935?spm=1001.2014.3001.5501

如何导包的时候，也尽量用什么导什么
例如需要用pandas的读文件功能
from pandas import read_csv即可
这样可能对打包的性能也有所提升
打包-D来打包时，程序运行速度上也会快一些

下面是我遇到的一些错误及解决方法：
一：
出现pkg_resources.DistributionNotFound：the “tqdm>=4.27” distribution was not found之类的错误时

[8144] WARNING: file already exists but should not: …这个错误可能也适用

在需要打包的python文件的同目录下创建一个hook-ctypes.macholib.py文件里面输入内容：

# -*- coding: utf-8 -*-

from PyInstaller.utils.hooks import copy_metadata

datas = copy_metadata('tqdm') + copy_metadata('regex') + copy_metadata("sacremoses") + copy_metadata("requests") + \\
        copy_metadata("packaging") + copy_metadata("filelock") + copy_metadata("numpy") + copy_metadata("tokenizers")  # 缺啥补啥（这是我自己缺）

然后终端打包的时候输入

pyinstaller -D (需要打包的python文件名).py --additional-hooks-dir=.

即可。

二：
Traceback (most recent call last):
File “torch_sources.py”, line 21, in get_source_lines_and_file
sourcelines, file_lineno = inspect.getsourcelines(obj)
File “inspect.py”, line 955, in getsourcelines
File “inspect.py”, line 786, in findsource
OSError: could not get source code

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “p.py”, line 2, in
File “frozen importlib._bootstrap>”, line 983, in _find_and_load
File “frozen importlib._bootstrap>”, line 967, in _find_and_load_unlocked
File “frozen importlib._bootstrap>”, line 677, in load_unlocked
File “PyInstaller\\loader\\pyimod03_importers.py”, line 495, in exec_module
File "ltp_init.py", line 8, in
File “frozen importlib._bootstrap>”, line 983, in _find_and_load
File “frozen importlib._bootstrap>”, line 967, in _find_and_load_unlocked
File “frozen importlib._bootstrap>”, line 677, in load_unlocked
File “PyInstaller\\loader\\pyimod03_importers.py”, line 495, in exec_module
File "ltp\\utils_init.py", line 11, in
File “frozen importlib._bootstrap>”, line 983, in _find_and_load
File “frozen importlib._bootstrap>”, line 967, in _find_and_load_unlocked
File “frozen importlib._bootstrap>”, line 677, in _load_unlocked
File “PyInstaller\\loader\\pyimod03_importers.py”, line 495, in exec_module
File “ltp\\utils\\deploy_model.py”, line 12, in
File “frozen importlib._bootstrap>”, line 983, in _find_and_load
File “frozen importlib._bootstrap>”, line 967, in _find_and_load_unlocked
File “frozen importlib._bootstrap>”, line 677, in _load_unlocked
File “PyInstaller\\loader\\pyimod03_importers.py”, line 495, in exec_module
File “ltp\\transformer_multitask.py”, line 8, in
File “frozen importlib.bootstrap>", line 1032, in handle_fromlist
File "transformers_init.py", line 2941, in getattr
File “transformers\\file_utils.py”, line 1890, in getattr
File “transformers\\file_utils.py”, line 1889, in getattr
File "transformers\\models\\auto_init.py”, line 210, in get_module
File "importlib_init.py", line 127, in import_module
File “frozen importlib._bootstrap>”, line 1006, in _gcd_import
File “frozen importlib._bootstrap>”, line 983, in _find_and_load
File “frozen importlib._bootstrap>”, line 967, in _find_and_load_unlocked
File “frozen importlib._bootstrap>”, line 677, in _load_unlocked
File “PyInstaller\\loader\\pyimod03_importers.py”, line 495, in exec_module
File “transformers\\models\\auto\\modeling_auto.py”, line 94, in
File “frozen importlib._bootstrap>”, line 983, in _find_and_load
File “frozen importlib._bootstrap>”, line 967, in _find_and_load_unlocked
File “frozen importlib._bootstrap>”, line 677, in _load_unlocked
File “PyInstaller\\loader\\pyimod03_importers.py”, line 495, in exec_module
File “transformers\\models\\deberta\\modeling_deberta.py”, line 465, in
File “torch\\jit_script.py”, line 1307, in script
ast = get_jit_def(obj, obj.name)
File “torch\\jit\\frontend.py”, line 233, in get_jit_def
parsed_def = parse_def(fn)
File “torch_sources.py”, line 95, in parse_def
sourcelines, file_lineno, filename = get_source_lines_and_file(fn, ErrorReport.call_stack())
File “torch_sources.py”, line 28, in get_source_lines_and_file
sourcelines, file_lineno, filename = get_source_lines_and_file(fn, ErrorReport.call_stack())
File “torch_sources.py”, line 28, in get_source_lines_and_file
raise OSError(msg) from e
OSError: Can’t get source for <function c2p_dynamic_expand at 0x000001D6187E98B8>. TorchScript requires source access in order to carry out compilation, make sure original .py files are a
vailable.
[13036] Failed to execute script ‘p’ due to unhandled exception!

这个报错很长，但是归根结底其实就是OSError: could not get source code这个错误，然后导致了下面那些错误。

这个问题找了蛮久的，据说与torchvision有关，但是我完全没用torchvison，因此感觉什么版本对应调整对我来说没啥用。

只需要在调用torch包的上面（直接调包的最顶上放着就好了）加入以下代码再重新打包一次就好了

def script_method(fn, _rcb=None):
    return fn


def script(obj, optimize=True, _frames_up=0, _rcb=None):
    return obj


import torch.jit
script_method1 = torch.jit.script_method
script1 = torch.jit.script
torch.jit.script_method = script_method
torch.jit.script = script