[Ray.Tune] [已解决] TypeError: ray.cloudpickle.dumps

Posted 2022-01-14 MarToony|名角

问题描述：

2021-12-24 17:17:36,223	WARNING experiment.py:256 -- No name detected on trainable. Using DEFAULT.
2021-12-24 17:17:36,224	INFO registry.py:70 -- Detected unknown callable for trainable. Converting to class.
Traceback (most recent call last):
  File "/home/bullet/PycharmProjects/LowlightRecognition/DecoupleGCN-DropGraph/main_hyper.py", line 679, in <module>
    local_dir="/home/bullet/PycharmProjects/LowlightRecognition/coldnight/n3_storage/ray_results_downsampled",
  File "/home/bullet/anaconda3/envs/yolox/lib/python3.7/site-packages/ray/tune/tune.py", line 444, in run
    restore=restore)
  File "/home/bullet/anaconda3/envs/yolox/lib/python3.7/site-packages/ray/tune/experiment.py", line 113, in __init__
    self._run_identifier = Experiment.register_if_needed(run)
  File "/home/bullet/anaconda3/envs/yolox/lib/python3.7/site-packages/ray/tune/experiment.py", line 265, in register_if_needed
    raise type(e)(str(e) + " " + extra_msg) from None
TypeError: ray.cloudpickle.dumps(<class 'ray.tune.function_runner.wrap_function.<locals>.ImplicitFunc'>) failed.
To check which non-serializable variables are captured in scope, re-run the ray script with 'RAY_PICKLE_VERBOSE_DEBUG=1'. Other options: 
-Try reproducing the issue by calling `pickle.dumps(trainable)`. 
-If the error is typing-related, try removing the type annotations and try again.

解决方法：
在trainable中的初始化随机种子的函数中，添加import torch。
大致的代码如下所示。

def init_seed(x):
    import torch
    ... # 其他的随机种子的设置
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

def trainable(arg,config....):
    arg.seed = config["seed"]
    init_seed(arg.seed)
    ......

参考：Can’t pickle CudnnModule objects