Tensorflow 2 对象检测 API：Numpy 版本错误

Posted

技术标签:

【中文标题】Tensorflow 2 对象检测 API：Numpy 版本错误【英文标题】：Tensorflow 2 Object Detection API: Numpy Version Errors 【发布时间】：2021-05-28 02:06:10 【问题描述】：

我遵循了“训练自定义对象检测器”教程 (https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html)

运行脚本以继续训练预训练模型时：

python model_main_tf2.py --model_dir=models/my_ssd_resnet50_v1_fpn --pipeline_config_path=models/my_ssd_resnet50_v1_fpn/pipeline.config

（在这里找到：https://github.com/tensorflow/models/blob/master/research/object_detection/model_main_tf2.py）

使用不同的 numpy 版本，我收到以下错误。

场景 #1：

张量流：2.2.0 numpy: 1.20.0-1

NotImplementedError: Cannot convert a symbolic Tensor (cond_2/strided_slice:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported

我在网上看了，它建议降级 numpy 版本 NotImplementedError: Cannot convert a symbolic Tensor (2nd_target:0) to a numpy array)。注意，对于 tensorflow 2.2.0，numpy 版本必须 >= 1.19.2。

场景 #2：

张量流：2.2.0 numpy：1.19.2-5

ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

不过，网上的建议是将numpy升级到>= 1.20.0。 (ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject)。有一个与此相关的 github 问题：https://github.com/tensorflow/models/issues/9749。

我不确定使用什么版本来运行代码。

附录： 场景 #1 错误代码（numpy：1.20.0-1）

  File "model_main_tf2.py", line 113, in <module>
    tf.compat.v1.app.run()
  File "C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\platform\app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\absl\app.py", line 303, in run
    _run_main(main, args)
  File "C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\absl\app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "model_main_tf2.py", line 104, in main
    model_lib_v2.train_loop(
  File "C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\object_detection\model_lib_v2.py", line 530, in train_loop
    train_input = strategy.experimental_distribute_datasets_from_function(
  File "C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\util\deprecation.py", line 340, in new_func
    return func(*args, **kwargs)
  File "C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\distribute\distribute_lib.py", line 1143, in experimental_distribute_datasets_from_function
    return self.distribute_datasets_from_function(dataset_fn, options)
  File "C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\distribute\distribute_lib.py", line 1134, in distribute_datasets_from_function
    return self._extended._distribute_datasets_from_function(  # pylint: disable=protected-access
  File "C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\distribute\mirrored_strategy.py", line 545, in _distribute_datasets_from_function
    return input_lib.get_distributed_datasets_from_function(
  File "C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\distribute\input_lib.py", line 161, in get_distributed_datasets_from_function
    return DistributedDatasetsFromFunction(dataset_fn, input_workers,
  File "C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\distribute\input_lib.py", line 1272, in __init__
    _create_datasets_from_function_with_input_context(
  File "C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\distribute\input_lib.py", line 1936, in _create_datasets_from_function_with_input_context
    dataset = dataset_fn(ctx)
  File "C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\object_detection\model_lib_v2.py", line 521, in train_dataset_fn
    train_input = inputs.train_input(
  File "C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\object_detection\inputs.py", line 893, in train_input
    dataset = INPUT_BUILDER_UTIL_MAP['dataset_build'](
  File "C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\object_detection\builders\dataset_builder.py", line 251, in build
    dataset = dataset_map_fn(dataset, decoder.decode, batch_size,
  File "C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\object_detection\builders\dataset_builder.py", line 236, in dataset_map_fn
    dataset = dataset.map_with_legacy_function(
  File "C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\util\deprecation.py", line 340, in new_func
    return func(*args, **kwargs)
  File "C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 2679, in map_with_legacy_function
    ParallelMapDataset(
  File "C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 4242, in __init__
    self._map_func = StructuredFunctionWrapper(
  File "C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 3493, in __init__
    self._function.add_to_graph(ops.get_default_graph())
  File "C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\framework\function.py", line 546, in add_to_graph
    self._create_definition_if_needed()
  File "C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\framework\function.py", line 378, in _create_definition_if_needed
    self._create_definition_if_needed_impl()
  File "C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\framework\function.py", line 400, in _create_definition_if_needed_impl
    temp_graph = func_graph_from_py_func(
  File "C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\framework\function.py", line 971, in func_graph_from_py_func
    outputs = func(*func_graph.inputs)
  File "C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 3485, in wrapper_fn
    ret = _wrapper_helper(*args)
  File "C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 3453, in _wrapper_helper
    ret = autograph.tf_convert(func, ag_ctx)(*nested_args)
  File "C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\autograph\impl\api.py", line 670, in wrapper
    raise e.ag_error_metadata.to_exception(e)
NotImplementedError: in user code:

    C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\object_detection\data_decoders\tf_example_decoder.py:524 default_groundtruth_weights  *
        [tf.shape(tensor_dict[fields.InputDataFields.groundtruth_boxes])[0]],
    C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\util\dispatch.py:201 wrapper  **
        return target(*args, **kwargs)
    C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\ops\array_ops.py:3120 ones
        output = _constant_if_small(one, shape, dtype, name)
    C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\ops\array_ops.py:2804 _constant_if_small
        if np.prod(shape) < 1000:
    <__array_function__ internals>:5 prod

    C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\numpy\core\fromnumeric.py:3030 prod
        return _wrapreduction(a, np.multiply, 'prod', axis, dtype, out,
    C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\numpy\core\fromnumeric.py:87 _wrapreduction
        return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
    C:\Users\212765830\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\framework\ops.py:852 __array__
        raise NotImplementedError(
NotImplementedError: Cannot convert a symbolic Tensor (cond_2/strided_slice:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported

场景 #2 错误代码（numpy：1.19.2-5）

2021-02-25 09:59:47.039105: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2021-02-25 09:59:47.044310: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
  File "model_main_tf2.py", line 32, in <module>
    from object_detection import model_lib_v2
  File "C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\object_detection\model_lib_v2.py", line 29, in <module>
    from object_detection import eval_util
  File "C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\object_detection\eval_util.py", line 35, in <module>
    from object_detection.metrics import coco_evaluation
  File "C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\object_detection\metrics\coco_evaluation.py", line 25, in <module>
    from object_detection.metrics import coco_tools
  File "C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\object_detection\metrics\coco_tools.py", line 51, in <module>
    from pycocotools import coco
  File "C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\pycocotools\coco.py", line 56, in <module>
    from . import mask as maskUtils
  File "C:\Users\212765830\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib\site-packages\pycocotools\mask.py", line 3, in <module>
    import pycocotools._mask as _mask
  File "pycocotools\_mask.pyx", line 1, in init pycocotools._mask
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject```

【问题讨论】：

【参考方案1】：

我也有同样的问题：

NotImplementedError：无法将符号张量 (cond_2/strided_slice:0) 转换为 numpy 数组。此错误可能表明您正在尝试将张量传递给 NumPy 调用，这是不受支持的

通过在此函数https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/array_ops.py 中将np.prod 更改为reduce_prod 解决了问题

def _constant_if_small(value, shape, dtype, name):
  try:
    if np.prod(shape) < 1000:
      return constant(value, shape=shape, dtype=dtype, name=name)
  except TypeError:
    # Happens when shape is a Tensor, list with Tensor elements, etc.
    pass
  return None

注意，需要在顶部导入reduce_prod：

from tensorflow.math import reduce_prod

【讨论】：

很好的发现！你认为你可以为此在 TensorFlow 上开一个 PR 吗？看来这仍然是一个问题 谢谢。节省了一天。可以通过将 import 修改为“from tensorflow.python.ops.math_ops import reduce_prod”从上面解决它【参考方案2】：

我有同样的错误。你在窗户上吗？如果是这样尝试（这对我有用）： pip uninstall pycocotools pip install pycocotools-windows

【讨论】：

是的，成功了！是的，我在 Windows 上运行。 知道什么对 macOS 用户有用吗？【参考方案3】：

我在 MacOS 上遇到了这个问题。我通过安装旧版本的 pycocotools 来修复它。

pip install pycocotools==2.0.0

【讨论】：

谢谢。我正在尝试使用 python 3.7（根据 colab）运行 tflite_model_maker 示例。出于某种原因，2.0.2 在 Windows 上使用 numpy==1.19.5 运行良好，但在 Ubuntu 上我需要降级到 2.0.0【参考方案4】：

我有同样的问题，我将我的 numpy 从 1.20.2 降级到 1.19.5，现在它可以工作了。

pip install numpy==1.19.5

您可以在这里查看趋势：https://github.com/tensorflow/models/issues/9706

【讨论】：

以上是关于Tensorflow 2 对象检测 API：Numpy 版本错误的主要内容，如果未能解决你的问题，请参考以下文章

Tensorflow 2 对象检测 API：Numpy 版本错误

如何解决 Tensorflow 2 对象检测 API 中的 UnicodeError

如何从代码运行 tensorflow 对象检测 api (model_main_tf2)？

使用 Tensorflow 2 对象检测 API 冻结了哪些层？

如何使用 Tensorflow 2 对象检测 API 恢复微调模型以进行测试？

为啥来自 Tensorflow 2 对象检测 API 的微调模型上的 mAP 较低？