生成 tf.record 文件时出现 Python 错误 - Tensorflow.python.framework.errors_impl.NotFoundError：没有这样的文件或目录

Posted 2023-02-16

【中文标题】生成 tf.record 文件时出现 Python 错误 - Tensorflow.python.framework.errors_impl.NotFoundError：没有这样的文件或目录

【英文标题】：Python error when generating tf.record file - Tensorflow.python.framework.errors_impl.NotFoundError: No such file or directory

【发布时间】：2021-10-25 21:50:15

【问题描述】：

我正在使用 TensorFlow 2.5.0（python 3.9、Ubuntu 20.04 和 Anaconda 3）来创建自定义对象检测器。我已经标记了我的图像，并按照this 教程来达到这一点。但是，我在尝试创建 tf.record 文件时收到以下错误（请参阅下面列出的最后一行错误）：

Traceback (most recent call last):
  File "/home/database/Custom-Object-Detector-Workspace/Project/scripts/generate_tfrecord.py", line 169, in <module>
    tf.app.run()
  File "/home/database/python3.9/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/home/database/python3.9/site-packages/absl/app.py", line 303, in run
    _run_main(main, args)
  File "/home/database/python3.9/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "/home/database/Custom-Object-Detector-Workspace/Project/scripts/generate_tfrecord.py", line 159, in main
    tf_example = create_tf_example(group, path)
  File "/home/database/tensorflow/Custom-Object-Detector-Workspace/Project/scripts/generate_tfrecord.py", line 113, in create_tf_example
    encoded_jpg = fid.read()
  File "/home/database/python3.9/site-packages/tensorflow/python/lib/io/file_io.py", line 117, in read
    self._preread_check()
  File "/home/database/anaconda3/envs/tensorflow/lib/python3.9/site-packages/tensorflow/python/lib/io/file_io.py", line 79, in _preread_check
    self._read_buf = _pywrap_file_io.BufferedInputStream(
tensorflow.python.framework.errors_impl.NotFoundError: /home/database/images/test/DJI_0001_008.jpg; No such file or directory

我的图像文件夹中没有名为 DJI_0001_008.jpg 的文件。为了确保照片的命名是唯一的，我在每个文件名（在 _008 和 .jpg 扩展名之间）附加了唯一的字母，以便在组合时可以将它们分别区分为单独的文件。一个快速的locate 命令会调出我的训练和测试文件夹中的所有文件，其中包含文件名：

/images/test/DJI_0001_008_r.jpg
/images/test/DJI_0001_008_r.xml
/images/train/DJI_0001_008_b.jpg
/images/train/DJI_0001_008_b.xml
/images/train/DJI_0001_008_m.jpg
/images/train/DJI_0001_008_m.xml

如您所见，每张 DJI_0001_008.jpg 都有一个唯一的字母，在其后附加到照片本身和文件扩展名前。但是 generate_tfrecord.py 脚本仍然抛出这个错误。

如何解决此错误？这是命名错误吗？如果是这样，最好重命名它们以便“DJI_0001_008”不在任何文件名中？我发现有些图片有 .jpg 扩展名，而另一些有 .JPG 扩展名，这会是一个因素吗？

#######################################################

错误是通过运行这些anaconda命令产生的

#Activate Anaconda environment
conda activate tensorflow

#Set workplace directory where the python script is in
cd /home/desired_directory

#run generate_tfrecord.py script, set for creating a tf.record file in the test folder:
python3 generate_tfrecord.py -x /home/tensorflow_ihb/Custom-Object-Detector-Workspace/Project/images/test -l /home/tensorflow_ihb/Custom-Object-Detector-Workspace/Project/annotations/label_map.pbtxt -o /home/tensorflow_ihb/Custom-Object-Detector-Workspace/Project/annotations/test.record -c /home/tensorflow_ihb/Custom-Object-Detector-Workspace/Project/annotations/test.csv

这里是生成错误的 generate_tfrecord.py 脚本：

""" Sample TensorFlow XML-to-TFRecord converter

usage: generate_tfrecord.py [-h] [-x XML_DIR] [-l LABELS_PATH] [-o OUTPUT_PATH] [-i IMAGE_DIR] [-c CSV_PATH]

optional arguments:
  -h, --help            show this help message and exit
  -x XML_DIR, --xml_dir XML_DIR
                        Path to the folder where the input .xml files are stored.
  -l LABELS_PATH, --labels_path LABELS_PATH
                        Path to the labels (.pbtxt) file.
  -o OUTPUT_PATH, --output_path OUTPUT_PATH
                        Path of output TFRecord (.record) file.
  -i IMAGE_DIR, --image_dir IMAGE_DIR
                        Path to the folder where the input image files are stored. Defaults to the same directory as XML_DIR.
  -c CSV_PATH, --csv_path CSV_PATH
                        Path of output .csv file. If none provided, then no file will be written.
"""

import os
import glob
import pandas as pd
import io
import xml.etree.ElementTree as ET
import argparse

os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'    # Suppress TensorFlow logging (1)
import tensorflow.compat.v1 as tf
from PIL import Image
from object_detection.utils import dataset_util, label_map_util
from collections import namedtuple

# Initiate argument parser
parser = argparse.ArgumentParser(
    description="Sample TensorFlow XML-to-TFRecord converter")
parser.add_argument("-x",
                    "--xml_dir",
                    help="Path to the folder where the input .xml files are stored.",
                    type=str)
parser.add_argument("-l",
                    "--labels_path",
                    help="Path to the labels (.pbtxt) file.", type=str)
parser.add_argument("-o",
                    "--output_path",
                    help="Path of output TFRecord (.record) file.", type=str)
parser.add_argument("-i",
                    "--image_dir",
                    help="Path to the folder where the input image files are stored. "
                         "Defaults to the same directory as XML_DIR.",
                    type=str, default=None)
parser.add_argument("-c",
                    "--csv_path",
                    help="Path of output .csv file. If none provided, then no file will be "
                         "written.",
                    type=str, default=None)

args = parser.parse_args()

if args.image_dir is None:
    args.image_dir = args.xml_dir

label_map = label_map_util.load_labelmap(args.labels_path)
label_map_dict = label_map_util.get_label_map_dict(label_map)


def xml_to_csv(path):
    """Iterates through all .xml files (generated by labelImg) in a given directory and combines
    them in a single Pandas dataframe.

    Parameters:
    ----------
    path : str
        The path containing the .xml files
    Returns
    -------
    Pandas DataFrame
        The produced dataframe
    """

    xml_list = []
    for xml_file in glob.glob(path + '/*.xml'):
        tree = ET.parse(xml_file)
        root = tree.getroot()
        for member in root.findall('object'):
            value = (root.find('filename').text,
                     int(root.find('size')[0].text),
                     int(root.find('size')[1].text),
                     member[0].text,
                     int(member[4][0].text),
                     int(member[4][1].text),
                     int(member[4][2].text),
                     int(member[4][3].text)
                     )
            xml_list.append(value)
    column_name = ['filename', 'width', 'height',
                   'class', 'xmin', 'xmax', 'ymin', 'ymax']
    xml_df = pd.DataFrame(xml_list, columns=column_name)
    xml_df.to_csv("dataset.csv")
    return xml_df


def class_text_to_int(row_label):
    return label_map_dict[row_label]


def split(df, group):
    data = namedtuple('data', ['filename', 'object'])
    gb = df.groupby(group)
    return [data(filename, gb.get_group(x)) for filename, x in zip(gb.groups.keys(), gb.groups)]


def create_tf_example(group, path):
    with tf.gfile.GFile(os.path.join(path, ''.format(group.filename)), 'rb') as fid:
        encoded_jpg = fid.read()
    encoded_jpg_io = io.BytesIO(encoded_jpg)
    image = Image.open(encoded_jpg_io)
    width, height = image.size

    filename = group.filename.encode('utf8')
    image_format = b'jpg'
    xmins = []
    xmaxs = []
    ymins = []
    ymaxs = []
    classes_text = []
    classes = []

    for index, row in group.object.iterrows():
        xmins.append(row['xmin'] / width)
        xmaxs.append(row['xmax'] / width)
        ymins.append(row['ymin'] / height)
        ymaxs.append(row['ymax'] / height)
        classes_text.append(row['class'].encode('utf8'))
        classes.append(class_text_to_int(row['class']))

    tf_example = tf.train.Example(features=tf.train.Features(feature=
        'image/height': dataset_util.int64_feature(height),
        'image/width': dataset_util.int64_feature(width),
        'image/filename': dataset_util.bytes_feature(filename),
        'image/source_id': dataset_util.bytes_feature(filename),
        'image/encoded': dataset_util.bytes_feature(encoded_jpg),
        'image/format': dataset_util.bytes_feature(image_format),
        'image/object/bbox/xmin': dataset_util.float_list_feature(xmins),
        'image/object/bbox/xmax': dataset_util.float_list_feature(xmaxs),
        'image/object/bbox/ymin': dataset_util.float_list_feature(ymins),
        'image/object/bbox/ymax': dataset_util.float_list_feature(ymaxs),
        'image/object/class/text': dataset_util.bytes_list_feature(classes_text),
        'image/object/class/label': dataset_util.int64_list_feature(classes),
    ))
    return tf_example


def main(_):

    writer = tf.python_io.TFRecordWriter(args.output_path)
    path = os.path.join(args.image_dir)
    examples = xml_to_csv(args.xml_dir)
    grouped = split(examples, 'filename')
    for group in grouped:
        tf_example = create_tf_example(group, path)
        writer.write(tf_example.SerializeToString())
    writer.close()
    print('Successfully created the TFRecord file: '.format(args.output_path))
    if args.csv_path is not None:
        examples.to_csv(args.csv_path, index=None)
        print('Successfully created the CSV file: '.format(args.csv_path))


if __name__ == '__main__':
    tf.app.run()

【问题讨论】：

更新 - 我已经重命名了每对 .jpg 和 .xml 文件，但我仍然得到相同的答案。我不知道发生了什么。

【参考方案1】：

重命名文件后，我未能重命名 .xml 文件本身中的文件名。

例如，“image9.jpg”的 .xml 注释仍然读取（更改名称后）：

<annotation>
    <folder>resized</folder>
    <filename>DJI_0001_007.jpg</filename>
    <path>C:\image_path\resized\DJI_0001_007.jpg</path>
    <source>
        <database>Unknown</database>
    </source>
    <size>
        <width>1024</width>
        <height>1024</height>
        <depth>3</depth>
    </size>
    <segmented>0</segmented>
    <object>
        <name>Class_name</name>
        <pose>Unspecified</pose>
        <truncated>1</truncated>
        <difficult>0</difficult>
        <bndbox>
            <xmin>1</xmin>
            <ymin>9</ymin>
            <xmax>643</xmax>
            <ymax>760</ymax>
        </bndbox>
    </object>
</annotation>

这似乎是问题所在。看来要么需要返回并再次注释图像以生成新的 .xml 文件，要么找到一种方法来替换 .xml 文件中的名称。