tensorflow：使用自定义生成器时您的输入用完了数据

Posted 2023-02-16

【中文标题】tensorflow：使用自定义生成器时您的输入用完了数据

【英文标题】：tensorflow:Your input ran out of data when using custom generator

【发布时间】：2021-11-25 02:56:09

【问题描述】：

我正在使用自定义生成器来传递我的数据。但是我一直遇到一个错误，说我已经用完了数据并在传递数据集时使用了 repeat()。我使用的是普通生成器，因此无法使用 repeat()。有人可以帮我解决这个问题吗

import os
import numpy as np
import cv2
def generator(idir,odir,batch_size,shuffle ):
    i_list=os.listdir(idir)
    o_list=os.listdir(odir)
    batch_index=0
    batch_size = batch_size
    sample_count=len(i_list)
    while True:
        input_image_batch=[]
        output_image_batch=[]
        
        for i in range(batch_index * batch_size, (batch_index + 1) * batch_size  ): 
        #iterate for  a batch
            j=i % sample_count # cycle j value over range of available  images
            k=j % batch_size  # cycle k value over batch size
            if shuffle == True: # if shuffle select a random integer between 0 and sample_count-1 to pick as the image=label pair
                m=np.random.randint(low=0, high=sample_count-1, size=None, dtype=int) 
            else:
                m=j
            path_to_in_img=os.path.join(idir,i_list[m])
            path_to_out_img=os.path.join(odir,o_list[m])
            print(path_to_in_img,path_to_out_img)
            input_image=cv2.imread(path_to_in_img)
            input_image=cv2.resize(input_image,(3200,3200))#create the target image from the input image 
            output_image=cv2.imread(path_to_out_img)
            output_image=cv2.resize(output_image,(3200,3200))
            input_image_batch.append(input_image)
            output_image_batch.append(output_image)
                    
        input_val1image_array=np.array(input_image_batch) 
        input_val1image_array = input_val1image_array / 255.0
        print (input_val1image_array)
        output_val2image_array=np.array(output_image_batch)
        output_val2image_array = output_val2image_array / 255.0
        batch_index= batch_index + 1 
        yield (input_val1image_array, output_val2image_array)
        if batch_index * batch_size > sample_count:
                 break

调用函数

    idir = r"D:\\image\\"
    odir=r"D:\\image1\\"
    train = generator(idir,odir,4,True)

model.compile(optimizer="adam", loss='mean_squared_error', metrics=['mean_squared_error'])

model.fit(train,validation_data = (valin_images,valout_images),batch_size= 5,epochs = 20,steps_per_epoch = int(560/batch_size))

错误

Epoch 1/20
186/186 [==============================] - 475s 3s/step - loss: 1779.7604 - mean_squared_error: 1779.7601 - val_loss: 28278.5488 - val_mean_squared_error: 28278.5488
Epoch 2/20
  1/186 [..............................] - ETA: 1:41 - loss: 275.7113 - mean_squared_error: 275.7113WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches (in this case, 3720 batches). You may need to use the repeat() function when building your dataset.
WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches (in this case, 187 batches). You may need to use the repeat() function when building your dataset.
186/186 [==============================] - 1s 235us/step - loss: 275.7113 - mean_squared_error: 275.7113

【参考方案1】：

如果您不使用重复（即使您使用它对调试很有用），您需要检查的第一件事是您的生成器创建了多少元素。 一种快速的方法是使用类似

len([g for g in generator(idir,odir,4,True)])

那么您需要确保每个 epoch 的步数乘以批量大小小于该数字。

即使使用该生成器，您也可以使用 repeat - 您只需要像这样用 tf.dataset 包装它：

gen = lambda : generator(idir,odir,4,True)
dataset = tf.data.Dataset.from_generator(gen, output_types=(<types>), output_shapes=(<shapes>)).repeat()

您必须指定输出类型和形状，但它可以正常工作。