停止迭代：generator_output = next(output_generator)

Posted 2023-02-23

【中文标题】停止迭代：generator_output = next(output_generator)

【英文标题】：StopIteration: generator_output = next(output_generator)

【发布时间】：2018-07-20 11:03:07

【问题描述】：

我有以下代码，我重写以处理大规模数据集。我正在使用 Python 生成器对模型逐批生成数据进行拟合。

def subtract_mean_gen(x_source,y_source,avg_image,batch):
    batch_list_x=[]
    batch_list_y=[]
    for line,y in zip(x_source,y_source):
        x=line.astype('float32')
        x=x-avg_image
        batch_list_x.append(x)
        batch_list_y.append(y)
        if len(batch_list_x) == batch:
            yield (np.array(batch_list_x),np.array(batch_list_y))
            batch_list_x=[]
            batch_list_y=[] 

model = resnet.ResnetBuilder.build_resnet_18((img_channels, img_rows, img_cols), nb_classes)
model.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

val = subtract_mean_gen(X_test,Y_test,avg_image_test,batch_size)
model.fit_generator(subtract_mean_gen(X_train,Y_train,avg_image_train,batch_size), steps_per_epoch=X_train.shape[0]//batch_size,epochs=nb_epoch,validation_data = val,
                    validation_steps = X_test.shape[0]//batch_size)

我收到以下错误：

239/249 [===========================>..] - ETA: 60s - loss: 1.3318 - acc: 0.8330Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keras/utils/data_utils.py", line 560, in data_generator_task
    generator_output = next(self._generator)
StopIteration

240/249 [===========================>..] - ETA: 54s - loss: 1.3283 - acc: 0.8337Traceback (most recent call last):
  File "cifa10-copy.py", line 125, in <module>
    validation_steps = X_test.shape[0]//batch_size)
  File "/usr/local/lib/python2.7/dist-packages/keras/legacy/interfaces.py", line 87, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1809, in fit_generator
    generator_output = next(output_generator)
StopIteration

我查看了here 发布的类似问题，但是，我无法解决引发 StopIteration 的错误。

【问题讨论】：

生成器没有任何东西可以给你 - 因此它引发了 StopIteration...

【参考方案1】：

keras 的生成器必须是无限的：

def subtract_mean_gen(x_source,y_source,avg_image,batch):
    while True:
        batch_list_x=[]
        batch_list_y=[]
        for line,y in zip(x_source,y_source):
            x=line.astype('float32')
            x=x-avg_image
            batch_list_x.append(x)
            batch_list_y.append(y)
            if len(batch_list_x) == batch:
                yield (np.array(batch_list_x),np.array(batch_list_y))
                batch_list_x=[]
                batch_list_y=[] 

发生错误是因为 keras 尝试获取新批次，但您的生成器已经结束。 （即使您定义了正确的步骤数，keras 也有一个队列，即使您处于最后一步，也会尝试从生成器中获取更多批次。）

显然，您有一个默认队列大小，即 10（异常出现在结束前 10 个批次，因为队列试图在结束后获得批次）。

【参考方案2】：

正如您提供的链接问题所示，Keras 生成器必须无限迭代，因此您可以根据需要将元素输出到训练中。更多关于 this Github issue 的信息。

为此，您必须对生成器进行一些修改，例如：

def subtract_mean_gen(x_source,y_source,avg_image,batch):
batch_list_x=[]
batch_list_y=[]
while 1: #run forever, so you can generate elements indefinitely
    for line,y in zip(x_source,y_source):
        x=line.astype('float32')
        x=x-avg_image    
        batch_list_x.append(x)
        batch_list_y.append(y)
        if len(batch_list_x) == batch:
            yield (np.array(batch_list_x),np.array(batch_list_y))
            batch_list_x=[]
            batch_list_y=[]

【讨论】：

我从重写代码的示例中删除了while True。