如何修复 InvalidArgumentError:logits 和标签必须是可广播的:logits_size=[32,198] labels_size=[32,3]
Posted
技术标签:
【中文标题】如何修复 InvalidArgumentError:logits 和标签必须是可广播的:logits_size=[32,198] labels_size=[32,3]【英文标题】:How to fix InvalidArgumentError: logits and labels must be broadcastable: logits_size=[32,198] labels_size=[32,3] 【发布时间】:2021-12-12 01:36:15 【问题描述】:我实际上是在尝试对图像进行表面缺陷检测(检查墙壁上的缺陷,如裂缝……)当我尝试拟合模型时,它会抛出错误 logits 并且标签必须是 broadcastable: logits_size=[32,198] labels_size=[32,3]
我尝试了几种方法,但都没有奏效。我该如何克服这个错误,或者我选择的方法有什么问题? 我正在使用的数据是未标记的图像数据(所有图像都在一个文件夹中)
from keras.preprocessing.image import ImageDataGenerator
train_model = ImageDataGenerator(rescale = 1./255,
shear_range = 0.2,
zoom_range = 0.2,
horizontal_flip = True)
test_model = ImageDataGenerator(rescale = 1./255)
training_data = train_model.flow_from_directory('/Users/nm2/Public/ai-dataset-training-100/5/23_463_DISTACCO_DEL_COPRIFERRO_Q100_training_dataset',
target_size = (224, 224),
batch_size = 32,
class_mode = 'categorical')
testing_data = test_model.flow_from_directory('/Users/nm2/Public/ai-dataset-training-100/5/23_463_DISTACCO_DEL_COPRIFERRO_Q100_training_dataset',
target_size = (224, 224),
batch_size = 32,
class_mode = 'categorical')
IMAGE_SIZE = [224, 224]
#Import the Vgg 16 and add the preprocessing layer to front of the VGG16 Here we will use ImageNet PreTrained Weights
vgg_model = VGG16(input_shape=IMAGE_SIZE + [3], weights='imagenet', include_top=False)
for layer in vgg_model.layers:
layer.trainable = False
x = Flatten()(vgg_model.output)
#We use glob function to find out how many files are there in the working directory and count the number of classes they belong to.
folder_count = glob('/Users/nm2/Public/ai-dataset-training-`100/5/23_493_PANORAMICA_LIVELLO_BASE_ISPEZIONE_Q100_training_dataset/*')`
prediction = Dense(len(folder_count), activation='softmax')(x)
#Create a Model
model = Model(inputs=vgg_model.input, outputs=prediction)
model.summary()
model.compile(
loss='categorical_crossentropy',
optimizer='adam',
metrics=['accuracy']
)
post_run = model.fit(training_data,
validation_data=testing_data,
epochs=10,
steps_per_epoch=len(training_data),
validation_steps=len(testing_data))
InvalidArgumentError: logits and labels must be broadcastable: logits_size=[32,198] labels_size=[32,3]
[[node categorical_crossentropy/softmax_cross_entropy_with_logits (defined at var/folders/3b/tfwxbsyd41j64kbrjghzrvcm0000gq/T/ipykernel_1068/3441923959.py:5) ]] [Op:__inference_train_function_1205]
Function call stack:
train_function
【问题讨论】:
【参考方案1】:这是应该为您工作的完整代码。
import tensorflow as tf
import tensorflow as tf
from tensorflow import keras
from keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.metrics import categorical_crossentropy
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense, Activation,Dropout
data_dir=r'C:\Temp\DATA' # directory where the image files are stored change to your directory
vsplit=.2 #percentage of data to be used for validation
IMAGE_SIZE = [224, 224]
IMAGE_SHAPE=[224,224,3]
train_model = ImageDataGenerator(rescale = 1./255, shear_range = 0.2, zoom_range = 0.2,
horizontal_flip = True, validation_split=vsplit)
test_model = ImageDataGenerator(rescale = 1./255, validation_split=vsplit)
training_data = train_model.flow_from_directory(data_dir, target_size = IMAGE_SIZE,batch_size = 32,
class_mode = 'categorical', subset='training',
shuffle = True, seed=123)
testing_data = test_model.flow_from_directory(data_dir, target_size = IMAGE_SIZE, batch_size = 32,
class_mode = 'categorical', subset='validation',
shuffle=True, seed=123)
class_dict=training_data.class_indices
classes=list(class_dict.keys())
print ('LIST OF CLASSES ', classes)
print ('CLASS DICTIONARY ',class_dict)
number_of_classes=len(classes) # this is the number of neurons in your top layer of the model
print ('Number of classes = ', number_of_classes)
base_model=tf.keras.applications.VGG19(include_top=False, weights="imagenet",input_shape=IMAGE_SHAPE, pooling='max')
# Note setting pooling='max' eliminates the need for a flatten layer
# I do not recommend using VGG it is a very large model I recommend using EfficientNetB3- note do not rescale
# the pixel for Efficient net.
x=base_model.output
base_model.trainable=False
x = Dense(256, activation='relu')(x)
x=Dropout(rate=.45, seed=123)(x)
output=Dense(number_of_classes, activation='softmax')(x)
model=Model(inputs=base_model.input, outputs=output)
model.compile(Adam(learning_rate=.001), loss='categorical_crossentropy', metrics=['accuracy'])
epochs=5
# I recommend the use of callbacks to control the learning rate and early stopping
rlronp=tf.keras.callbacks.ReduceLROnPlateau( monitor="val_loss", factor=0.5, patience=1, verbose=1)
estop=tf.keras.callbacks.EarlyStopping( monitor="val_loss", patience=3, verbose=1, restore_best_weights=True)
history=model.fit(x=training_data, epochs=epochs, verbose=1, validation_data=testing_data,
callbacks=[rlronp, estop], validation_steps=None, shuffle=True, initial_epoch=0)
# Fine Tune the model
base_model.trainable=True # make the base model trainable
epochs=5
history=model.fit(x=training_data, epochs=epochs, verbose=1, validation_data=testing_data,
callbacks=[rlronp, estop], validation_steps=None, shuffle=True, initial_epoch=0)
【讨论】:
val_acc 在整个 epochs val_accuracy: 0.7391 (from start to end) 中保持不变。早期停止是在第 7 阶段。下面我在 Epoch 6/10 4/4 - 38s 9s/step - loss: 0.8957 - accuracy: 0.6500 - val_loss: 0.7266 - val_accuracy: 0.7391 Epoch 7/10 4/ 之后放置了一些输出4 - 36s 12s/step - loss: 0.8366 - accuracy: 0.6400 - val_loss: 0.7243 - val_accuracy: 0.7391 Epoch 00007: ReduceLROnPlateau 将学习率降低到 1.5625000742147677e-05。从最佳时期结束时恢复模型权重。纪元 00007:提前停止【参考方案2】:你有这个代码作为你的模型顶层
prediction = Dense(len(folder_count), activation='softmax')(x)
这一层的神经元数量应该与你拥有的类的数量相同。同样在 model.fit 你有
steps_per_epoch=len(training_data), validation_steps=len(testing_data))
这应该是
batch_size=32
steps_per_epoch=len(training_data)/batch_size
validation_steps=len(testing_data)/batch_size
或者不指定这些值,model.fit 将在内部确定正确的值。你也有代码
vgg_model = VGG16(input_shape=IMAGE_SIZE + [3]
改成
vgg_model = VGG16(input_shape=[224,224,3]
【讨论】:
您好,感谢您的评论。我尝试了所有的更改。所有 10 个 epoch 的准确度都是 1.00。我该如何解决?或者我可以尝试哪些其他模型进行表面缺陷检测? 完整代码见下面我的新答案。我使用了不同的数据集,但它适用于您,只需将数据目录设置为指向您的数据集以上是关于如何修复 InvalidArgumentError:logits 和标签必须是可广播的:logits_size=[32,198] labels_size=[32,3]的主要内容,如果未能解决你的问题,请参考以下文章
InvalidArgumentError:输入必须是向量,得到形状:[]
Tensorflow 中带有 model.fit 的 InvalidArgumentError
InvalidArgumentError: input_1:0 已输入和提取