keras模型中卷积层的可视化

Posted

技术标签:

【中文标题】keras模型中卷积层的可视化【英文标题】：visualization of convolutional layer in keras model 【发布时间】：2017-01-09 21:15:54 【问题描述】：

我在 Keras 中创建了一个模型（我是新手），并且设法很好地训练了它。它需要 300x300 图像并尝试将它们分为两组。

# size of image in pixel
img_rows, img_cols = 300, 300
# number of classes (here digits 1 to 10)
nb_classes = 2
# number of convolutional filters to use
nb_filters = 16
# size of pooling area for max pooling
nb_pool = 20
# convolution kernel size
nb_conv = 20

X = np.vstack([X_train, X_test]).reshape(-1, 1, img_rows, img_cols)
y = np_utils.to_categorical(np.concatenate([y_train, y_test]), nb_classes)

# build model
model = Sequential()
model.add(Convolution2D(nb_filters, nb_conv, nb_conv, border_mode='valid', input_shape=(1, img_rows, img_cols)))
model.add(Activation('relu'))
model.add(Convolution2D(nb_filters, nb_conv, nb_conv))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(nb_pool, nb_pool)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(nb_classes))
model.add(Activation('softmax'))

# run model
model.compile(loss='categorical_crossentropy', optimizer='adadelta', metrics=['accuracy'])

现在我想可视化第二个卷积层，如果可能的话还有第一个密集层。 “灵感”取自keras blog。通过使用model.summary()，我找到了图层的名称。然后我创建了以下科学怪人代码：

from __future__ import print_function
from scipy.misc import imsave
import numpy as np
import time
#from keras.applications import vgg16
import keras
from keras import backend as K

# dimensions of the generated pictures for each filter.
img_width = 300
img_height = 300

# the name of the layer we want to visualize
# (see model definition at keras/applications/vgg16.py)
layer_name = 'convolution2d_2'
#layer_name = 'dense_1'

# util function to convert a tensor into a valid image
def deprocess_image(x):
    # normalize tensor: center on 0., ensure std is 0.1
    x -= x.mean()
    x /= (x.std() + 1e-5)
    x *= 0.1

    # clip to [0, 1]
    x += 0.5
    x = np.clip(x, 0, 1)

    # convert to RGB array
    x *= 255
    if K.image_dim_ordering() == 'th':
        x = x.transpose((1, 2, 0))
    x = np.clip(x, 0, 255).astype('uint8')
    return x

# load model
loc_json = 'my_model_short_architecture.json'
loc_h5 = 'my_model_short_weights.h5'

with open(loc_json, 'r') as json_file:
    loaded_model_json = json_file.read()

model = keras.models.model_from_json(loaded_model_json)

# load weights into new model
model.load_weights(loc_h5)
print('Model loaded.')

model.summary()

# this is the placeholder for the input images
input_img = model.input

# get the symbolic outputs of each "key" layer (we gave them unique names).
layer_dict = dict([(layer.name, layer) for layer in model.layers[1:]])


def normalize(x):
    # utility function to normalize a tensor by its L2 norm
    return x / (K.sqrt(K.mean(K.square(x))) + 1e-5)


kept_filters = []
for filter_index in range(0, 200):
    # we only scan through the first 200 filters,
    # but there are actually 512 of them
    print('Processing filter %d' % filter_index)
    start_time = time.time()

    # we build a loss function that maximizes the activation
    # of the nth filter of the layer considered
    layer_output = layer_dict[layer_name].output
    if K.image_dim_ordering() == 'th':
        loss = K.mean(layer_output[:, filter_index, :, :])
    else:
        loss = K.mean(layer_output[:, :, :, filter_index])


    # we compute the gradient of the input picture wrt this loss
    grads = K.gradients(loss, input_img)[0]

    # normalization trick: we normalize the gradient
    grads = normalize(grads)

    # this function returns the loss and grads given the input picture
    iterate = K.function([input_img], [loss, grads])

    # step size for gradient ascent
    step = 1.

    # we start from a gray image with some random noise
    if K.image_dim_ordering() == 'th':
        input_img_data = np.random.random((1, 3, img_width, img_height))
    else:
        input_img_data = np.random.random((1, img_width, img_height, 3))
    input_img_data = (input_img_data - 0.5) * 20 + 128

    # we run gradient ascent for 20 steps
    for i in range(20):
        loss_value, grads_value = iterate([input_img_data])
        input_img_data += grads_value * step

        print('Current loss value:', loss_value)
        if loss_value <= 0.:
            # some filters get stuck to 0, we can skip them
            break

    # decode the resulting input image
    if loss_value > 0:
        img = deprocess_image(input_img_data[0])
        kept_filters.append((img, loss_value))
    end_time = time.time()
    print('Filter %d processed in %ds' % (filter_index, end_time - start_time))

# we will stich the best 64 filters on a 8 x 8 grid.
n = 8

# the filters that have the highest loss are assumed to be better-looking.
# we will only keep the top 64 filters.
kept_filters.sort(key=lambda x: x[1], reverse=True)
kept_filters = kept_filters[:n * n]

# build a black picture with enough space for
# our 8 x 8 filters of size 128 x 128, with a 5px margin in between
margin = 5
width = n * img_width + (n - 1) * margin
height = n * img_height + (n - 1) * margin
stitched_filters = np.zeros((width, height, 3))

# fill the picture with our saved filters
for i in range(n):
    for j in range(n):
        img, loss = kept_filters[i * n + j]
        stitched_filters[(img_width + margin) * i: (img_width + margin) * i + img_width,
                         (img_height + margin) * j: (img_height + margin) * j + img_height, :] = img

# save the result to disk
imsave('stitched_filters_%dx%d.png' % (n, n), stitched_filters)

执行后我得到：

ValueError                                Traceback (most recent call last)
/home/user/conv_filter_visualization.py in <module>()
     97     # we run gradient ascent for 20 steps
/home/user/.local/lib/python3.4/site-packages/theano/compile/function_module.py in __call__(self, *args, **kwargs)
    857         t0_fn = time.time()
    858         try:
--> 859             outputs = self.fn()
    860         except Exception:
    861             if hasattr(self.fn, 'position_of_error'):

ValueError: CorrMM images and kernel must have the same stack size

Apply node that caused the error: CorrMMvalid, (1, 1)(convolution2d_input_1, Subtensor::, ::, ::int64, ::int64.0)
Toposort index: 8
Inputs types: [TensorType(float32, 4D), TensorType(float32, 4D)]
Inputs shapes: [(1, 3, 300, 300), (16, 1, 20, 20)]
Inputs strides: [(1080000, 360000, 1200, 4), (1600, 1600, -80, -4)]
Inputs values: ['not shown', 'not shown']
Outputs clients: [[Elemwiseadd,no_inplace(CorrMMvalid, (1, 1).0, Reshape4.0), ElemwiseComposite(i0 * (Abs(i1) + i2 + i3))[(0, 1)](TensorConstant(1, 1, 1, 1) of 0.5, Elemwiseadd,no_inplace.0, CorrMMvalid, (1, 1).0, Reshape4.0)]]

Backtrace when the node is created(use Theano flag traceback.limit=N to make it longer):
  File "/home/user/.local/lib/python3.4/site-packages/keras/models.py", line 787, in from_config
    model.add(layer)
  File "/home/user/.local/lib/python3.4/site-packages/keras/models.py", line 114, in add
    layer.create_input_layer(batch_input_shape, input_dtype)
  File "/home/user/.local/lib/python3.4/site-packages/keras/engine/topology.py", line 341, in create_input_layer
    self(x)
  File "/home/user/.local/lib/python3.4/site-packages/keras/engine/topology.py", line 485, in __call__
    self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
  File "/home/user/.local/lib/python3.4/site-packages/keras/engine/topology.py", line 543, in add_inbound_node
    Node.create_node(self, inbound_layers, node_indices, tensor_indices)
  File "/home/user/.local/lib/python3.4/site-packages/keras/engine/topology.py", line 148, in create_node
    output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
  File "/home/user/.local/lib/python3.4/site-packages/keras/layers/convolutional.py", line 356, in call
    filter_shape=self.W_shape)
  File "/home/user/.local/lib/python3.4/site-packages/keras/backend/theano_backend.py", line 862, in conv2d
    filter_shape=filter_shape)

我想我的尺寸有些不好，但我什至不知道从哪里开始。任何帮助，将不胜感激。谢谢。

【问题讨论】：

你想得到权重还是中间输出？ @MikaelRousson：我想在我的第二个卷积层中绘制每个“什么样的输入使每个过滤器最大化”。据我了解，我在这里造成了真正的混乱:) 【参考方案1】：

Keras 让获取层的权重和输出变得非常容易。看看https://keras.io/layers/about-keras-layers/ 或https://keras.io/getting-started/functional-api-guide/#the-concept-of-layer-node。

基本上可以通过每一层的属性weights和output来获取。

【讨论】：

【参考方案2】：

看看这个项目：

https://github.com/philipperemy/keras-visualize-activations

您可以提取每一层的激活图。它适用于所有 Keras 模型。

【讨论】：

【参考方案3】：

在您的网络中，第一个卷积层只有 16 个过滤器，下一个卷积层只有 16 个，因此您有 32 个卷积过滤器。但是您正在为 200 运行 for 循环。尝试将其更改为 16 或 32。我正在使用 TF 后端运行此代码，它适用于我的小型 CNN。 另外，更改图像拼接代码：

for i in range(n):
    for j in range(n):
        if(i * n + j)<=len(kept_filters)-1:

祝你好运……

【讨论】：

【参考方案4】：

只是一个简单的函数，比如

def plot_conv_weights(model, layer_name):
    W = model.get_layer(name=layer_name).get_weights()[0]
    if len(W.shape) == 4:
        W = np.squeeze(W)
        W = W.reshape((W.shape[0], W.shape[1], W.shape[2]*W.shape[3])) 
        fig, axs = plt.subplots(5,5, figsize=(8,8))
        fig.subplots_adjust(hspace = .5, wspace=.001)
        axs = axs.ravel()
        for i in range(25):
            axs[i].imshow(W[:,:,i])
            axs[i].set_title(str(i))

可以解决你的问题（只有卷积层）

【讨论】：

这种方法可以适用于 Conv1D 的情况吗？我试过了，它得到了一个空图像......我希望能够看到我正在研究的一维向量在中间层做了什么...... 在Conv1D中，每一层的核/权重是Rank1或Rank0张量，所以没有必要绘制它。我认为....mathworld.wolfram.com/TensorRank.html 有没有办法发现在一维卷积中被识别的总体趋势？我尝试绘制偏差。例如，如果训练一个二元分类器并且我绘制偏差并看到一条振荡线——这是否意味着我的数据具有规则模式——这可能会区分带有“1 标签”和“0s”的行？ @Merdan Memtimin 偏置可以理解为对应层的权重分布中心。换句话说，假设您的权重遵循正态分布，偏差更像是它们的平均值，所以，是的，偏差代表您的数据的某种模式，但不是主要部分，因为您可以训练您的 Deep净无偏见，但效率会低。回到你的问题，如果你能够以高斯投影你的权重（每一层），并且如果它们显示出多方差，那么我认为你能够证明你的观点。我建议你参考。 tf.summary.histogram @which_command 作为补充，请检查：***.com/questions/42315202/…@which_command

以上是关于keras模型中卷积层的可视化的主要内容，如果未能解决你的问题，请参考以下文章

卷积层（CNN）如何在 keras 中工作？

inception模型和卷积层的残差连接的keras实现

visualization of filters keras 基于Keras的卷积神经网络（CNN）可视化

visualization of filters keras 基于Keras的卷积神经网络（CNN）可视化

人工智能--卷积结果可视化

深度学习神经网络模型可视化的两个方法。