TensorFlow 从入门到精通：tensorflow.nn 详解
Posted pypypy
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了TensorFlow 从入门到精通：tensorflow.nn 详解相关的知识，希望对你有一定的参考价值。
看过前面的例子，会发现实现深度神经网络需要使用 tensorflow.nn 这个核心模块。我们通过源码来一探究竟。
1 # Copyright 2015 Google Inc. All Rights Reserved. 2 # 3 # Licensed under the Apache License, Version 2.0 (the "License"); 4 # you may not use this file except in compliance with the License. 5 # You may obtain a copy of the License at 6 # 7 # http://www.apache.org/licenses/LICENSE-2.0 8 # 9 # Unless required by applicable law or agreed to in writing, software 10 # distributed under the License is distributed on an "AS IS" BASIS, 11 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 # See the License for the specific language governing permissions and 13 # limitations under the License. 14 # ============================================================================== 15 16 # pylint: disable=unused-import,g-bad-import-order 17 """## Activation Functions 18 19 The activation ops provide different types of nonlinearities for use in neural 20 networks. These include smooth nonlinearities (`sigmoid`, `tanh`, `elu`, 21 `softplus`, and `softsign`), continuous but not everywhere differentiable 22 functions (`relu`, `relu6`, and `relu_x`), and random regularization 23 (`dropout`). 24 25 All activation ops apply componentwise, and produce a tensor of the same 26 shape as the input tensor. 27 28 @@relu 29 @@relu6 30 @@elu 31 @@softplus 32 @@softsign 33 @@dropout 34 @@bias_add 35 @@sigmoid 36 @@tanh 37 38 ## Convolution 39 40 The convolution ops sweep a 2-D filter over a batch of images, applying the 41 filter to each window of each image of the appropriate size. The different 42 ops trade off between generic vs. specific filters: 43 44 * `conv2d`: Arbitrary filters that can mix channels together. 45 * `depthwise_conv2d`: Filters that operate on each channel independently. 46 * `separable_conv2d`: A depthwise spatial filter followed by a pointwise filter. 47 48 Note that although these ops are called "convolution", they are strictly 49 speaking "cross-correlation" since the filter is combined with an input window 50 without reversing the filter. For details, see [the properties of 51 cross-correlation](https://en.wikipedia.org/wiki/Cross-correlation#Properties). 52 53 The filter is applied to image patches of the same size as the filter and 54 strided according to the `strides` argument. `strides = [1, 1, 1, 1]` applies 55 the filter to a patch at every offset, `strides = [1, 2, 2, 1]` applies the 56 filter to every other image patch in each dimension, etc. 57 58 Ignoring channels for the moment, and assume that the 4-D `input` has shape 59 `[batch, in_height, in_width, ...]` and the 4-D `filter` has shape 60 `[filter_height, filter_width, ...]`, then the spatial semantics of the 61 convolution ops are as follows: first, according to the padding scheme chosen 62 as `‘SAME‘` or `‘VALID‘`, the output size and the padding pixels are computed. 63 For the `‘SAME‘` padding, the output height and width are computed as: 64 65 out_height = ceil(float(in_height) / float(strides[1])) 66 out_width = ceil(float(in_width) / float(strides[2])) 67 68 and the padding on the top and left are computed as: 69 70 pad_along_height = ((out_height - 1) * strides[1] + 71 filter_height - in_height) 72 pad_along_width = ((out_width - 1) * strides[2] + 73 filter_width - in_width) 74 pad_top = pad_along_height / 2 75 pad_left = pad_along_width / 2 76 77 Note that the division by 2 means that there might be cases when the padding on 78 both sides (top vs bottom, right vs left) are off by one. In this case, the 79 bottom and right sides always get the one additional padded pixel. For example, 80 when `pad_along_height` is 5, we pad 2 pixels at the top and 3 pixels at the 81 bottom. Note that this is different from existing libraries such as cuDNN and 82 Caffe, which explicitly specify the number of padded pixels and always pad the 83 same number of pixels on both sides. 84 85 For the `‘VALID`‘ padding, the output height and width are computed as: 86 87 out_height = ceil(float(in_height - filter_height + 1) / float(strides[1])) 88 out_width = ceil(float(in_width - filter_width + 1) / float(strides[2])) 89 90 and the padding values are always zero. The output is then computed as 91 92 output[b, i, j, :] = 93 sum_{di, dj} input[b, strides[1] * i + di - pad_top, 94 strides[2] * j + dj - pad_left, ...] * 95 filter[di, dj, ...] 96 97 where any value outside the original input image region are considered zero ( 98 i.e. we pad zero values around the border of the image). 99 100 Since `input` is 4-D, each `input[b, i, j, :]` is a vector. For `conv2d`, these 101 vectors are multiplied by the `filter[di, dj, :, :]` matrices to produce new 102 vectors. For `depthwise_conv_2d`, each scalar component `input[b, i, j, k]` 103 is multiplied by a vector `filter[di, dj, k]`, and all the vectors are 104 concatenated. 105 106 @@conv2d 107 @@depthwise_conv2d 108 @@separable_conv2d 109 @@conv2d_transpose 110 111 ## Pooling 112 113 The pooling ops sweep a rectangular window over the input tensor, computing a 114 reduction operation for each window (average, max, or max with argmax). Each 115 pooling op uses rectangular windows of size `ksize` separated by offset 116 `strides`. For example, if `strides` is all ones every window is used, if 117 `strides` is all twos every other window is used in each dimension, etc. 118 119 In detail, the output is 120 121 output[i] = reduce(value[strides * i:strides * i + ksize]) 122 123 where the indices also take into consideration the padding values. Please refer 124 to the `Convolution` section for details about the padding calculation. 125 126 @@avg_pool 127 @@max_pool 128 @@max_pool_with_argmax 129 130 ## Normalization 131 132 Normalization is useful to prevent neurons from saturating when inputs may 133 have varying scale, and to aid generalization. 134 135 @@l2_normalize 136 @@local_response_normalization 137 @@sufficient_statistics 138 @@normalize_moments 139 @@moments 140 141 ## Losses 142 143 The loss ops measure error between two tensors, or between a tensor and zero. 144 These can be used for measuring accuracy of a network in a regression task 145 or for regularization purposes (weight decay). 146 147 @@l2_loss 148 149 ## Classification 150 151 TensorFlow provides several operations that help you perform classification. 152 153 @@sigmoid_cross_entropy_with_logits 154 @@softmax 155 @@log_softmax 156 @@softmax_cross_entropy_with_logits 157 @@sparse_softmax_cross_entropy_with_logits 158 @@weighted_cross_entropy_with_logits 159 160 ## Embeddings 161 162 TensorFlow provides library support for looking up values in embedding 163 tensors. 164 165 @@embedding_lookup 166 @@embedding_lookup_sparse 167 168 ## Evaluation 169 170 The evaluation ops are useful for measuring the performance of a network. 171 Since they are nondifferentiable, they are typically used at evaluation time. 172 173 @@top_k 174 @@in_top_k 175 176 ## Candidate Sampling 177 178 Do you want to train a multiclass or multilabel model with thousands 179 or millions of output classes (for example, a language model with a 180 large vocabulary)? Training with a full Softmax is slow in this case, 181 since all of the classes are evaluated for every training example. 182 Candidate Sampling training algorithms can speed up your step times by 183 only considering a small randomly-chosen subset of contrastive classes 184 (called candidates) for each batch of training examples. 185 186 See our [Candidate Sampling Algorithms Reference] 187 (../../extras/candidate_sampling.pdf) 188 189 ### Sampled Loss Functions 190 191 TensorFlow provides the following sampled loss functions for faster training. 192 193 @@nce_loss 194 @@sampled_softmax_loss 195 196 ### Candidate Samplers 197 198 TensorFlow provides the following samplers for randomly sampling candidate 199 classes when using one of the sampled loss functions above. 200 201 @@uniform_candidate_sampler 202 @@log_uniform_candidate_sampler 203 @@learned_unigram_candidate_sampler 204 @@fixed_unigram_candidate_sampler 205 206 ### Miscellaneous candidate sampling utilities 207 208 @@compute_accidental_hits 209 210 """ 211 from __future__ import absolute_import 212 from __future__ import division 213 from __future__ import print_function 214 215 from six.moves import xrange # pylint: disable=redefined-builtin 216 217 from tensorflow.python.framework import dtypes 218 from tensorflow.python.framework import ops 219 from tensorflow.python.framework import tensor_shape 220 from tensorflow.python.ops import array_ops 221 from tensorflow.python.ops import candidate_sampling_ops 222 from tensorflow.python.ops import constant_op 223 from tensorflow.python.ops import control_flow_ops 224 from tensorflow.python.ops import embedding_ops 225 from tensorflow.python.ops import init_ops 226 from tensorflow.python.ops import math_ops 227 from tensorflow.python.ops import nn_grad 228 from tensorflow.python.ops import nn_ops 229 from tensorflow.python.ops import numerics 230 from tensorflow.python.ops import random_ops 231 from tensorflow.python.ops import rnn_cell 232 from tensorflow.python.ops import seq2seq 233 from tensorflow.python.ops import sparse_ops 234 from tensorflow.python.ops import variable_scope as vs 235 from tensorflow.python.ops.math_ops import sigmoid 236 from tensorflow.python.ops.math_ops import tanh 237 from tensorflow.python.util.all_util import make_all 238 239 # Bring more nn-associated functionality into this package. 240 # go/tf-wildcard-import 241 # pylint: disable=wildcard-import 242 from tensorflow.python.ops.nn_ops import * 243 from tensorflow.python.ops.candidate_sampling_ops import * 244 from tensorflow.python.ops.embedding_ops import * 245 from tensorflow.python.ops.rnn import * 246 # pylint: enable=wildcard-import 247 248 249 def sigmoid_cross_entropy_with_logits(logits, targets, name=None): 250 """Computes sigmoid cross entropy given `logits`. 251 252 Measures the probability error in discrete classification tasks in which each 253 class is independent and not mutually exclusive. For instance, one could 254 perform multilabel classification where a picture can contain both an elephant 255 and a dog at the same time. 256 257 For brevity, let `x = logits`, `z = targets`. The logistic loss is 258 259 z * -log(sigmoid(x)) + (1 - z) * -log(1 - sigmoid(x)) 260 = z * -log(1 / (1 + exp(-x))) + (1 - z) * -log(exp(-x) / (1 + exp(-x))) 261 = z * log(1 + exp(-x)) + (1 - z) * (-log(exp(-x)) + log(1 + exp(-x))) 262 = z * log(1 + exp(-x)) + (1 - z) * (x + log(1 + exp(-x)) 263 = (1 - z) * x + log(1 + exp(-x)) 264 = x - x * z + log(1 + exp(-x)) 265 266 To ensure stability and avoid overflow, the implementation uses 267 268 max(x, 0) - x * z + log(1 + exp(-abs(x))) 269 270 `logits` and `targets` must have the same type and shape. 271 272 Args: 273 logits: A `Tensor` of type `float32` or `float64`. 274 targets: A `Tensor` of the same type and shape as `logits`. 275 name: A name for the operation (optional). 276 277 Returns: 278 A `Tensor` of the same shape as `logits` with the componentwise 279 logistic losses. 280 281 Raises: 282 ValueError: If `logits` and `targets` do not have the same shape. 283 """ 284 with ops.op_scope([logits, targets], name, "logistic_loss") as name: 285 logits = ops.convert_to_tensor(logits, name="logits") 286 targets = ops.convert_to_tensor(targets, name="targets") 287 try: 288 targets.get_shape().merge_with(logits.get_shape()) 289 except ValueError: 290 raise ValueError( 291 "logits and targets must have the same shape (%s vs %s)" 292 % (logits.get_shape(), targets.get_shape())) 293 294 # The logistic loss formula from above is 295 # x - x * z + log(1 + exp(-x)) 296 # For x < 0, a more numerically stable formula is 297 # -x * z + log(1 + exp(x)) 298 # To avoid branching, we use the combined version 299 # max(x, 0) - x * z + log(1 + exp(-abs(x))) 300 return math_ops.add(nn_ops.relu(logits) - logits * targets, 301 math_ops.log(1 + math_ops.exp(-math_ops.abs(logits))), 302 name=name) 303 304 305 def weighted_cross_entropy_with_logits(logits, targets, pos_weight, 306 name=None): 307 """Computes a weighted cross entropy. 308 309 This is like `sigmoid_cross_entropy_with_logits()` except that `pos_weight`, 310 allows one to trade off recall and precision by up- or down-weighting the 311 cost of a positive error relative to a negative error. 312 313 The usual cross-entropy cost is defined as: 314 315 targets * -log(sigmoid(logits)) + (1 - targets) * -log(1 - sigmoid(logits)) 316 317 The argument `pos_weight` is used as a multiplier for the positive targets: 318 319 targets * -log(sigmoid(logits)) * pos_weight + 320 (1 - targets) * -log(1 - sigmoid(logits)) 321 322 For brevity, let `x = logits`, `z = targets`, `q = pos_weight`. 323 The loss is: 324 325 qz * -log(sigmoid(x)) + (1 - z) * -log(1 - sigmoid(x)) 326 = qz * -log(1 / (1 + exp(-x))) + (1 - z) * -log(exp(-x) / (1 + exp(-x))) 327 = qz * log(1 + exp(-x)) + (1 - z) * (-log(exp(-x)) + log(1 + exp(-x))) 328 = qz * log(1 + exp(-x)) + (1 - z) * (x + log(1 + exp(-x)) 329 = (1 - z) * x + (qz + 1 - z) * log(1 + exp(-x)) 330 = (1 - z) * x + (1 + (q - 1) * z) * log(1 + exp(-x)) 331 332 Setting `l = (1 + (q - 1) * z)`, to ensure stability and avoid overflow, 333 the implementation uses 334 335 (1 - z) * x + l * (log(1 + exp(-abs(x))) + max(-x, 0)) 336 337 `logits` and `targets` must have the same type and shape. 338 339 Args: 340 logits: A `Tensor` of type `float32` or `float64`. 341 targets: A `Tensor` of the same type and shape as `logits`. 342 pos_weight: A coefficient to use on the positive examples. 343 name: A name for the operation (optional). 344 345 Returns: 346 A `Tensor` of the same shape as `logits` with the componentwise 347 weightedlogistic losses. 348 349 Raises: 350 ValueError: If `logits` and `targets` do not have the same shape. 351 """ 352 with ops.op_scope([logits, targets], name, "logistic_loss") as name: 353 logits = ops.convert_to_tensor(logits, name="logits") 354 targets = ops.convert_to_tensor(targets, name="targets") 355 try: 356 targets.get_shape().merge_with(logits.get_shape()) 357 except ValueError: 358 raise ValueError( 359 "logits and targets must have the same shape (%s vs %s)" 360 % (logits.get_shape(), targets.get_shape())) 361 362 # The logistic loss formula from above is 363 # (1 - z) * x + (1 + (q - 1) * z) * log(1 + exp(-x)) 364 # For x < 0, a more numerically stable formula is 365 # (1 - z) * x + (1 + (q - 1) * z) * log(1 + exp(x)) - l * x 366 # To avoid branching, we use the combined version 367 # (1 - z) * x + l * (log(1 + exp(-abs(x))) + max(-x, 0)) 368 log_weight = 1 + (pos_weight - 1) * targets 369 return math_ops.add( 370 (1 - targets) * logits, 371 log_weight * (math_ops.log(1 + math_ops.exp(-math_ops.abs(logits))) + 372 nn_ops.relu(-logits)), 373 name=name) 374 375 376 def relu_layer(x, weights, biases, name=None): 377 """Computes Relu(x * weight + biases). 378 379 Args: 380 x: a 2D tensor. Dimensions typically: batch, in_units 381 weights: a 2D tensor. Dimensions typically: in_units, out_units 382 biases: a 1D tensor. Dimensions: out_units 383 name: A name for the operation (optional). If not specified 384 "nn_relu_layer" is used. 385 386 Returns: 387 A 2-D Tensor computing relu(matmul(x, weights) + biases). 388 Dimensions typically: batch, out_units. 389 """ 390 with ops.op_scope([x, weights, biases], name, "relu_layer") as name: 391 x = ops.convert_to_tensor(x, name="x") 392 weights = ops.convert_to_tensor(weights, name="weights") 393 biases = ops.convert_to_tensor(biases, name="biases") 394 xw_plus_b = nn_ops.bias_add(math_ops.matmul(x, weights), biases) 395 return nn_ops.relu(xw_plus_b, name=name) 396 397 398 def l2_normalize(x, dim, epsilon=1e-12, name=None): 399 """Normalizes along dimension `dim` using an L2 norm. 400 401 For a 1-D tensor with `dim = 0`, computes 402 403 output = x / sqrt(max(sum(x**2), epsilon)) 404 405 For `x` with more dimensions, independently normalizes each 1-D slice along 406 dimension `dim`. 407 408 Args: 409 x: A `Tensor`. 410 dim: Dimension along which to normalize. 411 epsilon: A lower bound value for the norm. Will use `sqrt(epsilon)` as the 412 divisor if `norm < sqrt(epsilon)`. 413 name: A name for this operation (optional). 414 415 Returns: 416 A `Tensor` with the same shape as `x`. 417 """ 418 with ops.op_scope([x], name, "l2_normalize") as name: 419 x = ops.convert_to_tensor(x, name="x") 420 square_sum = math_ops.reduce_sum(math_ops.square(x), [dim], keep_dims=True) 421 x_inv_norm = math_ops.rsqrt(math_ops.maximum(square_sum, epsilon)) 422 return math_ops.mul(x, x_inv_norm, name=name) 423 424 425 def zero_fraction(value, name=None): 426 """Returns the fraction of zeros in `value`. 427 428 If `value` is empty, the result is `nan`. 429 430 This is useful in summaries to measure and report sparsity. For example, 431 432 z = tf.Relu(...) 433 summ = tf.scalar_summary(‘sparsity‘, tf.nn.zero_fraction(z)) 434 435 Args: 436 value: A tensor of numeric type. 437 name: A name for the operation (optional). 438 439 Returns: 440 The fraction of zeros in `value`, with type `float32`. 441 """ 442 with ops.op_scope([value], name, "zero_fraction"): 443 value = ops.convert_to_tensor(value, name="value") 444 zero = constant_op.constant(0, dtype=value.dtype, name="zero") 445 return math_ops.reduce_mean(math_ops.cast(math_ops.equal(value, zero), 446 dtypes.float32)) 447 448 449 def depthwise_conv2d(input, filter, strides, padding, name=None): 450 """Depthwise 2-D convolution. 451 452 Given an input tensor of shape `[batch, in_height, in_width, in_channels]` 453 and a filter tensor of shape 454 `[filter_height, filter_width, in_channels, channel_multiplier]` 455 containing `in_channels` convolutional filters of depth 1, `depthwise_conv2d` 456 applies a different filter to each input channel (expanding from 1 channel 457 to `channel_multiplier` channels for each), then concatenates the results 458 together. The output has `in_channels * channel_multiplier` channels. 459 460 In detail, 461 462 output[b, i, j, k * channel_multiplier + q] = 463 sum_{di, dj} input[b, strides[1] * i + di, strides[2] * j + dj, k] * 464 filter[di, dj, k, q] 465 466 Must have `strides[0] = strides[3] = 1`. For the most common case of the 467 same horizontal and vertical strides, `strides = [1, stride, stride, 1]`. 468 469 Args: 470 input: 4-D with shape `[batch, in_height, in_width, in_channels]`. 471 filter: 4-D with shape 472 `[filter_height, filter_width, in_channels, channel_multiplier]`. 473 strides: 1-D of size 4. The stride of the sliding window for each 474 dimension of `input`. 475 padding: A string, either `‘VALID‘` or `‘SAME‘`. The padding algorithm. 476 name: A name for this operation (optional). 477 478 Returns: 479 A 4-D `Tensor` of shape 480 `[batch, out_height, out_width, in_channels * channel_multiplier].` 481 """ 482 with ops.op_scope([input, filter], name, "depthwise") as name: 483 input = ops.convert_to_tensor(input, name="tensor_in") 484 filter = ops.convert_to_tensor(filter, name="filter_in") 485 # A shape is required to statically compute the number of separable filters. 486 if filter.get_shape().ndims is not None: 487 assert len(filter.get_shape()) == 4 488 in_channels = filter.get_shape()[2] 489 # Sanity checks, if shape information is available for the inputs. 490 if input.get_shape().ndims is not None: 491 assert len(input.get_shape()) == 4 492 assert input.get_shape()[3] == in_channels, ( 493 "Mismatched input depth %d and number of depthwise filters %d." % ( 494 input.get_shape()[3].value, in_channels)) 495 else: 496 assert input.get_shape().ndims is not None, ( 497 "Either tensor must provide static shape information.") 498 assert input.get_shape().ndims == 4 499 in_channels = input.get_shape()[3] 500 501 if in_channels == 1: 502 return nn_ops.conv2d(input, filter, strides, padding, name=name) 503 else: 504 # Create one separate convolution per channel. 505 convs = [] 506 for channel in xrange(in_channels): 507 with ops.name_scope("depth%d" % channel) as channel_scope: 508 t_in = array_ops.slice(input, [0, 0, 0, channel], [-1, -1, -1, 1], 509 name="slice_inputs") 510 f_in = array_ops.slice(filter, [0, 0, channel, 0], [-1, -1, 1, -1], 511 name="slice_params") 512 convs.append(nn_ops.conv2d(t_in, f_in, 513 strides, padding, name=channel_scope)) 514 # Concatenate the per-channel convolutions along the channel dimension. 515 return array_ops.concat(3, convs, name=name) 516 517 518 def separable_conv2d(input, depthwise_filter, pointwise_filter, strides, 519 padding, 520 name=None): 521 """2-D convolution with separable filters. 522 523 Performs a depthwise convolution that acts separately on channels followed by 524 a pointwise convolution that mixes channels. Note that this is separability 525 between dimensions `[1, 2]` and `3`, not spatial separability between 526 dimensions `1` and `2`. 527 528 In detail, 529 530 output[b, i, j, k] = sum_{di, dj, q, r] 531 input[b, strides[1] * i + di, strides[2] * j + dj, q] * 532 depthwise_filter[di, dj, q, r] * 533 pointwise_filter[0, 0, q * channel_multiplier + r, k] 534 535 `strides` controls the strides for the depthwise convolution only, since 536 the pointwise convolution has implicit strides of `[1, 1, 1, 1]`. Must have 537 `strides[0] = strides[3] = 1`. For the most common case of the same 538 horizontal and vertical strides, `strides = [1, stride, stride, 1]`. 539 540 Args: 541 input: 4-D `Tensor` with shape `[batch, in_height, in_width, in_channels]`. 542 depthwise_filter: 4-D `Tensor` with shape 543 `[filter_height, filter_width, in_channels, channel_multiplier]`. 544 Contains `in_channels` convolutional filters of depth 1. 545 pointwise_filter: 4-D `Tensor` with shape 546 `[1, 1, channel_multiplier * in_channels, out_channels]`. Pointwise 547 filter to mix channels after `depthwise_filter` has convolved spatially. 548 strides: 1-D of size 4. The strides for the depthwise convolution for 549 each dimension of `input`. 550 padding: A string, either `‘VALID‘` or `‘SAME‘`. The padding algorithm. 551 name: A name for this operation (optional). 552 553 Returns: 554 A 4-D `Tensor` of shape `[batch, out_height, out_width, out_channels]`. 555 """ 556 with ops.op_scope([input, depthwise_filter, pointwise_filter], 557 name, "separable_conv2d") as name: 558 input = ops.convert_to_tensor(input, name="tensor_in") 559 depthwise_filter = ops.convert_to_tensor(depthwise_filter, 560 name="depthwise_filter") 561 pointwise_filter = ops.convert_to_tensor(pointwise_filter, 562 name="pointwise_filter") 563 564 if pointwise_filter.get_shape().ndims is not None: 565 assert len(pointwise_filter.get_shape()) == 4 566 assert pointwise_filter.get_shape()[0] == 1 567 assert pointwise_filter.get_shape()[1] == 1 568 if depthwise_filter.get_shape().ndims and input.get_shape().ndims: 569 channel_multiplier = depthwise_filter.get_shape()[3] 570 in_channels = input.get_shape()[3] 571 out_channels = pointwise_filter.get_shape()[3] 572 # This would mean the separable convolutions is over-parametrized. 573 assert channel_multiplier * in_channels < out_channels 574 # The layout of the ops in the graph are expected to be as follows: 575 # separable_conv2d // Conv2D op corresponding to the pointwise conv. 576 # separable_conv2d/depthwise // Concat op for the deptwise outputs. 577 # separable_conv2d/depthwise/depth0 // Conv2D op for depth 0 578 # separable_conv2d/depthwise/depth1 // Conv2D op for depth 1 579 # separable_conv2d/depthwise/depth2 // Conv2D op for depth 2 580 depthwise = depthwise_conv2d(input, depthwise_filter, strides, 581 padding, name="depthwise") 582 return nn_ops.conv2d(depthwise, pointwise_filter, [1, 1, 1, 1], 583 padding="VALID", name=name) 584 585 586 def sufficient_statistics(x, axes, shift=True, keep_dims=False, name=None): 587 """Calculate the sufficient statistics for the mean and variance of `x`. 588 589 These sufficient statistics are computed using the one pass algorithm on 590 an input that‘s optionally shifted using the value of the 1st element in `x`. 591 See: 592 https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Computing_shifted_data 593 594 Args: 595 x: A `Tensor`. 596 axes: Array of ints. Axes along which to compute mean and variance. 597 shift: If true, shift the data to provide more numerically stable results. 598 keep_dims: produce statistics with the same dimensionality as the input. 599 name: Name used to scope the operations that compute the sufficient stats. 600 601 Returns: 602 Four `Tensor` objects of the same type as `x`: 603 * the count (number of elements to average over). 604 * the (possibly shifted) sum of the elements in the array. 605 * the (possibly shifted) sum of squares of the elements in the array. 606 * the shift by which the mean must be corrected or None if `shift` is False. 607 """ 608 with ops.op_scope([x, axes], name, "sufficient_statistics"): 609 x = ops.convert_to_tensor(x, name="x") 610 x_shape = x.get_shape() 611 if x_shape.is_fully_defined(): 612 counts = 1 613 m_shape = [] 614 for d in xrange(x_shape.ndims): 615 dim = x_shape[d].value 616 if d in set(axes): 617 counts *= dim 618 dim = 1 619 m_shape.append(dim) 620 counts = constant_op.constant(counts, dtype=x.dtype) 621 else: # shape needs to be inferred at runtime. 622 x_shape = array_ops.shape(x) 623 select_axes = sparse_ops.sparse_to_dense(axes, array_ops.shape(x_shape), 624 True, False) 625 m_shape = math_ops.select(select_axes, array_ops.ones_like(x_shape), 626 x_shape) 627 counts = math_ops.cast( 628 math_ops.reduce_prod(x_shape / m_shape), 629 x.dtype, 630 name="count") 631 if shift: 632 shift_value = array_ops.slice(x, array_ops.zeros_like(m_shape), m_shape) 633 m_ss = math_ops.sub(x, shift_value) 634 v_ss = math_ops.squared_difference(x, shift_value) 635 if keep_dims: 636 shift_value = array_ops.identity(shift_value, name="shift") 637 else: 638 shift_value = array_ops.squeeze(shift_value, 639 squeeze_dims=axes, 640 name="shift") 641 else: # not shift. 642 m_ss = x 643 v_ss = math_ops.square(x) 644 shift_value = None 645 m_ss = math_ops.reduce_sum(m_ss, axes, keep_dims=keep_dims, name="mean_ss") 646 v_ss = math_ops.reduce_sum(v_ss, axes, keep_dims=keep_dims, name="var_ss") 647 return counts, m_ss, v_ss, shift_value 648 649 650 def normalize_moments(counts, mean_ss, variance_ss, shift, name=None): 651 """Calculate the mean and variance of based on the sufficient statistics. 652 653 Args: 654 counts: A `Tensor` containing a the total count of the data (one value). 655 mean_ss: A `Tensor` containing the mean sufficient statistics: the (possibly 656 shifted) sum of the elements to average over. 657 variance_ss: A `Tensor` containing the variance sufficient statistics: the 658 (possibly shifted) squared sum of the data to compute the variance over. 659 shift: A `Tensor` containing the value by which the data is shifted for 660 numerical stability, or `None` if no shift was performed. 661 name: Name used to scope the operations that compute the moments. 662 663 Returns: 664 Two `Tensor` objects: `mean` and `variance`. 665 """ 666 with ops.op_scope([counts, mean_ss, variance_ss, shift], name, "normalize"): 667 divisor = math_ops.inv(counts, name="divisor") 668 if shift is not None: 669 shifted_mean = math_ops.mul(mean_ss, divisor, name="shifted_mean") 670 mean = math_ops.add(shifted_mean, shift, name="mean") 671 else: # no shift. 672 shifted_mean = math_ops.mul(mean_ss, divisor, name="mean") 673 mean = shifted_mean 674 variance = math_ops.sub( 675 math_ops.mul(variance_ss, divisor), 676 math_ops.square(shifted_mean), 677 name="variance") 678 return (mean, variance) 679 680 681 def moments(x, axes, name=None, keep_dims=False): 682 """Calculate the mean and variance of `x`. 683 684 The mean and variance are calculated by aggregating the contents of `x` 685 across `axes`. If `x` is 1-D and `axes = [0]` this is just the mean 686 and variance of a vector. 687 688 When using these moments for batch normalization (see 689 `tf.nn.batch_normalization`): 690 * for so-called "global normalization", used with convolutional filters with 691 shape `[batch, height, width, depth]`, pass `axes=[0, 1, 2]`. 692 * for simple batch normalization pass `axes=[0]` (batch only). 693 694 Args: 695 x: A `Tensor`. 696 axes: array of ints. Axes along which to compute mean and 697 variance. 698 keep_dims: produce moments with the same dimensionality as the input. 699 name: Name used to scope the operations that compute the moments. 700 701 Returns: 702 Two `Tensor` objects: `mean` and `variance`. 703 """ 704 with ops.op_scope([x, axes], name, "moments"): 705 counts, m_ss, v_ss, shift = sufficient_statistics(x, 706 axes, 707 keep_dims=keep_dims, 708 name=name) 709 return normalize_moments(counts, m_ss, v_ss, shift, name=name) 710 711 712 def batch_normalization(x, 713 mean, 714 variance, 715 offset, 716 scale, 717 variance_epsilon, 718 name=None): 719 """Batch normalization. 720 721 As described in http://arxiv.org/abs/1502.03167. 722 Normalizes a tensor by `mean` and `variance`, and applies (optionally) a 723 `scale` \\(gamma\\) to it, as well as an `offset` \\(\beta\\): 724 725 \\(\frac{gamma(x-mu)}{sigma}+\beta\\) 726 727 `mean`, `variance`, `offset` and `scale` are all expected to be of one of two 728 shapes: 729 * In all generality, they can have the same number of dimensions as the 730 input `x`, with identical sizes as `x` for the dimensions that are not 731 normalized over (the ‘depth‘ dimension(s)), and dimension 1 for the 732 others which are being normalized over. 733 `mean` and `variance` in this case would typically be the outputs of 734 `tf.nn.moments(..., keep_dims=True)` during training, or running averages 735 thereof during inference. 736 * In the common case where the ‘depth‘ dimension is the last dimension in 737 the input tensor `x`, they may be one dimensional tensors of the same 738 size as the ‘depth‘ dimension. 739 This is the case for example for the common `[batch, depth]` layout of 740 fully-connected layers, and `[batch, height, width, depth]` for 741 convolutions. 742 `mean` and `variance` in this case would typically be the outputs of 743 `tf.nn.moments(..., keep_dims=False)` during training, or running averages 744 thereof during inference. 745 746 Args: 747 x: Input `Tensor` of arbitrary dimensionality. 748 mean: A mean `Tensor`. 749 variance: A variance `Tensor`. 750 offset: An offset `Tensor`, often denoted \\(\beta\\) in equations, or 751 None. If present, will be added to the normalized tensor. 752 scale: A scale `Tensor`, often denoted \\(gamma\\) in equations, or 753 `None`. If present, the scale is applied to the normalized tensor. 754 variance_epsilon: A small float number to avoid dividing by 0. 755 name: A name for this operation (optional). 756 757 Returns: 758 the normalized, scaled, offset tensor. 759 """ 760 with ops.op_scope([x, mean, variance, scale, offset], name, "batchnorm"): 761 inv = math_ops.rsqrt(variance + variance_epsilon) 762 if scale is not None: 763 inv *= scale 764 return x * inv + ( 765 offset - mean * inv if offset is not None else -mean * inv) 766 767 768 def batch_norm_with_global_normalization(t, 769 m, 770 v, 771 beta, 772 gamma, 773 variance_epsilon, 774 scale_after_normalization, 775 name=None): 776 """Batch normalization. 777 778 This op is deprecated. See `tf.nn.batch_normalization`. 779 780 Args: 781 t: A 4D input Tensor. 782 m: A 1D mean Tensor with size matching the last dimension of t. 783 This is the first output from tf.nn.moments, 784 or a saved moving average thereof. 785 v: A 1D variance Tensor with size matching the last dimension of t. 786 This is the second output from tf.nn.moments, 787 or a saved moving average thereof. 788 beta: A 1D beta Tensor with size matching the last dimension of t. 789 An offset to be added to the normalized tensor. 790 gamma: A 1D gamma Tensor with size matching the last dimension of t. 791 If "scale_after_normalization" is true, this tensor will be multiplied 792 with the normalized tensor. 793 variance_epsilon: A small float number to avoid dividing by 0. 794 scale_after_normalization: A bool indicating whether the resulted tensor 795 needs to be multiplied with gamma. 796 name: A name for this operation (optional). 797 798 Returns: 799 A batch-normalized `t`. 800 """ 801 return batch_normalization(t, m, v, beta, gamma if scale_after_normalization 802 else None, variance_epsilon, name) 803 804 805 def _sum_rows(x): 806 """Returns a vector summing up each row of the matrix x.""" 807 # _sum_rows(x) is equivalent to math_ops.reduce_sum(x, 1) when x is 808 # a matrix. The gradient of _sum_rows(x) is more efficient than 809 # reduce_sum(x, 1)‘s gradient in today‘s implementation. Therefore, 810 # we use _sum_rows(x) in the nce_loss() computation since the loss 811 # is mostly used for training. 812 cols = array_ops.shape(x)[1] 813 ones_shape = array_ops.pack([cols, 1]) 814 ones = array_ops.ones(ones_shape, x.dtype) 815 return array_ops.reshape(math_ops.matmul(x, ones), [-1]) 816 817 818 def _compute_sampled_logits(weights, biases, inputs, labels, num_sampled, 819 num_classes, num_true=1, 820 sampled_values=None, 821 subtract_log_q=True, 822 remove_accidental_hits=False, 823 partition_strategy="mod", 824 name=None): 825 """Helper function for nce_loss and sampled_softmax_loss functions. 826 827 Computes sampled output training logits and labels suitable for implementing 828 e.g. noise-contrastive estimation (see nce_loss) or sampled softmax (see 829 sampled_softmax_loss). 830 831 Note: In the case where num_true > 1, we assign to each target class 832 the target probability 1 / num_true so that the target probabilities 833 sum to 1 per-example. 834 835 Args: 836 weights: A `Tensor` of shape `[num_classes, dim]`, or a list of `Tensor` 837 objects whose concatenation along dimension 0 has shape 838 `[num_classes, dim]`. The (possibly-partitioned) class embeddings. 839 biases: A `Tensor` of shape `[num_classes]`. The class biases. 840 inputs: A `Tensor` of shape `[batch_size, dim]`. The forward 841 activations of the input network. 842 labels: A `Tensor` of type `int64` and shape `[batch_size, 843 num_true]`. The target classes. Note that this format differs from 844 the `labels` argument of `nn.softmax_cross_entropy_with_logits`. 845 num_sampled: An `int`. The number of classes to randomly sample per batch. 846 num_classes: An `int`. The number of possible classes. 847 num_true: An `int`. The number of target classes per training example. 848 sampled_values: a tuple of (`sampled_candidates`, `true_expected_count`, 849 `sampled_expected_count`) returned by a `*_candidate_sampler` function. 850 (if None, we default to `log_uniform_candidate_sampler`) 851 subtract_log_q: A `bool`. whether to subtract the log expected count of 852 the labels in the sample to get the logits of the true labels. 853 Default is True. Turn off for Negative Sampling. 854 remove_accidental_hits: A `bool`. whether to remove "accidental hits" 855 where a sampled class equals one of the target classes. Default is 856 False. 857 partition_strategy: A string specifying the partitioning strategy, relevant 858 if `len(weights) > 1`. Currently `"div"` and `"mod"` are supported. 859 Default is `"mod"`. See `tf.nn.embedding_lookup` for more details. 860 name: A name for the operation (optional). 861 Returns: 862 out_logits, out_labels: `Tensor` objects each with shape 863 `[batch_size, num_true + num_sampled]`, for passing to either 864 `nn.sigmoid_cross_entropy_with_logits` (NCE) or 865 `nn.softmax_cross_entropy_with_logits` (sampled softmax). 866 """ 867 868 if not isinstance(weights, list): 869 weights = [weights] 870 871 with ops.op_scope( 872 weights + [biases, inputs, labels], name, "compute_sampled_logits"): 873 if labels.dtype != dtypes.int64: 874 labels = math_ops.cast(labels, dtypes.int64) 875 labels_flat = array_ops.reshape(labels, [-1]) 876 877 # Sample the negative labels. 878 # sampled shape: [num_sampled] tensor 879 # true_expected_count shape = [batch_size, 1] tensor 880 # sampled_expected_count shape = [num_sampled] tensor 881 if sampled_values is None: 882 sampled_values = candidate_sampling_ops.log_uniform_candidate_sampler( 883 true_classes=labels, 884 num_true=num_true, 885 num_sampled=num_sampled, 886 unique=True, 887 range_max=num_classes) 888 # NOTE: pylint cannot tell that ‘sampled_values‘ is a sequence 889 # pylint: disable=unpacking-non-sequence 890 sampled, true_expected_count, sampled_expected_count = sampled_values 891 # pylint: enable=unpacking-non-sequence 892 893 # labels_flat is a [batch_size * num_true] tensor 894 # sampled is a [num_sampled] int tensor 895 all_ids = array_ops.concat(0, [labels_flat, sampled]) 896 897 # weights shape is [num_classes, dim] 898 all_w = embedding_ops.embedding_lookup( 899 weights, all_ids, partition_strategy=partition_strategy) 900 all_b = embedding_ops.embedding_lookup(biases, all_ids) 901 # true_w shape is [batch_size * num_true, dim] 902 # true_b is a [batch_size * num_true] tensor 903 true_w = array_ops.slice( 904 all_w, [0, 0], array_ops.pack([array_ops.shape(labels_flat)[0], -1])) 905 true_b = array_ops.slice(all_b, [0], array_ops.shape(labels_flat)) 906 907 # inputs shape is [batch_size, dim] 908 # true_w shape is [batch_size * num_true, dim] 909 # row_wise_dots is [batch_size, num_true, dim] 910 dim = array_ops.shape(true_w)[1:2] 911 new_true_w_shape = array_ops.concat(0, [[-1, num_true], dim]) 912 row_wise_dots = math_ops.mul( 913 array_ops.expand_dims(inputs, 1), 914 array_ops.reshape(true_w, new_true_w_shape)) 915 # We want the row-wise dot plus biases which yields a 916 # [batch_size, num_true] tensor of true_logits. 917 dots_as_matrix = array_ops.reshape(row_wise_dots, 918 array_ops.concat(0, [[-1], dim])) 919 true_logits = array_ops.reshape(_sum_rows(dots_as_matrix), [-1, num_true]) 920 true_b = array_ops.reshape(true_b, [-1, num_true]) 921 true_logits += true_b 922 923 # Lookup weights and biases for sampled labels. 924 # sampled_w shape is [num_sampled, dim] 925 # sampled_b is a [num_sampled] float tensor 926 sampled_w = array_ops.slice( 927 all_w, array_ops.pack([array_ops.shape(labels_flat)[0], 0]), [-1, -1]) 928 sampled_b = array_ops.slice(all_b, array_ops.shape(labels_flat), [-1]) 929 930 # inputs has shape [batch_size, dim] 931 # sampled_w has shape [num_sampled, dim] 932 # sampled_b has shape [num_sampled] 933 # Apply X*W‘+B, which yields [batch_size, num_sampled] 934 sampled_logits = math_ops.matmul(inputs, 935 sampled_w, 936 transpose_b=True) + sampled_b 937 938 if remove_accidental_hits: 939 acc_hits = candidate_sampling_ops.compute_accidental_hits( 940 labels, sampled, num_true=num_true) 941 acc_indices, acc_ids, acc_weights = acc_hits 942 943 # This is how SparseToDense expects the indices. 944 acc_indices_2d = array_ops.reshape(acc_indices, [-1, 1]) 945 acc_ids_2d_int32 = array_ops.reshape(math_ops.cast( 946 acc_ids, dtypes.int32), [-1, 1]) 947 sparse_indices = array_ops.concat( 948 1, [acc_indices_2d, acc_ids_2d_int32], "sparse_indices") 949 # Create sampled_logits_shape = [batch_size, num_sampled] 950 sampled_logits_shape = array_ops.concat( 951 0, 952 [array_ops.shape(labels)[:1], array_ops.expand_dims(num_sampled, 0)]) 953 if sampled_logits.dtype != acc_weights.dtype: 954 acc_weights = math_ops.cast(acc_weights, sampled_logits.dtype) 955 sampled_logits += sparse_ops.sparse_to_dense( 956 sparse_indices, sampled_logits_shape, acc_weights, 957 default_value=0.0, validate_indices=False) 958 959 if subtract_log_q: 960 # Subtract log of Q(l), prior probability that l appears in sampled. 961 true_logits -= math_ops.log(true_expected_count) 962 sampled_logits -= math_ops.log(sampled_expected_count) 963 964 # Construct output logits and labels. The true labels/logits start at col 0. 965 out_logits = array_ops.concat(1, [true_logits, sampled_logits]) 966 # true_logits is a float tensor, ones_like(true_logits) is a float tensor 967 # of ones. We then divide by num_true to ensure the per-example labels sum 968 # to 1.0, i.e. form a proper probability distribution. 969 out_labels = array_ops.concat( 970 1, [array_ops.ones_like(true_logits) / num_true, 971 array_ops.zeros_like(sampled_logits)]) 972 973 return out_logits, out_labels 974 975 976 def nce_loss(weights, biases, inputs, labels, num_sampled, num_classes, 977 num_true=1, 978 sampled_values=None, 979 remove_accidental_hits=False, 980 partition_strategy="mod", 981 name="nce_loss"): 982 """Computes and returns the noise-contrastive estimation training loss. 983 984 See [Noise-contrastive estimation: A new estimation principle for 985 unnormalized statistical models] 986 (http://www.jmlr.org/proceedings/papers/v9/gutmann10a/gutmann10a.pdf). 987 Also see our [Candidate Sampling Algorithms Reference] 988 (../../extras/candidate_sampling.pdf) 989 990 Note: In the case where `num_true` > 1, we assign to each target class 991 the target probability 1 / `num_true` so that the target probabilities 992 sum to 1 per-example. 993 994 Note: It would be useful to allow a variable number of target classes per 995 example. We hope to provide this functionality in a future release. 996 For now, if you have a variable number of target classes, you can pad them 997 out to a constant number by either repeating them or by padding 998 with an otherwise unused class. 999 1000 Args: 1001 weights: A `Tensor` of shape `[num_classes, dim]`, or a list of `Tensor` 1002 objects whose concatenation along dimension 0 has shape 1003 [num_classes, dim]. The (possibly-partitioned) class embeddings. 1004 biases: A `Tensor` of shape `[num_classes]`. The class biases. 1005 inputs: A `Tensor` of shape `[batch_size, dim]`. The forward 1006 activations of the input network. 1007 labels: A `Tensor` of type `int64` and shape `[batch_size, 1008 num_true]`. The target classes. 1009 num_sampled: An `int`. The number of classes to randomly sample per batch. 1010 num_classes: An `int`. The number of possible classes. 1011 num_true: An `int`. The number of target classes per training example. 1012 sampled_values: a tuple of (`sampled_candidates`, `true_expected_count`, 1013 `sampled_expected_count`) returned by a `*_candidate_sampler` function. 1014 (if None, we default to `log_uniform_candidate_sampler`) 1015 remove_accidental_hits: A `bool`. Whether to remove "accidental hits" 1016 where a sampled class equals one of the target classes. If set to 1017 `True`, this is a "Sampled Logistic" loss instead of NCE, and we are 1018 learning to generate log-odds instead of log probabilities. See 1019 our [Candidate Sampling Algorithms Reference] 1020 (../../extras/candidate_sampling.pdf). 1021 Default is False. 1022 partition_strategy: A string specifying the partitioning strategy, relevant 1023 if `len(weights) > 1`. Currently `"div"` and `"mod"` are supported. 1024 Default is `"mod"`. See `tf.nn.embedding_lookup` for more details. 1025 name: A name for the operation (optional). 1026 1027 Returns: 1028 A `batch_size` 1-D tensor of per-example NCE losses. 1029 """ 1030 logits, labels = _compute_sampled_logits( 1031 weights, biases, inputs, labels, num_sampled, num_classes, 1032 num_true=num_true, 1033 sampled_values=sampled_values, 1034 subtract_log_q=True, 1035 remove_accidental_hits=remove_accidental_hits, 1036 partition_strategy=partition_strategy, 1037 name=name) 1038 sampled_losses = sigmoid_cross_entropy_with_logits(logits, 1039 labels, 1040 name="sampled_losses") 1041 # sampled_losses is batch_size x {true_loss, sampled_losses...} 1042 # We sum out true and sampled losses. 1043 return _sum_rows(sampled_losses) 1044 1045 1046 def sampled_softmax_loss(weights, biases, inputs, labels, num_sampled, 1047 num_classes, num_true=1, 1048 sampled_values=None, 1049 remove_accidental_hits=True, 1050 partition_strategy="mod", 1051 name="sampled_softmax_loss"): 1052 """Computes and returns the sampled softmax training loss. 1053 1054 This is a faster way to train a softmax classifier over a huge number of 1055 classes. 1056 1057 This operation is for training only. It is generally an underestimate of 1058 the full softmax loss. 1059 1060 At inference time, you can compute full softmax probabilities with the 1061 expression `tf.nn.softmax(tf.matmul(inputs, weights) + biases)`. 1062 1063 See our [Candidate Sampling Algorithms Reference] 1064 (../../extras/candidate_sampling.pdf) 1065 1066 Also see Section 3 of [Jean et al., 2014](http://arxiv.org/abs/1412.2007) 1067 ([pdf](http://arxiv.org/pdf/1412.2007.pdf)) for the math. 1068 1069 Args: 1070 weights: A `Tensor` of shape `[num_classes, dim]`, or a list of `Tensor` 1071 objects whose concatenation along dimension 0 has shape 1072 [num_classes, dim]. The (possibly-sharded) class embeddings. 1073 biases: A `Tensor` of shape `[num_classes]`. The class biases. 1074 inputs: A `Tensor` of shape `[batch_size, dim]`. The forward 1075 activations of the input network. 1076 labels: A `Tensor` of type `int64` and shape `[batch_size, 1077 num_true]`. The target classes. Note that this format differs from 1078 the `labels` argument of `nn.softmax_cross_entropy_with_logits`. 1079 num_sampled: An `int`. The number of classes to randomly sample per batch. 1080 num_classes: An `int`. The number of possible classes. 1081 num_true: An `int`. The number of target classes per training example. 1082 sampled_values: a tuple of (`sampled_candidates`, `true_expected_count`, 1083 `sampled_expected_count`) returned by a `*_candidate_sampler` function. 1084 (if None, we default to `log_uniform_candidate_sampler`) 1085 remove_accidental_hits: A `bool`. whether to remove "accidental hits" 1086 where a sampled class equals one of the target classes. Default is 1087 True. 1088 partition_strategy: A string specifying the partitioning strategy, relevant 1089 if `len(weights) > 1`. Currently `"div"` and `"mod"` are supported. 1090 Default is `"mod"`. See `tf.nn.embedding_lookup` for more details. 1091 name: A name for the operation (optional). 1092 1093 Returns: 1094 A `batch_size` 1-D tensor of per-example sampled softmax losses.
logits, labels = _compute_sampled_logits( weights, biases, inputs, labels, num_sampled, num_classes, num_true=num_true, sampled_values=sampled_values, subtract_log_q=True, remove_accidental_hits=remove_accidental_hits, partition_strategy=partition_strategy, name=name) sampled_losses = nn_ops.softmax_cross_entropy_with_logits(logits, labels) # sampled_losses is a [batch_size] tensor. return sampled_losses # TODO(cwhipkey): sigmoid and tanh should not be exposed from tf.nn. __all__ = make_all(__name__) __all__.append("zero_fraction") # documented in training.py # Modules whitelisted for reference through tf.nn. # TODO(cwhipkey): migrate callers to use the submodule directly. __all__.extend(["nn_ops", "rnn_cell", "seq2seq"]) # Symbols whitelisted for export without documentation. # TODO(cwhipkey): review these and move to contrib or expose through # documentation. __all__.extend([ "all_candidate_sampler", "batch_norm_with_global_normalization", "batch_normalization", "bidirectional_rnn", "conv2d_backprop_filter", "conv2d_backprop_input", "depthwise_conv2d_native", "dynamic_rnn", "lrn", "relu_layer", "rnn", "state_saving_rnn", "xw_plus_b", ])