keras_3-1
Posted ls1314
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了keras_3-1相关的知识,希望对你有一定的参考价值。
0. Review
Computation graph
TensorFlow separates definition of computations from their execution
Phase 1: assemble a graph
Phase 2: use a session to execute operations in the graph.
TensorBoard
tf.constant and tf.Variable
Constant values are stored in the graph definition
Sessions allocate memory to store variable values
tf.placeholder and feed_dict
Feed values into placeholders with a dictionary (feed_dict)
Easy to use but poor performance
Avoid lazy loading
- Separate the assembling of graph and executing ops
- Use Python attribute to ensure a function is only loaded the first time it’s called
1. Linear regression on birth_life data
Phase 1: Assemble our graph
- Read in data
- Create placeholders for inputs and labels
- Create weight and bias
- Inference
- Specify loss function
- Create optimizer
Phase 2: Train our model
- Initialize variables
- Run optimizer
在完成phase 1之后,使用tf.summary创建graph的log文件,以备tensorboard可视化
Write log files using a FileWriter:
writer = tf.summary.FileWriter(‘./graphs/linear_reg‘, sess.graph)
- See it on TensorBoard
- run code
- execute tensorboard command
2. Control Flow
e.g. Implementing Huber loss
description:
code:
def huber_loss(labels, predictions, delta=14.0): residual = tf.abs(labels - predictions) def f1(): return 0.5 * tf.square(residual) def f2(): return delta * residual - 0.5 * tf.square(delta) return tf.cond(residual < delta, f1, f2)
TF Control Flow-table
Control Flow Ops tf.group, tf.count_up_to, tf.cond, tf.case, tf.while_loop, ... Comparison Ops tf.equal, tf.not_equal, tf.less, tf.greater, tf.where, ... Logical Ops tf.logical_and, tf.logical_not, tf.logical_or, tf.logical_xor Debugging Ops tf.is_finite, tf.is_inf, tf.is_nan, tf.Assert, tf.Print, ...
3. tf.data
Placeholder
Pro: put the data processing outside TensorFlow, making it easy to do in Python
Cons: users often end up processing their data in a single thread and creating data bottleneck that slows execution down.
tf.data:
Instead of doing inference with placeholders and feeding in data later, do inference directly with data
tf.data.Dataset
tf.data.Iterator
tf.data.Dataset
from variables
tf.data.Dataset.from_tensor_slices((features, labels)) tf.data.Dataset.from_generator(gen, output_types, output_shapes) dataset = tf.data.Dataset.from_tensor_slices((data[:,0], data[:,1]))
from file
tf.data.TextLineDataset(filenames) tf.data.FixedLengthRecordDataset(filenames) tf.data.TFRecordDataset(filenames)
tf.data.Iterator
Create an iterator to iterate through samples in Dataset
e.g.
iterator = dataset.make_one_shot_iterator() # Iterates through the dataset exactly once. No need to initialization. iterator = dataset.make_initializable_iterator() Iterates through the dataset as many times as we want. Need to initialize with each epoch. # iterator = dataset.make_one_shot_iterator() X, Y = iterator.get_next() # X is the birth rate, Y is the life expectancy with tf.Session() as sess: print(sess.run([X, Y])) # >> [1.822, 74.82825] print(sess.run([X, Y])) # >> [3.869, 70.81949] print(sess.run([X, Y])) # >> [3.911, 72.15066] # iterator = dataset.make_initializable_iterator() ... for i in range(100): sess.run(iterator.initializer) total_loss = 0 try: while True: sess.run([optimizer]) except tf.errors.OutOfRangeError: pass
Handling data in TensorFlow
e.g.
dataset = dataset.shuffle(1000) dataset = dataset.repeat(100) dataset = dataset.batch(128) dataset = dataset.map(lambda x: tf.one_hot(x, 10)) # convert each elem of dataset to one_hot vector
- Does tf.data really perform better?
- Should we always use tf.data?
- For prototyping, feed dict can be faster and easier to write (pythonic)
- tf.data is tricky to use when you have complicated preprocessing or multiple data sources
- NLP data is normally just a sequence of integers. In this case, transferring the data over to GPU is pretty quick, so the speedup of tf.data isn‘t that large
How does TensorFlow know what variables to update? Optimizer
4. Optimizers, gradients
e.g.
optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01).minimize(loss) _, l = sess.run([optimizer, loss], feed_dict={X: x, Y:y}) # Session looks at all trainable variables that loss depends on and update them # 比如weight,bias,gradiets_1丢到GradientDescent(),但是只要weights和bias是trainable
Trainable variables
tf.Variable(initial_value=None, trainable=True,...) # Specify if a variable should be trained or not # By default, all variables are trainable
List of optimizers in TF
tf.train.GradientDescentOptimizer tf.train.AdagradOptimizer tf.train.MomentumOptimizer tf.train.AdamOptimizer tf.train.FtrlOptimizer tf.train.RMSPropOptimizer ... # #“Advanced” optimizers work better when tuned, but are generally harder to tune
- question:
- How to know that our model is correct?
- How to improve our model?
5. Logistic regression on MNIST
MNIST Database: Each image is a 28x28 array, flattened out to be a 1-d tensor of size 784.
X: image of a handwritten digit
Y: the digit value
Recognize the digit in the image
Model: Inference:
Y_predicted = softmax(X * w + b)
Cross entropy loss: -log(Y_predicted)
Process data:
from tensorflow.examples.tutorials.mnist import input_data MNIST = input_data.read_data_sets('data/mnist', one_hot=True) # MNIST.train: 55,000 examples # MNIST.validation: 5,000 examples # MNIST.test: 10,000 examples # No immediate way to convert Python generators to tf.data.Dataset # mnist_folder = 'data/mnist' utils.download_mnist(mnist_folder) train, val, test = utils.read_mnist(mnist_folder, flatten=True) # # train_data = tf.data.Dataset.from_tensor_slices(train) train_data = train_data.shuffle(10000) # optional test_data = tf.data.Dataset.from_tensor_slices(test) # # iterator = train_data.make_initializable_iterator() img, label = iterator.get_next() # problem: # Can only do inference with train_data. # Need to build another subgraph with another iterator for test_data!!! # so improvement。。。 # # 因为这里的train_data,test_data的type及shape都是相同,同一个dataset来的,只是有不同用途,所以可以共用一个iterator的base class iterator = tf.data.Iterator.from_structure(train_data.output_types, train_data.output_shapes) img, label = iterator.get_next() train_init = iterator.make_initializer(train_data) # initializer for train_data test_init = iterator.make_initializer(test_data) # initializer for train_data # # # Initialize iterator with the dataset you want with tf.Session() as sess: ... for i in range(n_epochs): sess.run(train_init) # use train_init during training loop try: while True: _, l = sess.run([optimizer, loss]) except tf.errors.OutOfRangeError: pass # # test the model sess.run(test_init) # use test_init during testing try: while True: sess.run(accuracy) except tf.errors.OutOfRangeError: pass
Phase 1: Assemble our graph
Step 1: Read in data
Step 2: Create datasets and iterator
train_data = tf.data.Dataset.from_tensor_slices(train) train_data = train_data.shuffle(10000) # optional(train必须置乱D_train) train_data = train_data.batch(batch_size) test_data = tf.data.Dataset.from_tensor_slices(test) test_data = test_data.batch(batch_size) iterator = tf.data.Iterator.from_structure(train_data.output_types, train_data.output_shapes) img, label = iterator.get_next() train_init = iterator.make_initializer(train_data) test_init = iterator.make_initializer(test_data)
Step 3: Create weights and biases
use tf.get_variable()
Step 4: Build model to predict Y
logits = tf.matmul(img, w) + b # We don’t do softmax here, as we’ll do softmax together with cross_entropy loss. # It’s more efficient to compute gradients w.r.t. logits than w.r.t. softmax
Step 5: Specify loss function
entropy = tf.nn.softmax_cross_entropy_with_logits(labels=label, logits=logits) # 等价于先做softmax,再对概率化后的logits和本身就是0或1的labels做cross entropy loss = tf.reduce_mean(entropy)
Step 6: Create optimizer
tf.train.AdamOptimizer(learning_rate=0.01).minimize(loss)
Phase 2: Train our model
- Step 1: Initialize variables
- Step 2: Run optimizer op
TensorBoard it
6. Loss functions
- cross entropy: 写上我写的一个link
以上是关于keras_3-1的主要内容,如果未能解决你的问题,请参考以下文章
HCIA-AI_深度学习_TensorFlow2模块tf.keras基本用法
keras快速上手-基于python的深度学习实践_第8章_文字生成源代码
使用 keras.utils.Sequence 和 keras.model.fit_generator 时出现 KeyError。