tensorflow serving slim配置流程

Posted 帅气的小王子

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了tensorflow serving slim配置流程相关的知识,希望对你有一定的参考价值。

发现国内tensorflow serving方面的博文非常少,包括国外也不多,主要依靠官方文档,但官方文档有些碎片化并包含一些冗余,因此整合为一篇文章方便国人。

之前在我在生产环境下用tensorflow的方式是启动一个thrift服务,load进模型之后不断处理请求,但是在面对大批量数据时有些力不从心,对机器的压力很大,因此考虑使用tensorflow serving的方式部署服务。

服务调用的模型是slim下的inception v3 图像分类模型,对模型进行了迁移学习,得到了新的分类模型,运用该模型对单张图片或者批量图片做预测或者提取特征。

本文假设你已成功安装tensorflow,并利用slim的代码,迁移学习获得了自己的inception v3模型,本文主要阐述如何将该模型部署到tensorflow serving 上。

tensorflow 版本:r.1.3 , python版本:2.7.8 ,无GPU

Step1:安装tensorflow Serving r1.3

参考官方文档:https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/setup.md

前期准备:

安装bazel

pip install tensorflow-serving-api
pip install grpcio

安装过程:

  1.git clone --recurse-submodules https://github.com/tensorflow/serving

     cd serving

  2. cd tensorflow

   ./configure

 

You have bazel 0.7.0- (@non-git) installed.
Please specify the location of python. [Default is /data0/home/wang16/local/python/bin/python]:


Found possible Python library paths:
/data0/home/wang16/local/python/lib/python2.7/site-packages/
/data0/install/caffe/python/
/data0/home/wang16/simba/trunk/src/content_analysis/
/data0/home/wang16/local/python/lib/python2.7/site-packages
Please input the desired Python library path to use. Default is [/data0/home/wang16/local/python/lib/python2.7/site-packages/]

Do you wish to build TensorFlow with jemalloc as malloc support? [Y/n]: 
jemalloc as malloc support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Google Cloud Platform support? [Y/n]: n
No Google Cloud Platform support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Hadoop File System support? [Y/n]: n
No Hadoop File System support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Amazon S3 File System support? [Y/n]: n
No Amazon S3 File System support will be enabled for TensorFlow.

Do you wish to build TensorFlow with XLA JIT support? [y/N]: 
No XLA JIT support will be enabled for TensorFlow.

Do you wish to build TensorFlow with GDR support? [y/N]: 
No GDR support will be enabled for TensorFlow.

Do you wish to build TensorFlow with VERBS support? [y/N]: 
No VERBS support will be enabled for TensorFlow.

Do you wish to build TensorFlow with OpenCL support? [y/N]: 
No OpenCL support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: 
No CUDA support will be enabled for TensorFlow.

Do you wish to build TensorFlow with MPI support? [y/N]: 
No MPI support will be enabled for TensorFlow.

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]:


Add "--config=mkl" to your bazel command to build with MKL support.
Please note that MKL on MacOS or windows is still not supported.
If you would like to use a local MKL instead of downloading, please set the environment variable "TF_MKL_ROOT" every time before build.
Configuration finished


 

 cd ..

  3. bazel build -c opt --copt=-msse4.1 --copt=-msse4.2 --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-O3 tensorflow_serving/...(耗时较长)

这里如果是centos可能会报错:fatal error: stropts.h: No such file or directory

在/usr/include建一个空的文件。命名stropts.h就可以了。

  4. bazel test -c opt tensorflow_serving/...

 

Step2:修改代码以支持新的模型

参考文档:https://gyang274.github.io/docker-tensorflow-serving-slim/0x02b00.slim.inception.v4.htmlhttps://github.com/gyang274/docker-tensorflow-serving-slim

参考文档:https://www.tensorflow.org/serving/serving_inception

这部分是最难的,官方提供的示例代码很少,幸亏找到了一篇相关文档,照猫画虎,才算完成。

  1. 回到 serving 主目录,目录下有如下内容

 mkdir -p tf_checkpoints/slim/my_inception_v3

 拷贝迁移学习训练好的模型到该目录中,拷贝完目录内容如下:

  2.参照示例写模型导出代码:tensorflow_serving/example/my_inception_v3_saved_model.py

# Copyright 2016 Google Inc. All Rights Reserved.

#

# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at

#

# http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

# ==============================================================================

#!/usr/bin/env python2.7

"""Export inception v3 model given existing training checkpoints.

The model is exported as SavedModel with proper signatures that can be loaded by

standard tensorflow_model_server.

"""

import os.path

# This is a placeholder for a Google-internal import.

import tensorflow as tf

slim = tf.contrib.slim

# from preprocessing import inception_preprocessing

from nets import inception

tf.app.flags.DEFINE_string('checkpoint_dir', '/tmp/slim_inception_v3_train',

 """Directory where to read training checkpoints.""")

tf.app.flags.DEFINE_string('output_dir', '/tmp/slim_inception_v3_output',

 """Directory where to export inference model.""")

tf.app.flags.DEFINE_integer('model_version', 1,

 """Version number of the model.""")

tf.app.flags.DEFINE_integer('image_size', 299,

 """Needs to provide same value as in training.""")

FLAGS = tf.app.flags.FLAGS

NUM_CLASSES = 76

NUM_TOP_CLASSES = 5

WORKING_DIR = os.path.dirname(os.path.realpath(__file__))

 

def preprocess_image(image_buffer):

 """Preprocess JPEG encoded bytes to 3D float Tensor."""

 # Decode the string as an RGB JPEG.

 # Note that the resulting image contains an unknown height and width

 # that is set dynamically by decode_jpeg. In other words, the height

 # and width of image is unknown at compile-time.

 image = tf.image.decode_jpeg(image_buffer, channels=3)

 # After this point, all image pixels reside in [0,1)

 # until the very end, when they're rescaled to (-1, 1). The various

 # adjust_* ops all require this range for dtype float.

 image = tf.image.convert_image_dtype(image, dtype=tf.float32)

 # Crop the central region of the image with an area containing 87.5% of

 # the original image.

 image = tf.image.central_crop(image, central_fraction=0.875)

 # Resize the image to the original height and width.

 image = tf.expand_dims(image, 0)

 image = tf.image.resize_bilinear(

 image, [FLAGS.image_size, FLAGS.image_size], align_corners=False)

 image = tf.squeeze(image, [0])

 # Finally, rescale to [-1,1] instead of [0, 1)

 image = tf.subtract(image, 0.5)

 image = tf.multiply(image, 2.0)

 return image

 

def export():

 with tf.Graph().as_default():

 # build inference model

 # labels

 f = open('/data0/home/wang16/package/serving/tf_checkpoints/slim/my_inception_v3/labels.txt')

 lines = f.readlines()

 names = 0:'background'

 for line in lines:

 i,label = line.split(':')

 names[int(i)]=label

 

 names_tensor = tf.constant(

 names.values()

 )

 names_lookup_table = tf.contrib.lookup.index_to_string_table_from_tensor(

 names_tensor

 )

 # input transformation

 serialized_tf_example = tf.placeholder(

 tf.string, name='tf_example'

 )

 feature_configs =

 'image/encoded': tf.FixedLenFeature(

 shape=[], dtype=tf.string

 ),

 

 tf_example = tf.parse_example(

 serialized_tf_example, feature_configs

 )

 jpegs = tf_example['image/encoded']

 images = tf.map_fn(

 preprocess_image, jpegs, dtype=tf.float32

 )

 # run inference

 with slim.arg_scope(

 inception.inception_v3_arg_scope()

 ):

 # inception v3 models

 logits, end_points = inception.inception_v3(

 images, num_classes=NUM_CLASSES, is_training=False

 )

 # logits = tf.Print(logits, [logits])

 probs = tf.nn.softmax(logits)

 # transform output to topk result

 topk_probs, topk_indices = tf.nn.top_k(

 probs, NUM_TOP_CLASSES

 )

 topk_names = names_lookup_table.lookup(

 tf.to_int64(topk_indices)

 )

 init_fn = slim.assign_from_checkpoint_fn(

 os.path.join(FLAGS.checkpoint_dir, 'model.ckpt-100000'),

 slim.get_model_variables('InceptionV3')

 )

 # sess config

 config = tf.ConfigProto(

 # device_count =

 # 'GPU': 0

 # ,

 gpu_options=

 'allow_growth': 1,

 # 'per_process_gpu_memory_fraction': 0.01

 ,

 allow_soft_placement=True,

 log_device_placement=False,

 )

 with tf.Session(config=config) as sess:

 init_fn(sess)

 # init on 2017.08.05

 # prelogits = sess.graph.get_tensor_by_name(

 # 'InceptionV3/Logits/PreLogitsFlatten/Reshape:0'

 # )

 # update on 2017.10.22

 # note: looks like the graphdef is updated for slim-inception-v3

 print('Graph Node Tensor Name:')

 for node_tensor in tf.get_default_graph().as_graph_def().node:

 if str(node_tensor.name).startswith('InceptionV3/Logits'):

 print str(node_tensor.name)

 prelogits = sess.graph.get_tensor_by_name(

 'InceptionV3/Logits/Dropout_1b/Identity:0'

 )

 # an optional alternative

 # prelogits = end_points['PreLogitsFlatten']

 # export inference model.

 output_path = os.path.join(

 tf.compat.as_bytes(FLAGS.output_dir),

 tf.compat.as_bytes(str(FLAGS.model_version))

 )

 print 'Exporting trained model to', output_path

 builder = tf.saved_model.builder.SavedModelBuilder(output_path)

 # build the signature_def_map.

 predict_inputs_tensor_info = tf.saved_model.utils.build_tensor_info(

 jpegs

 )

 classes_output_tensor_info = tf.saved_model.utils.build_tensor_info(

 topk_names

 )

 scores_output_tensor_info = tf.saved_model.utils.build_tensor_info(

 topk_probs

 )

 prelogits_output_tensor_info = tf.saved_model.utils.build_tensor_info(

 prelogits

 )

 prediction_signature = (

 tf.saved_model.signature_def_utils.build_signature_def(

 inputs=

 'images': predict_inputs_tensor_info

 ,

 outputs=

 'classes': classes_output_tensor_info,

 'scores': scores_output_tensor_info,

 'prelogits': prelogits_output_tensor_info

 ,

 method_name=tf.saved_model.signature_constants.PREDICT_METHOD_NAME

 )

 )

 legacy_init_op = tf.group(

 tf.tables_initializer(), name='legacy_init_op'

 )

 builder.add_meta_graph_and_variables(

 sess, [tf.saved_model.tag_constants.SERVING],

 signature_def_map=

 'predict_images':

 prediction_signature,

 ,

 legacy_init_op=legacy_init_op

 )

 builder.save()

 print 'Successfully exported model to %s' % FLAGS.output_dir

 

def main(unused_argv=None):

 export()

 

if __name__ == '__main__':

 tf.app.run()


     参照示例写模型服务代码:tensorflow_serving/example/my_inception_v3_client.py

# Copyright 2016 Google Inc. All Rights Reserved.

#

# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at

#

# http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

# ==============================================================================

#!/usr/bin/env python2.7

"""Send JPEG image to tensorflow_model_server loaded with inception model.

"""

from __future__ import print_function

import urllib2

# This is a placeholder for a Google-internal import.

from grpc.beta import implementations

import tensorflow as tf

from tensorflow_serving.apis import predict_pb2

from tensorflow_serving.apis import prediction_service_pb2

 

tf.app.flags.DEFINE_string(

 'server', 'localhost:9000', 'PredictionService host:port'

)

tf.app.flags.DEFINE_string(

 'image_url', '', 'URL to image in JPEG format'

)

FLAGS = tf.app.flags.FLAGS

 

def main(_):

 host, port = FLAGS.server.split(':')

 channel = implementations.insecure_channel(host, int(port))

 stub = prediction_service_pb2.beta_create_PredictionService_stub(channel)

 # Send request

 # See prediction_service.proto for gRPC request/response details.

 image_bytes = urllib2.urlopen(FLAGS.image_url).read()

 request = predict_pb2.PredictRequest()

 request.model_spec.name = 'my_inception_v3'

 request.model_spec.signature_name = 'predict_images'

 request.inputs['images'].CopyFrom(

 tf.contrib.util.make_tensor_proto(

 image_bytes, shape=[1]

 )

 )

 result = stub.Predict(request, 60.0) # 60 secs timeout

 print(result)

 

if __name__ == '__main__':

 tf.app.run()


     参照参考文献修改:tensorflow_serving/example/BUILD

...

 py_binary(

 name = "my_inception_v3_saved_model",

 srcs = [

 "my_inception_v3_saved_model.py",

 ],

 srcs_version = "PY2AND3",

 deps = [

 "@slim_model//:dataset_factory",

 "@slim_model//:preprocessing_factory",

 "@slim_model//:nets",

 "@org_tensorflow//tensorflow:tensorflow_py",

 ],

)

py_binary(

 name = "my_inception_v3_client",

 srcs = [

 "my_inception_v3_client.py",

 ],

 srcs_version = "PY2AND3",

 deps = [

 "//tensorflow_serving/apis:predict_proto_py_pb2",

 "//tensorflow_serving/apis:prediction_service_proto_py_pb2",

 "@org_tensorflow//tensorflow:tensorflow_py",

 ],

)

...

    参照参考文献修改:/tensorflow_serving/workspace.bzl

...

 native.new_local_repository(

 name = "slim_model",

 path = "tf_models/research/slim",

 build_file = "tf_models/research/slim/BUILD",

 )

...

      修改代码: /tf_models/research/slim/BUILD,注释掉所有  "//tensorflow",

      编译代码:

      bazel build -c opt --copt=-msse4.1 --copt=-msse4.2 --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-O3 tensorflow_serving/...

      提取模型:      

    bazel-bin/tensorflow_serving/example/my_inception_v3_saved_model \\
      --checkpoint_dir=tf_checkpoints/slim/my_inception_v3 \\
      --output_dir=tf_servables/slim/my_inception_v3 \\
      --model_version=1 \\
      --image_size=299

      提取后的模型目录:

 

 

Step3:部署服务

   1、部署服务

    bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server --model_name=my_inception_v3 --model_base_path=$PWD/tf_servables/slim/my_inception_v3 --port=9000

   2、测试服务

    python my_inception_v3_client.py

  3、返回结果

outputs
key: "classes"
value
dtype: DT_STRING
tensor_shape
dim
size: 1

dim
size: 5


string_val: "BeautifulSignt\\n"
string_val: "Building\\n"
string_val: "Cartoon\\n"
string_val: "Society\\n"
string_val: "Trip\\n"


outputs
key: "prelogits"
value
dtype: DT_FLOAT
tensor_shape
dim
size: 1

dim
size: 1

dim
size: 1

dim
size: 2048


float_val: 0.29251652956
float_val: 0.283589184284
float_val: 0.0797990858555
float_val: 0.427444338799
float_val: 0.147782608867
...
float_val: 0.784989655018
float_val: 0.0607083588839
float_val: 0.256205767393
float_val: 0.177307203412


outputs
key: "scores"
value
dtype: DT_FLOAT
tensor_shape
dim
size: 1

dim
size: 5


float_val: 0.995890200138
float_val: 0.00202985620126
float_val: 0.000434114364907
float_val: 0.000334182434017
float_val: 0.00018374362844

以上是关于tensorflow serving slim配置流程的主要内容,如果未能解决你的问题,请参考以下文章

Tensorflow(辅助工具)--tensorflow slim(TF-Slim) 使用笔记

如何配置 Tensorflow Serving 以服务来自 HDFS 的模型?

pyhton环境地址中配置slim及research

『TensorFlow』slim高级模块

TensorFlow和最近发布的slim

Tensorflow : tf.slim库解读