tensorflow serving slim配置流程

Posted 2022-12-07 帅气的小王子

发现国内tensorflow serving方面的博文非常少，包括国外也不多，主要依靠官方文档，但官方文档有些碎片化并包含一些冗余，因此整合为一篇文章方便国人。

之前在我在生产环境下用tensorflow的方式是启动一个thrift服务，load进模型之后不断处理请求，但是在面对大批量数据时有些力不从心，对机器的压力很大，因此考虑使用tensorflow serving的方式部署服务。

服务调用的模型是slim下的inception v3 图像分类模型，对模型进行了迁移学习，得到了新的分类模型，运用该模型对单张图片或者批量图片做预测或者提取特征。

本文假设你已成功安装tensorflow，并利用slim的代码，迁移学习获得了自己的inception v3模型，本文主要阐述如何将该模型部署到tensorflow serving 上。

tensorflow 版本：r.1.3 ， python版本：2.7.8 ，无GPU

Step1：安装tensorflow Serving r1.3

参考官方文档：https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/setup.md

前期准备：

安装bazel

pip install tensorflow-serving-api

pip install grpcio

安装过程：

  1.git clone --recurse-submodules https://github.com/tensorflow/serving

     cd serving

  2. cd tensorflow

   ./configure

---

 

You have bazel 0.7.0- (@non-git) installed.
Please specify the location of python. [Default is /data0/home/wang16/local/python/bin/python]:

Found possible Python library paths:
/data0/home/wang16/local/python/lib/python2.7/site-packages/
/data0/install/caffe/python/
/data0/home/wang16/simba/trunk/src/content_analysis/
/data0/home/wang16/local/python/lib/python2.7/site-packages
Please input the desired Python library path to use. Default is [/data0/home/wang16/local/python/lib/python2.7/site-packages/]

Do you wish to build TensorFlow with jemalloc as malloc support? [Y/n]: 
jemalloc as malloc support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Google Cloud Platform support? [Y/n]: n
No Google Cloud Platform support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Hadoop File System support? [Y/n]: n
No Hadoop File System support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Amazon S3 File System support? [Y/n]: n
No Amazon S3 File System support will be enabled for TensorFlow.

Do you wish to build TensorFlow with XLA JIT support? [y/N]: 
No XLA JIT support will be enabled for TensorFlow.

Do you wish to build TensorFlow with GDR support? [y/N]: 
No GDR support will be enabled for TensorFlow.

Do you wish to build TensorFlow with VERBS support? [y/N]: 
No VERBS support will be enabled for TensorFlow.

Do you wish to build TensorFlow with OpenCL support? [y/N]: 
No OpenCL support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: 
No CUDA support will be enabled for TensorFlow.

Do you wish to build TensorFlow with MPI support? [y/N]: 
No MPI support will be enabled for TensorFlow.

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]:

Add "--config=mkl" to your bazel command to build with MKL support.
Please note that MKL on MacOS or windows is still not supported.
If you would like to use a local MKL instead of downloading, please set the environment variable "TF_MKL_ROOT" every time before build.
Configuration finished

---

 

 cd ..

  3. bazel build -c opt --copt=-msse4.1 --copt=-msse4.2 --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-O3 tensorflow_serving/...(耗时较长)

这里如果是centos可能会报错：fatal error: stropts.h: No such file or directory

在/usr/include建一个空的文件。命名stropts.h就可以了。

  4. bazel test -c opt tensorflow_serving/...

 

Step2：修改代码以支持新的模型

参考文档：https://gyang274.github.io/docker-tensorflow-serving-slim/0x02b00.slim.inception.v4.html，https://github.com/gyang274/docker-tensorflow-serving-slim

参考文档：https://www.tensorflow.org/serving/serving_inception

这部分是最难的，官方提供的示例代码很少，幸亏找到了一篇相关文档，照猫画虎，才算完成。

  1. 回到 serving 主目录，目录下有如下内容

 mkdir -p tf_checkpoints/slim/my_inception_v3

 拷贝迁移学习训练好的模型到该目录中，拷贝完目录内容如下：

  2.参照示例写模型导出代码：tensorflow_serving/example/my_inception_v3_saved_model.py

# Copyright 2016 Google Inc. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================== #!/usr/bin/env python2.7 """Export inception v3 model given existing training checkpoints. The model is exported as SavedModel with proper signatures that can be loaded by standard tensorflow_model_server. """ import os.path # This is a placeholder for a Google-internal import. import tensorflow as tf slim = tf.contrib.slim # from preprocessing import inception_preprocessing from nets import inception tf.app.flags.DEFINE_string('checkpoint_dir', '/tmp/slim_inception_v3_train',  """Directory where to read training checkpoints.""") tf.app.flags.DEFINE_string('output_dir', '/tmp/slim_inception_v3_output',  """Directory where to export inference model.""") tf.app.flags.DEFINE_integer('model_version', 1,  """Version number of the model.""") tf.app.flags.DEFINE_integer('image_size', 299,  """Needs to provide same value as in training.""") FLAGS = tf.app.flags.FLAGS NUM_CLASSES = 76 NUM_TOP_CLASSES = 5 WORKING_DIR = os.path.dirname(os.path.realpath(__file__))   def preprocess_image(image_buffer):  """Preprocess JPEG encoded bytes to 3D float Tensor."""  # Decode the string as an RGB JPEG.  # Note that the resulting image contains an unknown height and width  # that is set dynamically by decode_jpeg. In other words, the height  # and width of image is unknown at compile-time.  image = tf.image.decode_jpeg(image_buffer, channels=3)  # After this point, all image pixels reside in [0,1)  # until the very end, when they're rescaled to (-1, 1). The various  # adjust_* ops all require this range for dtype float.  image = tf.image.convert_image_dtype(image, dtype=tf.float32)  # Crop the central region of the image with an area containing 87.5% of  # the original image.  image = tf.image.central_crop(image, central_fraction=0.875)  # Resize the image to the original height and width.  image = tf.expand_dims(image, 0)  image = tf.image.resize_bilinear(  image, [FLAGS.image_size, FLAGS.image_size], align_corners=False)  image = tf.squeeze(image, [0])  # Finally, rescale to [-1,1] instead of [0, 1)  image = tf.subtract(image, 0.5)  image = tf.multiply(image, 2.0)  return image   def export():  with tf.Graph().as_default():  # build inference model  # labels  f = open('/data0/home/wang16/package/serving/tf_checkpoints/slim/my_inception_v3/labels.txt')  lines = f.readlines()  names = 0:'background'  for line in lines:  i,label = line.split(':')  names[int(i)]=label    names_tensor = tf.constant(  names.values()  )  names_lookup_table = tf.contrib.lookup.index_to_string_table_from_tensor(  names_tensor  )  # input transformation  serialized_tf_example = tf.placeholder(  tf.string, name='tf_example'  )  feature_configs =  'image/encoded': tf.FixedLenFeature(  shape=[], dtype=tf.string  ),    tf_example = tf.parse_example(  serialized_tf_example, feature_configs  )  jpegs = tf_example['image/encoded']  images = tf.map_fn(  preprocess_image, jpegs, dtype=tf.float32  )  # run inference  with slim.arg_scope(  inception.inception_v3_arg_scope()  ):  # inception v3 models  logits, end_points = inception.inception_v3(  images, num_classes=NUM_CLASSES, is_training=False  )  # logits = tf.Print(logits, [logits])  probs = tf.nn.softmax(logits)  # transform output to topk result  topk_probs, topk_indices = tf.nn.top_k(  probs, NUM_TOP_CLASSES  )  topk_names = names_lookup_table.lookup(  tf.to_int64(topk_indices)  )  init_fn = slim.assign_from_checkpoint_fn(  os.path.join(FLAGS.checkpoint_dir, 'model.ckpt-100000'),  slim.get_model_variables('InceptionV3')  )  # sess config  config = tf.ConfigProto(  # device_count =  # 'GPU': 0  # ,  gpu_options=  'allow_growth': 1,  # 'per_process_gpu_memory_fraction': 0.01  ,  allow_soft_placement=True,  log_device_placement=False,  )  with tf.Session(config=config) as sess:  init_fn(sess)  # init on 2017.08.05  # prelogits = sess.graph.get_tensor_by_name(  # 'InceptionV3/Logits/PreLogitsFlatten/Reshape:0'  # )  # update on 2017.10.22  # note: looks like the graphdef is updated for slim-inception-v3  print('Graph Node Tensor Name:')  for node_tensor in tf.get_default_graph().as_graph_def().node:  if str(node_tensor.name).startswith('InceptionV3/Logits'):  print str(node_tensor.name)  prelogits = sess.graph.get_tensor_by_name(  'InceptionV3/Logits/Dropout_1b/Identity:0'  )  # an optional alternative  # prelogits = end_points['PreLogitsFlatten']  # export inference model.  output_path = os.path.join(  tf.compat.as_bytes(FLAGS.output_dir),  tf.compat.as_bytes(str(FLAGS.model_version))  )  print 'Exporting trained model to', output_path  builder = tf.saved_model.builder.SavedModelBuilder(output_path)  # build the signature_def_map.  predict_inputs_tensor_info = tf.saved_model.utils.build_tensor_info(  jpegs  )  classes_output_tensor_info = tf.saved_model.utils.build_tensor_info(  topk_names  )  scores_output_tensor_info = tf.saved_model.utils.build_tensor_info(  topk_probs  )  prelogits_output_tensor_info = tf.saved_model.utils.build_tensor_info(  prelogits  )  prediction_signature = (  tf.saved_model.signature_def_utils.build_signature_def(  inputs=  'images': predict_inputs_tensor_info  ,  outputs=  'classes': classes_output_tensor_info,  'scores': scores_output_tensor_info,  'prelogits': prelogits_output_tensor_info  ,  method_name=tf.saved_model.signature_constants.PREDICT_METHOD_NAME  )  )  legacy_init_op = tf.group(  tf.tables_initializer(), name='legacy_init_op'  )  builder.add_meta_graph_and_variables(  sess, [tf.saved_model.tag_constants.SERVING],  signature_def_map=  'predict_images':  prediction_signature,  ,  legacy_init_op=legacy_init_op  )  builder.save()  print 'Successfully exported model to %s' % FLAGS.output_dir   def main(unused_argv=None):  export()   if __name__ == '__main__':  tf.app.run()

     参照示例写模型服务代码：tensorflow_serving/example/my_inception_v3_client.py

# Copyright 2016 Google Inc. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================== #!/usr/bin/env python2.7 """Send JPEG image to tensorflow_model_server loaded with inception model. """ from __future__ import print_function import urllib2 # This is a placeholder for a Google-internal import. from grpc.beta import implementations import tensorflow as tf from tensorflow_serving.apis import predict_pb2 from tensorflow_serving.apis import prediction_service_pb2   tf.app.flags.DEFINE_string(  'server', 'localhost:9000', 'PredictionService host:port' ) tf.app.flags.DEFINE_string(  'image_url', '', 'URL to image in JPEG format' ) FLAGS = tf.app.flags.FLAGS   def main(_):  host, port = FLAGS.server.split(':')  channel = implementations.insecure_channel(host, int(port))  stub = prediction_service_pb2.beta_create_PredictionService_stub(channel)  # Send request  # See prediction_service.proto for gRPC request/response details.  image_bytes = urllib2.urlopen(FLAGS.image_url).read()  request = predict_pb2.PredictRequest()  request.model_spec.name = 'my_inception_v3'  request.model_spec.signature_name = 'predict_images'  request.inputs['images'].CopyFrom(  tf.contrib.util.make_tensor_proto(  image_bytes, shape=[1]  )  )  result = stub.Predict(request, 60.0) # 60 secs timeout  print(result)   if __name__ == '__main__':  tf.app.run()

     参照参考文献修改：tensorflow_serving/example/BUILD

...  py_binary(  name = "my_inception_v3_saved_model",  srcs = [  "my_inception_v3_saved_model.py",  ],  srcs_version = "PY2AND3",  deps = [  "@slim_model//:dataset_factory",  "@slim_model//:preprocessing_factory",  "@slim_model//:nets",  "@org_tensorflow//tensorflow:tensorflow_py",  ], ) py_binary(  name = "my_inception_v3_client",  srcs = [  "my_inception_v3_client.py",  ],  srcs_version = "PY2AND3",  deps = [  "//tensorflow_serving/apis:predict_proto_py_pb2",  "//tensorflow_serving/apis:prediction_service_proto_py_pb2",  "@org_tensorflow//tensorflow:tensorflow_py",  ], ) ...

    参照参考文献修改：/tensorflow_serving/workspace.bzl

...  native.new_local_repository(  name = "slim_model",  path = "tf_models/research/slim",  build_file = "tf_models/research/slim/BUILD",  ) ...

      修改代码： /tf_models/research/slim/BUILD，注释掉所有  "//tensorflow",

      编译代码:

      bazel build -c opt --copt=-msse4.1 --copt=-msse4.2 --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-O3 tensorflow_serving/...

      提取模型：      

    bazel-bin/tensorflow_serving/example/my_inception_v3_saved_model \\
      --checkpoint_dir=tf_checkpoints/slim/my_inception_v3 \\
      --output_dir=tf_servables/slim/my_inception_v3 \\
      --model_version=1 \\
      --image_size=299

      提取后的模型目录：

 

 

Step3：部署服务

   1、部署服务

    bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server --model_name=my_inception_v3 --model_base_path=$PWD/tf_servables/slim/my_inception_v3 --port=9000

   2、测试服务

    python my_inception_v3_client.py

  3、返回结果

outputs
key: "classes"
value
dtype: DT_STRING
tensor_shape
dim
size: 1

dim
size: 5


string_val: "BeautifulSignt\\n"
string_val: "Building\\n"
string_val: "Cartoon\\n"
string_val: "Society\\n"
string_val: "Trip\\n"


outputs
key: "prelogits"
value
dtype: DT_FLOAT
tensor_shape
dim
size: 1

dim
size: 1

dim
size: 1

dim
size: 2048


float_val: 0.29251652956
float_val: 0.283589184284
float_val: 0.0797990858555
float_val: 0.427444338799
float_val: 0.147782608867
...
float_val: 0.784989655018
float_val: 0.0607083588839
float_val: 0.256205767393
float_val: 0.177307203412


outputs
key: "scores"
value
dtype: DT_FLOAT
tensor_shape
dim
size: 1

dim
size: 5


float_val: 0.995890200138
float_val: 0.00202985620126
float_val: 0.000434114364907
float_val: 0.000334182434017
float_val: 0.00018374362844