TensorFlow：组合两个图后从两个检查点恢复变量

Posted 2023-02-19

【中文标题】TensorFlow：组合两个图后从两个检查点恢复变量

【英文标题】：TensorFlow: Restoring variables from two checkpoints after combining two graphs

【发布时间】：2018-09-26 14:26:41

【问题描述】：

我在使用 TensorFlow 时遇到问题，希望您能提供帮助：D

我的目标：假设我有两个图，图 A 和图 B，每个图由三个卷积层组成。我想要的是将图 A 的第二层链接到图 B 的第三层。

我已经使用此处提供的信息成功地组合了两个图表：Tensorflow: How to replace a node in a calculation graph?

这是我得到的：

当我尝试加载可变权重时，问题就来了。我遵循了这个解决方案，但没有奏效：TensorFlow: Restoring variables from from multiple checkpoints

加载图A的权重没有问题，但是加载图B的权重时出现这个错误：

INFO:tensorflow:Restoring parameters from ./models/merge_graph_A.ckpt
INFO:tensorflow:Restoring parameters from ./models/merge_graph_B.ckpt
---------------------------------------------------------------------------
NotFoundError                             Traceback (most recent call last)
<ipython-input-27-1e08224fde39> in <module>()
      1 with tf.Session(graph=graphs_merged) as sess:
      2     saver_A.restore(sess, './models/merge_graph_A.ckpt')
----> 3     saver_B.restore(sess, './models/merge_graph_B.ckpt')
      4 
      5     outcome = sess.run(output, feed_dict=

/home/solano/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.pyc in restore(self, sess, save_path)
   1753     if context.in_graph_mode():
   1754       sess.run(self.saver_def.restore_op_name,
-> 1755                self.saver_def.filename_tensor_name: save_path)
   1756     else:
   1757       self._build_eager(save_path, build_save=False, build_restore=True)

/home/solano/.local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in run(self, fetches, feed_dict, options, run_metadata)
    903     try:
    904       result = self._run(None, fetches, feed_dict, options_ptr,
--> 905                          run_metadata_ptr)
    906       if run_metadata:
    907         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/home/solano/.local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _run(self, handle, fetches, feed_dict, options, run_metadata)
   1135     if final_fetches or final_targets or (handle and feed_dict_tensor):
   1136       results = self._do_run(handle, final_targets, final_fetches,
-> 1137                              feed_dict_tensor, options, run_metadata)
   1138     else:
   1139       results = []

/home/solano/.local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1353     if handle is None:
   1354       return self._do_call(_run_fn, self._session, feeds, fetches, targets,
-> 1355                            options, run_metadata)
   1356     else:
   1357       return self._do_call(_prun_fn, self._session, handle, feeds, fetches)

/home/solano/.local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_call(self, fn, *args)
   1372         except KeyError:
   1373           pass
-> 1374       raise type(e)(node_def, op, message)
   1375 
   1376   def _extend_graph(self):

NotFoundError: Key Variables_A/W_c1 not found in checkpoint
     [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

Caused by op u'save/RestoreV2', defined at:
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/solano/.local/lib/python2.7/site-packages/ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()
  File "/home/solano/.local/lib/python2.7/site-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/home/solano/.local/lib/python2.7/site-packages/ipykernel/kernelapp.py", line 486, in start
    self.io_loop.start()
  File "/home/solano/.local/lib/python2.7/site-packages/zmq/eventloop/ioloop.py", line 177, in start
    super(ZMQIOLoop, self).start()
  File "/home/solano/.local/lib/python2.7/site-packages/tornado/ioloop.py", line 888, in start
    handler_func(fd_obj, events)
  File "/home/solano/.local/lib/python2.7/site-packages/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/solano/.local/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 440, in _handle_events
    self._handle_recv()
  File "/home/solano/.local/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 472, in _handle_recv
    self._run_callback(callback, msg)
  File "/home/solano/.local/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback
    callback(*args, **kwargs)
  File "/home/solano/.local/lib/python2.7/site-packages/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/solano/.local/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "/home/solano/.local/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 233, in dispatch_shell
    handler(stream, idents, msg)
  File "/home/solano/.local/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 399, in execute_request
    user_expressions, allow_stdin)
  File "/home/solano/.local/lib/python2.7/site-packages/ipykernel/ipkernel.py", line 208, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "/home/solano/.local/lib/python2.7/site-packages/ipykernel/zmqshell.py", line 537, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "/home/solano/.local/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2718, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "/home/solano/.local/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2822, in run_ast_nodes
    if self.run_code(code, result):
  File "/home/solano/.local/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2882, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-22-4397b861df66>", line 5, in <module>
    return_elements=["conv_A_2/output:0"], name="")
  File "/home/solano/.local/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 432, in new_func
    return func(*args, **kwargs)
  File "/home/solano/.local/lib/python2.7/site-packages/tensorflow/python/framework/importer.py", line 553, in import_graph_def
    op_def=op_def)
  File "/home/solano/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3271, in create_op
    op_def=op_def)
  File "/home/solano/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1650, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

NotFoundError (see above for traceback): Key Variables_A/W_c1 not found in checkpoint
     [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

真正有趣的是，如果我改变了合并的顺序，从 B 中取出前两层，从 A 中取出第三层，加载图的权重时出现错误A，正好相反。

---

代码

正在加载图表 A。

tf.reset_default_graph()
graph_A = tf.Graph()

with graph_A.as_default():
    saver_A = tf.train.import_meta_graph('./models/merge_graph_A.ckpt.meta')

graph_A_def = graph_A.as_graph_def()

正在加载图表 B。

tf.reset_default_graph()
graph_B = tf.Graph()

with graph_B.as_default():
    saver_B = tf.train.import_meta_graph('./models/merge_graph_B.ckpt.meta')

graph_B_def = graph_B.as_graph_def()

合并两个图：

tf.reset_default_graph() # not necessary
with tf.Graph().as_default() as graphs_merged:
    input_image = tf.placeholder(tf.float32, (None, None, None, 3), name='input')

    bottleneck, = tf.import_graph_def(graph_A_def, input_map="input_A:0": input_image,
                           return_elements=["conv_A_2/output:0"], name="")

    # Notice how the input is the bottleneck.
    output, = tf.import_graph_def(graph_B_def, input_map="conv_B_2/output:0": bottleneck, # HERE'S THE PROBLEM
                           return_elements=["conv_B_3/output:0"], name="")

装载重量：

with tf.Session(graph=graphs_merged) as sess:
    saver_A.restore(sess, './models/merge_graph_A.ckpt')
    saver_B.restore(sess, './models/merge_graph_B.ckpt') # HERE'S THE PROBLEM

    outcome = sess.run(output, feed_dict=
        input_image: image_test
    )

谢谢大家！！！

---

编辑：这是保护程序的问题。所有检查点都以相同的前缀“保存”保存。这导致，在合并时，TensorFlow 将第二个图的前缀更改为“save_1”，使得 saver 无法找到它必须恢复的变量，而是尝试使用以“save”为前缀的操作来恢复变量。

所以问题变了：有没有办法在此处导入元图后更改此操作的名称：saver_A = tf.train.import_meta_graph('./models/merge_graph_A.ckpt.meta')？

save/Const
save/SaveV2/tensor_names
save/SaveV2/shape_and_slices
save/SaveV2
save/control_dependency
save/RestoreV2/tensor_names
save/RestoreV2/shape_and_slices
save/RestoreV2
save/Assign
save/Assign_1
save/Assign_2
save/Assign_3
save/Assign_4
save/Assign_5
save/restore_all

save_1/Const
save_1/SaveV2/tensor_names
save_1/SaveV2/shape_and_slices
save_1/SaveV2
save_1/control_dependency
save_1/RestoreV2/tensor_names
save_1/RestoreV2/shape_and_slices
save_1/RestoreV2
save_1/Assign
save_1/Assign_1
save_1/Assign_2
save_1/Assign_3
save_1/Assign_4
save_1/Assign_5
save_1/restore_all

【问题讨论】：

你最终解决了这个问题吗？

【参考方案1】：

在保存两个checkpoint的时候尝试定义不同的saver名称，就可以了。

示例：

保存：

    saver1 = tf.train.Saver(name="saver1")
    saver1.save(sess, model_path1)
    saver2 = tf.train.Saver(name="saver2")
    saver2.save(sess, model_path2)

恢复：

    saver1 = tf.train.import_meta_graph(self.meta_path1)
    saver2 = tf.train.import_meta_graph(self.meta_path2)
    saver1.restore(sess, tf.train.latest_checkpoint(self.model_path1))
    saver2.restore(sess, tf.train.latest_checkpoint(self.model_path2))

【讨论】：

提高您的回答质量。格式正确，切中要害。