如何在 Tensorflow 中实现反转梯度？

Posted 2023-02-23

【中文标题】如何在 Tensorflow 中实现反转梯度？

【英文标题】：How to implement inverting gradient in Tensorflow?

【发布时间】：2019-05-18 17:29:15

【问题描述】：

我正在尝试在 Tensorflow 中实现 DDPG。动作空间与上限P_max 和下限P_min 连续。基于this paper，反转梯度是连续动作空间的好方法。但是，我在更新演员网络时遇到了困难。我将在下面介绍我的代码。

首先，我为 state、next_state 和 reward 构建占位符。其中S_DIM 是状态维度。

self.S = tf.placeholder(tf.float32, [None, S_DIM], name='state')
self.S_ = tf.placeholder(tf.float32, [None, S_DIM], name='next_state')
self.R = tf.placeholder(tf.float32, [None, 1], name='reward')

为actor和critic构建神经网络，其中A_DIM是动作空间：

def build_a(self, s, scope, trainable):
    with tf.variable_scope('actor'):
        with tf.variable_scope(scope):
            l1 = tf.layers.dense(s, 100, tf.nn.relu,trainable=trainable)
            a = tf.layers.dense(l1, A_DIM, trainable=trainable)                
            return a

def build_c(self, s, a, scope, trainable):
    with tf.variable_scope('critic'):
        with tf.variable_scope(scope):
            concat_layer = tf.concat([s, a], axis=1)
            l1 = tf.layers.dense(concat_layer, 100, tf.nn.relu, trainable=trainable)
            q = tf.layers.dense(l1, 1, trainable=trainable)
            return q

self.a = self.build_a(self.S, scope='evaluation', trainable=True)
self.a_ = self.build_a(self.S_, scope='target', trainable=False)
self.q = self.build_c(self.S, self.a, scope='evaluation', trainable=True)
self.q_ = self.build_c(self.S_, a_, scope='target', trainable=False)

访问神经网络中的参数以备后用：

self.ae_params = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope='actor/evaluation')
self.at_params = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope='actor/target')
self.ce_params = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope='critic/evaluation')
self.ct_params = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope='critic/target')

然后，通过最小化q_target 和q 之间的差异，使用时差贝尔曼方程更新评论家。 GAMMA是折扣系数，例如：0.99

q_target = self.R + GAMMA * q_
self.c_loss = tf.losses.mean_squared_error(q_target, self.q_)
self.ctrain = tf.train.AdamOptimizer(0.001).minimize(self.c_loss, var_list=self.ce_params)

最后，更新演员（我卡住的地方）：

dq_da = tf.gradients(q, self.a)[0]  # partial Q, partial a
upper_method = lambda: dq_da * (upper - self.a) / (upper - lower)
lower_method = lambda: dq_da * (self.a - lower) / (upper - lower)
# if gradient suggests increasing action, apply upper method
# else, lower method
adjust_dq_da = tf.cond(tf.greater(dq_da, 0), upper_method, lower_method)
grad = tf.gradients(self.a, self.ae_params, grad_ys=adjust_dq_da)
# apply gradient to the parameters in actor network
self.atrain = tf.train.AdamOptimizer(-0.0001).apply_gradients(zip(grad, self.ae_params))

我得到了错误：

ValueError: Shape must be rank 0 but is rank 2 for 'actor_gradient/cond/Switch' (op: 'Switch') with input shapes: [?,1], [?,1].

有什么办法可以改善吗？

【参考方案1】：

最后，我通过创建一个用于反转渐变的占位符来解决我的问题。

inverting_gradients_placeholder = tf.placeholder(tf.float32, shape=[None, 1], name='inverting_gradeints')

第一步，我计算Q值w.r.t动作a的梯度。

dq_da = tf.gradient(q, a)[0]

其次，我采取行动梯度a w.r.t 演员参数a_params。还要定义actor更新操作。

grad = tf.gradients(a, a_params, grad_ys=inverting_gradients_placeholder)
train_actor = tf.train.AdamOptimizer(learning_rate_actor).apply_gradients(zip(grad, a_params))

最后，实现反转渐变。

# get dq/da array, action array
dq_das, actions = sess.run([dq_da, a], feed_dict=state_placeholder: batch_state)
# inverting gradients, if dq_da >= 0, apply upper method, else lower method
inverting_gradients = []
for dq_da, action in zip(dq_das, actions):
    if dq_da >= 0.0:
        inverting_gradients.append(dq_da * (upper - action) / (upper - lower))
    else:
        inverting_gradients.append(dq_da * (action - lower) / (upper - lower))
inverting_gradients = np.array(inverting_gradients).reshape(-1, 1)
# update actor
sess.run(train_actor, feed_dict=state_placeholder: batch_state, inverting_gradients_placeholder: inverting_gradients)