余弦衰减学习率与linear warmup结合版代码

Posted 2023-04-07 氵文大师

以下代码摘自tensorflow官方tpu仓库

def cosine_learning_rate_with_linear_warmup(global_step,
                                            init_learning_rate,
                                            warmup_learning_rate,
                                            warmup_steps,
                                            total_steps):
    """Creates the cosine learning rate tensor with linear warmup."""
    global_step = tf.cast(global_step, dtype=tf.float32)
    linear_warmup = (warmup_learning_rate + global_step / warmup_steps *
                   (init_learning_rate - warmup_learning_rate))
    cosine_learning_rate = (
        init_learning_rate * (tf.cos(
            np.pi * (global_step - warmup_steps) / (total_steps - warmup_steps))
                              + 1.0) / 2.0)
    learning_rate = tf.where(global_step < warmup_steps,
                             linear_warmup, cosine_learning_rate)
    return learning_rate

五个参数的含义，直接看图吧，代码瞅一眼也简单

在 warmup阶段，学习率从 warmup_learning_rate 变为 init_learning_rate，该阶段中学习率是线性递增或递减的

在余弦衰减阶段，学习率是这样衰减的：

$$lr=\frac{cos\left(\frac{gl-w}{t-w}\pi \right)+1}{2}\ast init\_learning\_rate$$

$cos$中的变量：

• $gl$ 是 $global\_step$
• $w$ 是 $warmup\_steps$
• $t$ 是 $total\_step$

衰减曲线如下图蓝色框中的部分所示：

下降程度先逐渐加快，之后逐渐变慢，收敛到一个很小的值

学习率预热(transformers.get_linear_schedule_with_warmup)

学习率预热

• 在预热期间，学习率从0线性增加到优化器中的初始lr。

• 在预热阶段之后创建一个schedule，使其学习率从优化器中的初始lr线性降低到0

Parameters

• optimizer (Optimizer) – The optimizer for which to schedule the learning rate.

• num_warmup_steps (int) – The number of steps for the warmup phase.

• num_training_steps (int) – The total number of training steps.

• last_epoch (int, optional, defaults to -1) – The index of the last epoch when resuming training.

Returns

• torch.optim.lr_scheduler.LambdaLR with the appropriate schedule.

# training steps 的数量: [number of batches] x [number of epochs].
total_steps = len(train_dataloader) * epochs

# 设计 learning rate scheduler
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps = 50, 
                                            num_training_steps = total_steps)