PyTorch笔记 - Normalization Layer (Batch/Layer/Instance/Group/Weight)

Posted 2023-03-03 SpikeKing

欢迎关注我的CSDN：https://blog.csdn.net/caroline_wendy
本文地址：https://blog.csdn.net/caroline_wendy/article/details/128416962

Normalization in NN:

1. Batch Normalization:
   1. per channel across mini-batch
   2. torch.nn.BatchNorm1d / torch.nn.BatchNorm2d
2. Layer Normalization:
   1. per sample, per layer
   2. torch.nn.LayerNorm
3. Instance Normalization:
   1. per sample, per channel
   2. torch.nn.InstanceNorm1d / torch.nn.InstanceNorm2d
4. Group Normalization:
   1. per sample, per group
   2. torch.nn.GroupNorm
5. Weight Norm