pytorch loss

Posted 2022-01-30 东东就是我

1.损失函数简介

损失函数，又叫目标函数，用于计算真实值和预测值之间差异的函数。
pytorch loss基类是_Loss ,其中_Loss又继承Module类

其中每个loss，只需要实现forward就好

其中每次训练的时候都要
loss.backward()
但是我在loss函数里没找到backward()这个函数，就有点奇怪。后来发现原来loss是一个tensor，而tensor中是有backward的，

而且是调用的autograd里面的backward

1.1 L1Loss

$$\ell (x,y)=L=l_1,\dots ,l_N{}^{\top },l_n=\left|x_n-y_n\right|,$$
N是batch_size 如果没有设置reduction ，默认采用mean
$$\ell (x,y)=\begin{array}{ll}\mathrm{mean}(L),& \text{if reduction}=\text{’mean’;}\\ \mathrm{sum}(L),& \text{if reduction}=\text{’sum’.}\end{array}$$

import torch.nn as nn
import torch

def validate_loss(input,target):
    return torch.mean(torch.abs(input-target))
loss=nn.L1Loss()
input=torch.randn(3,5,requires_grad=True)
target=torch.randn(3,5)
output=loss(input,target)
print("default loss:", output)
output = validate_loss(input, target)
print("validate loss:", output)

1.2NLLLoss 多分类

https://zhuanlan.zhihu.com/p/338318581

$$\ell (x,y)=L=l_1,\dots ,l_N{}^{\top },l_n=-w_{y_n}{x}_{n,y_n},w_c=\text{weight}[c]\cdot 1c\ne \text{ignore\_index}$$
x是输入，y是label ，w是类别权重

$$\ell (x,y)=\begin{array}{ll}{\sum }_{n=1}^N\frac{1}{{\sum }_{n=1}^N{w}_{y_n}}{l}_n,& \text{if reduction}=\text{’mean’;}\\ {\sum }_{n=1}^N{l}_n,& \text{if reduction}=\text{’sum’.}\end{array}$$

def validate_loss(input,target):
    val = 0
    for li_x, li_y in zip(input, target):
        val+=li_x[li_y]
    return torch.abs(val / len(target))
loss = nn.NLLLoss()
m = nn.LogSoftmax(dim=1)
input=torch.randn(3,5,requires_grad=True)
target = torch.tensor([1, 0, 4])
output = loss(m(input), target)
print("default loss:", output)
output = validate_loss(m(input), target)
print("validate loss:", output)
>>>
>>>
# 2D loss example (used, for example, with image inputs)
def validate_loss(input,target):
    val = 0
    for li_x, li_y in zip(input, target):
        dim0, dim1 = li_y.shape
        li_x=li_x.tolist()
        li_y=li_y.tolist()
        # 遍历张量
        for i in range(dim0):
            for j in range(dim1):
                element = li_y[i][j]
                res=li_x[element][i][j]
                val+=res
    return val / 320

N, C = 5, 4
loss = nn.NLLLoss()
# input is of size N x C x height x width
data = torch.randn(N, 16, 10, 10)
conv = nn.Conv2d(16, C, (3, 3))
m = nn.LogSoftmax(dim=1)
# each element in target has to have 0 <= value < C
target = torch.empty(N, 8, 8, dtype=torch.long).random_(0, C)
input=m(conv(data))
output = loss(input, target)
print("default loss:", output)
output = validate_loss(input, target)
print("validate loss:", output)

1.3PoissonNLLoss

真实标签服从泊松分布的负对数似然损失，神经网络的输出作为泊松分布的参数λ 。
$$\text{target}\sim \mathrm{P}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{o}\mathrm{n}(\text{input})$$
$$\text{loss}(\text{input},\text{target})=\text{input}-\text{target}\ast \log (\text{input})+\log (\text{target!})$$

loss = nn.PoissonNLLLoss()
log_input = torch.randn(5, 2, requires_grad=True)
target = torch.randn(5, 2)
output = loss(log_input, target)
output.backward()

1.4KLDivLoss

$$l(x,y)=L=l_1,\dots ,l_N,l_n=y_n\cdot \left(\log y_n-x_n\right)$$

ℓ ( x , y ) = mean ⁡ ( L ) , if reduction = ’mean’; sum

以上是关于pytorch loss的主要内容，如果未能解决你的问题，请参考以下文章

[pytorch]pytorch loss function 总结

pytorch - loss.backward() 和 optimizer.step() 之间的连接

PyTorch：_thnn_nll_loss_forward 未针对类型 torch.LongTensor 实现

focal loss 之 pytorch 实现

pytorch模型训练之 loss function选择

为啥pytorch loss越来越高？