Tutorial on Losses in Convolutional Neural Networks(edit 2)

Posted 2021-11-30 chinafield-blog

Tutorial on Losses in Convolutional Neural Networks

Abstract

 

1. Introduction

 

2.Related Work

 

3. Cross-Entropy Loss

The Cross-Entropy Loss is actually the only loss we are discussing here. The other losses names written in the title are other variations of it. The Cross-Entropy Loss (CE Loss) is defined as:

 $$CE =  - \\sum\\limits_{i = 1}^C {{y_i}\\log {z_i}} $$

where $y_i$ and $z_i$ is the groundtruth and the CNN score for each class i in C. As usually an activation function (sigmoid / softmax) is applied to the scores before the CE Loss computation, we denote ${a_i} = f({z_i})$ as the activations.

In a binary classification problem, where $C‘ = 2$ , the Cross Entropy Loss will be written as:

$$CE =  - \\sum\\limits_{i = 1}^{{C^‘} = 2} {{y_i}\\log {z_i}}  =  - {y_1}\\log {z_1} - (1 - {y_1})\\log (1 - {z_1})$$

Where it assumes that there are two classes: ${c_1}$  and ${c_2}$.  ${y_1}$ and ${z_1}$ are the groundtruth and score for ${c_1}$, and ${y_2}=1-{y_1}$ and ${z_2}=1-{z_1}$ are the groundtruth and score for ${c_2}$. That is the case when we split a Multi-Label classification problem in C binary classification problems.

3.1 Categorical Cross-Entropy Loss / Softmax Loss

Categorial Cross-Entropy Loss is a Softmax activation plus a Cross-Entropy Loss. if we use this loss, we will train a CNN to output a probability over C classes for each image. It is used in muti-class classification, where each image only has one positive class and its groundtruth is a one-hot vector.

In the specific (and usual) case of Multi-Class classification the labels are one-hot, so only the positive class ${c_p}$ keeps its term in the loss. There is only one element of the groundtruth vector y which is not zero ${y_p} = 1$. So discarding the elements of the summation which are zero due to groundtruth labels, we can write:

$$CE =  - \\log {{{e^{{z_p}}}} \\over {\\sum\\nolimits_{j = 1}^C {{e^{{z_j}}}} }}$$

Where ${z_p}$ is the CNN score for the positve class.

Defined the loss, now we’ll have to compute its gradient respect to the output neurons of the CNN in order to backpropagate it through the net and optimize the defined loss function tuning the net parameters. So we need to compute the gradient of CE Loss respect each CNN class score in z$\mathrm{.\; The\; loss\; terms\; coming\; from\; the\; negative\; classes\; are\; zero.\; However,\; the\; loss\; gradient\; respect\; those\; negative\; classes\; is\; not\; cancelled,\; since\; the\; Softmax\; of\; the\; positive\; class\; also\; depends\; on\; the\; negative\; classes\; scores.}$

$\mathrm{After\; some\; calculus,\; the\; derivative\; respect\; to\; the\; positive\; class\; is:}$

$\mathrm{\$\$\{\backslash \backslash partial \; \backslash \backslash over\; \{\backslash \backslash partial\; \{z\_p\}\}\}(\; -\; \backslash \backslash log\; \{\{\{e^\{\{z\_p\}\}\}\}\; \backslash \backslash over\; \{\backslash \backslash sum\backslash \backslash nolimits\_\{j\; =\; 1\}^C\; \{\{e^\{\{z\_j\}\}\}\}\; \}\})\; =\; \{\{\{e^\{\{z\_p\}\}\}\}\; \backslash \backslash over\; \{\backslash \backslash sum\backslash \backslash nolimits\_\{j\; =\; 1\}^C\; \{\{e^\{\{z\_j\}\}\}\}\; \}\}\; -\; 1\$\$}$

$\mathrm{And\; the\; derivative\; respect\; to\; the\; other\; (negative)\; classes\; is:}$

$\mathrm{\$\$\{\backslash \backslash partial \; \backslash \backslash over\; \{\backslash \backslash partial\; \{z\_n\}\}\}(\; -\; \backslash \backslash log\; \{\{\{e^\{\{z\_p\}\}\}\}\; \backslash \backslash over\; \{\backslash \backslash sum\backslash \backslash nolimits\_\{j\; =\; 1\}^C\; \{\{e^\{\{z\_j\}\}\}\}\; \}\})\; =\; \{\{\{e^\{\{z\_p\}\}\}\}\; \backslash \backslash over\; \{\backslash \backslash sum\backslash \backslash nolimits\_\{j\; =\; 1\}^C\; \{\{e^\{\{z\_j\}\}\}\}\; \}\}\$\$}$

$\mathrm{Where\; \$\{y\_n\}\$\; is\; the\; score\; of\; any\; negative\; class\; in\; C\; different\; from\; \$\{c\_p\}\$.}$

$\mathrm{When\; Softmax\; loss\; is\; used\; is\; a\; multi-label\; scenario,\; the\; gradients\; get\; a\; bit\; more\; complex,\; since\; the\; loss\; contains\; an\; element\; for\; each\; positive\; class.\; Consider M are\; the\; positive\; classes\; of\; a\; sample.\; The\; CE\; Loss\; with\; Softmax\; activations\; would\; be:}$

$${1 \\over M}\\sum\\limits_{i = 1}^M {( - \\log {{{e^{{z_{{p_i}}}}}} \\over {\\sum\\nolimits_{j = 1}^C {{e^{{z_j}}}} }}} )$$

Where each ${z_{{p_i}}}$${\mathrm{in M is\; the\; CNN\; score\; for\; each\; positive\; class. As\; in\; Facebook\; paper,\; I\; introduce\; a\; scaling\; factor 1/M to\; make\; the\; loss\; invariant\; to\; the\; number\; of\; positive\; classes,\; which\; may\; be\; different\; per\; sample.}}^{}$

${\mathrm{The\; gradient\; has\; different\; expressions\; for\; positive\; and\; negative\; classes.\; For\; positive\; classes:}}^{}$

${\mathrm{\$\$\{\backslash \backslash partial \; \backslash \backslash over\; \{\backslash \backslash partial\; \{z\_\{\{p\_k\}\}\}\}\}(\{1\; \backslash \backslash over\; M\}\backslash \backslash sum\backslash \backslash limits\_\{i\; =\; 1\}^M\; \{(\; -\; \backslash \backslash log\; \{\{\{e^\{\{z\_\{\{p\_i\}\}\}\}\}\}\; \backslash \backslash over\; \{\backslash \backslash sum\backslash \backslash nolimits\_\{j\; =\; 1\}^C\; \{\{e^\{\{z\_j\}\}\}\}\; \}\}\}\; ))\; =\; \{1\; \backslash \backslash over\; M\}((\{\{\{e^\{\{z\_\{\{p\_k\}\}\}\}\}\}\; \backslash \backslash over\; \{\backslash \backslash sum\backslash \backslash nolimits\_\{j\; =\; 1\}^C\; \{\{e^\{\{z\_j\}\}\}\}\; \}\}\; -\; 1)\; +\; (M\; -\; 1)\{\{\{e^\{\{z\_\{\{p\_k\}\}\}\}\}\}\; \backslash \backslash over\; \{\backslash \backslash sum\backslash \backslash nolimits\_\{j\; =\; 1\}^C\; \{\{e^\{\{z\_j\}\}\}\}\; \}\})\$\$}}^{}$

${\mathrm{Where \$\{\{z\_\{\{p\_k\}\}\}\}\$ is\; the\; score\; of\; any\; positive\; class.}}^{}$

${\mathrm{For\; negative\; classes:}}^{}$

${\mathrm{\$\$\{\backslash \backslash partial \; \backslash \backslash over\; \{\backslash \backslash partial\; \{z\_n\}\}\}(\{1\; \backslash \backslash over\; M\}\backslash \backslash sum\backslash \backslash limits\_\{i\; =\; 1\}^M\; \{(\; -\; \backslash \backslash log\; \{\{\{e^\{\{z\_\{\{p\_i\}\}\}\}\}\}\; \backslash \backslash over\; \{\backslash \backslash sum\backslash \backslash nolimits\_\{j\; =\; 1\}^C\; \{\{e^\{\{z\_j\}\}\}\}\; \}\}\}\; ))\; =\; \{\{\{e^\{\{z\_n\}\}\}\}\; \backslash \backslash over\; \{\backslash \backslash sum\backslash \backslash nolimits\_\{j\; =\; 1\}^C\; \{\{e^\{\{z\_j\}\}\}\}\; \}\}\$\$}}^{}$

${\mathrm{To\; be\; noticed\; all\; the\; formulas\; above\; are\; only\; for\; one\; example.\; To\; compute\; the\; loss\; for\; a\; batch\; size\; examples,\; we\; should\; add\; the\; individual\; losses\; up\; and\; divide\; it\; by\; the\; batch\; size\; number.}}^{}$

3.2 Binary Cross-Entropy Loss

Also called Sigmoid Cross-Entropy loss. It is a Sigmoid activation plus a Cross-Entropy loss. Unlike Softmax loss it is independent for each vector component (class), meaning that the loss computed for every CNN output vector component is not affected by other component values. That’s why it is used for multi-label classification, where the insight of an element belonging to a certain class should not influence the decision for another class. It’s called Binary Cross-Entropy Loss because it sets up a binary classification problem between ${\mathrm{C\prime =2 classes\; for\; every\; class\; in C,\; as\; explained\; above.\; So\; when\; using\; this\; Loss,\; the\; formulation\; of Cross\; Entroypy\; Loss for\; binary\; problems\; is\; often\; used:}}^{}$

${\mathrm{\$\$CE\; = \; -\; \backslash \backslash sum\backslash \backslash limits\_\{i\; =\; 1\}^\{\{C^‘\}\; =\; 2\}\; \{\{y\_i\}\backslash \backslash log\; \{z\_i\}\} \; = \; -\; \{y\_1\}\backslash \backslash log\; \{z\_1\}\; -\; (1\; -\; \{y\_1\})\backslash \backslash log\; (1\; -\; \{z\_1\})\$\$}}^{}$