Introduction to batchnorm

This learning module has many interactive demos. It is easier to work with them on a larger screen. Bookmark and revisit if you are currently on a small screen device.

\(\DeclareMathOperator*{\argmax}{arg\,max} \DeclareMathOperator*{\argmin}{arg\,min} \DeclareMathOperator*{\asterisk}{\ast} \newcommand{\sup}{\text{sup}} \newcommand{\inf}{\text{inf}} \newcommand{\min}{\text{min}\;} \newcommand{\max}{\text{max}\;} \newcommand{\maxunder}[1]{\underset{#1}{\max}} \newcommand{\minunder}[1]{\underset{#1}{\min}} \newcommand{\real}{\mathbb{R}} \newcommand{\natural}{\mathbb{N}} \newcommand{\integer}{\mathbb{Z}} \newcommand{\rational}{\mathbb{Q}} \newcommand{\irrational}{\mathbb{I}} \newcommand{\complex}{\mathbb{C}} \newcommand{\cardinality}[1]{|#1|} \newcommand{\vec}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\star}[1]{#1^*} \newcommand{\inv}[1]{#1^{-1}} \newcommand{\indicator}[1]{\mathcal{I}(#1)} \renewcommand{\BigO}[1]{\mathcal{O}(#1)} \renewcommand{\BigOsymbol}{\mathcal{O}} \renewcommand{\smallo}[1]{\mathcal{o}(#1)} \renewcommand{\smallosymbol}[1]{\mathcal{o}} \newcommand{\set}[1]{\mathbb{#1}} \newcommand{\complement}[1]{#1^c} \newcommand{\powerset}[1]{\mathcal{P}(#1)} \newcommand{\setdiff}{\setminus} \newcommand{\setsymmdiff}{\oplus} \newcommand{\dash}[1]{#1^{'}} \newcommand{\permutation}[2]{{}_{#1} \mathrm{ P }_{#2}} \newcommand{\combination}[2]{{}_{#1} \mathrm{ C }_{#2}} \newcommand{\prob}[1]{P(#1)} \newcommand{\pmf}[1]{P(#1)} \newcommand{\pdf}[1]{p(#1)} \newcommand{\cdf}[1]{F(#1)} \newcommand{\expect}[2]{E_{#1}\left[#2\right]} \newcommand{\entropy}[1]{\mathcal{H}\left[#1\right]} \newcommand{\expe}[1]{\mathrm{e}^{#1}} \newcommand{\textexp}[1]{\text{exp}\left(#1\right)} \def\independent{\perp\!\!\!\perp} \def\notindependent{\not\!\independent} \newcommand{\yhat}{\hat{y}} \newcommand{\vs}{\vec{s}} \newcommand{\vt}{\vec{t}} \newcommand{\vu}{\vec{u}} \newcommand{\vv}{\vec{v}} \newcommand{\vw}{\vec{w}} \newcommand{\vx}{\vec{x}} \newcommand{\vy}{\vec{y}} \newcommand{\vz}{\vec{z}} \newcommand{\va}{\vec{a}} \newcommand{\vb}{\vec{b}} \newcommand{\vc}{\vec{c}} \newcommand{\vd}{\vec{d}} \newcommand{\ve}{\vec{e}} \newcommand{\vg}{\vec{g}} \newcommand{\vh}{\vec{h}} \newcommand{\vi}{\vec{i}} \newcommand{\vk}{\vec{k}} \newcommand{\vo}{\vec{o}} \newcommand{\vp}{\vec{p}} \newcommand{\vq}{\vec{q}} \newcommand{\vr}{\vec{r}} \newcommand{\vs}{\vec{s}} \newcommand{\vmu}{\vec{\mu}} \newcommand{\vsigma}{\vec{\sigma}} \newcommand{\vphi}{\vec{\phi}} \newcommand{\vtau}{\vec{\tau}} \newcommand{\vtheta}{\vec{\theta}} \newcommand{\mA}{\mat{A}} \newcommand{\mB}{\mat{B}} \newcommand{\mC}{\mat{C}} \newcommand{\mD}{\mat{D}} \newcommand{\mE}{\mat{E}} \newcommand{\mH}{\mat{H}} \newcommand{\mK}{\mat{K}} \newcommand{\mP}{\mat{P}} \newcommand{\mQ}{\mat{Q}} \newcommand{\mR}{\mat{R}} \newcommand{\mS}{\mat{S}} \newcommand{\mU}{\mat{U}} \newcommand{\mV}{\mat{V}} \newcommand{\mW}{\mat{W}} \newcommand{\mX}{\mat{X}} \newcommand{\mY}{\mat{Y}} \newcommand{\mZ}{\mat{Z}} \newcommand{\mI}{\mat{I}} \newcommand{\mLambda}{\mat{\Lambda}} \newcommand{\mSigma}{\mat{\Sigma}} \newcommand{\mTheta}{\mat{\theta}} \newcommand{\setsymb}[1]{#1} \newcommand{\sA}{\setsymb{A}} \newcommand{\sB}{\setsymb{B}} \newcommand{\sC}{\setsymb{C}} \newcommand{\sO}{\setsymb{O}} \newcommand{\sP}{\setsymb{P}} \newcommand{\sQ}{\setsymb{Q}} \newcommand{\sH}{\setsymb{H}} \newcommand{\sX}{\setsymb{X}} \newcommand{\sY}{\setsymb{Y}} \newcommand{\norm}[2]{||{#1}||_{#2}} \newcommand{\infnorm}[1]{\norm{#1}{\infty}} \newcommand{\fillinblank}{\text{ }\underline{\text{ ? }}\text{ }} \newcommand{\lbrace}{\left\{} \newcommand{\rbrace}{\right\}} \newcommand{\set}[1]{\lbrace #1 \rbrace} \newcommand{\seq}[1]{\left( #1 \right)} \newcommand{\ndim}{N} \newcommand{\ndimsmall}{n} \newcommand{\dataset}{\mathbb{D}} \newcommand{\ndata}{D} \newcommand{\ndatasmall}{d} \newcommand{\labeledset}{\mathbb{L}} \newcommand{\nlabeled}{L} \newcommand{\nlabeledsmall}{l} \newcommand{\unlabeledset}{\mathbb{U}} \newcommand{\nunlabeled}{U} \newcommand{\nunlabeledsmall}{u} \newcommand{\nclass}{M} \newcommand{\nclasssmall}{m} \newcommand{\loss}{\mathcal{L}} \newcommand{\sign}{\text{sign}} \newcommand{\Gauss}{\mathcal{N}} \newcommand{\hadamard}{\circ} \newcommand{\doh}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\dox}[1]{\doh{#1}{x}} \newcommand{\doy}[1]{\doh{#1}{y}} \newcommand{\doxx}[1]{\doh{#1}{x^2}} \newcommand{\doyy}[1]{\doh{#1}{y^2}} \newcommand{\doxy}[1]{\frac{\partial #1}{\partial x \partial y}} \newcommand{\doyx}[1]{\frac{\partial #1}{\partial y \partial x}} \newcommand{\qed}{\tag*{$\blacksquare$}}\)

        Batch normalization (batchnorm)
        Deep Learning
      

Introduction

Batch normalization CITE[szegedy-2015], succinctly known as batchnorm, is a strategy for adaptive reparametrization to facilitate training of very deep neural networks. It was originally proposed as a mechanism to limit the impact of parameter updates in one layer of the network on other layers that came before or after the said layer. It works by normalizing the activations of a training minibatch after a layer to zero mean and unit variance.

Although not the original intention, it also has the side-effect of regularizing the training of the neural network by restricting the minibatch activations and consequently the gradient updates. It is a particularly effective strategy that is now a default recommendation in most deep neural networks, especially those employing convolutional layers.

Prerequisites

To understand BatchNorm, we recommend familiarity with the concepts in

Probability: A sound understanding of conditional and marginal probabilities and Bayes Theorem is desirable.
Introduction to machine learning: An introduction to basic concepts in machine learning such as classification, training instances, features, and feature types.
Backpropagation
Multilayer perceptrons

Follow the above links to first get acquainted with the corresponding concepts.

Motivation

Deep neural networks are composites of several functions, in some cases represented as layers of the network. These composite functions are trained by gradient-based optimization strategies, the gradients being computed by backpropagation. The training proceeds through iterations, typically over mini-batches of training data, each time updating the parameters with functions of computed gradients.

But there is a key caveat: the gradients are partial derivatives. The gradients suggest the update to a parameter that will minimize the loss, assuming all the other parameters of the model remain the same as before. Neural networks are never trained to satisfy this assumption. After each mini-batch, we update all the parameters in the model. This means, the update may be futile for some parameters, depending on the magnitude of updates to other related parameters of the model.

One alternative to tackle this issue is to use higher order derivatives. Higher order derivatives incorporate relationships between parameters. For example, second-order derivatives will inform the optimization algorithm of pairwise interactions. But very deep neural networks are complicated composites. Higher-order derivatives, although better than first-order derivatives at handling inter-parameter relationships, will still not overcome the challenge of dealing with updates over deep composites.

We need something better to coordinate updates across many layers. And that solution is offered by batchnorm. But first, let's understand this challenge mathematically.

The mathematical motivation

Let's consider a simple illustrative example. Consider a deep neural network, with no nonlinear activations, and multiple layers, each with a single unit. From the input $ x $ to the output $ \yhat $, the composite transformation is straightforward — $ \yhat = x w_1 w_2 \ldots w_l $, where $ w_i $ denotes the parameter weight of the $ i$-th layer, with $ i=1 $ being the input layer. In terms of activations, for the $ i$-th layer, the activation is $ h_i = w_i h_{i-1} $.

With gradient-based optimization, each parameter will get updated by its gradient $ g_i $ as $ w_i \leftarrow w_i - \eta g_i $, where $ \eta $ is the learning rate of the gradient-based approach.
As a result, the new output with the updated model will be $ \dash{\yhat} = x (w_1-\eta g_1) (w_2-\eta g_2) \ldots (w_l - \eta g_l) $. Is this desirable? Let's find out.

Treating the neural network as function of its parameters and input, we can write the output as $ \yhat = f_x(\vw) $, where $ \vw = [w_1,\ldots,w_l] $. By changing the parameters, we change the output for the same input. The Taylor series approximation to the first-order tells us that.

$$ f_x(\dash{\vw}) = f_x(\vw) + \nabla_{\vw}\left(f(\vw)\right) (\dash{\vw} - \vw) $$

If $ \dash{\vw} $ denotes the updated values of the weights, then we can compute the difference between the new output $ \dash{\yhat} $ and the output before update, $ \yhat $. That is

\begin{aligned} \dash{\yhat} - \yhat &= f_x(\dash{\vw}) - f_x(\vw) \\\\ &= f_x(\vw) + \nabla_{\vw}\left(f(\vw)\right) (\dash{\vw} - \vw) - f_x(\vw) \\\\ &= \nabla_{\vw}\left(f(\vw)\right) (\dash{\vw} - \vw) \\\\ &= \nabla_{\vw}\left(f(\vw)\right) \eta \vg \\\\ &= \eta\vg^T \vg \\\\ \end{aligned}

where, we have used the representation $ \vg = [g_1,\ldots,g_l] $ to denote the vector of gradients, that is $ \nabla_{\vw}\left(f(\vw)\right) $. Thus, with a gradient-based update of $ \eta \vg $, we expect the new output to be different by $ \eta \vg^T \vg $ from the output before the gradient update $ \yhat $.

But we saw earlier that

\begin{aligned} \dash{\yhat} - \yhat &= x (w_1-\eta g_1) (w_2-\eta g_2) \ldots (w_l - \eta g_l) - x w_1 w_2 \ldots w_l \end{aligned}

Clearly, the changes we propagated will not have the desired results because there are many orders of interactions between gradients and weights, not just $ \eta \vg^T \vg $.

This challenge is worsened by increasing depths in the network, as more inter-parameter interactions come into play.

batchnorm: The mechanics

Batchnorm works by normalizing the output of each layer for each mini-batch; hence the name.

Suppose $ \mH $ denotes the activations of a layer on a mini-batch, with the activations of each input example from the mini-batch arranged as one row of $ \mH $.

We first compute a vector of means, $ \vmu $, of these activations per unit in that layer. This is the average over the $ m $ rows of the activation matrix $ \mH $.

$$ \vmu = \frac{1}{m} \sum_{i} \mH_{i,:} $$

Then, we compute the standard deviation, $ \vsigma $ of the activation matrix $ \mH $.

$$ \vsigma = \sqrt{ \delta + \frac{1}{m} \sum_i \left(\mH_{i,:} - \vmu\right)^2} $$

where, $ \delta $ is a small positive term, say $ \delta = 10^{-10} $, to ensure that we do not have zero standard deviation.

With these batch-level statistics, we can now normalize the activations $ \mH $ by standardizing all columns to have zero mean and unit variance. This can be achieved by subtracting the mean $ \vmu $ from each row, and then dividing the row by the vector of standard deviations.

$$ \dash{\mH_{i,:}} = \frac{\mH_{i,:} - \vmu}{\vsigma}, ~~~~\forall i=1,\ldots,m $$

Note that the small positive value of $ \delta $ in our calculation of $ \vsigma $ will ensure that the divisor is not zero.

That's it.

Maintaining expressive power with batchnorm

As discussed earlier, the batchnorm will limit the expressive power of the neural network through restricting parameter updates. This is effectively the result of reducing the standard deviation of the activations to 1 and the mean to 0. Maybe this is too restrictive. And leaving the standard deviation as $ \vsigma $ and mean as $ \vmu $ is too flexible. How about a learnable middle ground?

We could allow the standard variation to be $ \gamma $ and the mean to be $ \beta $ and let these parameters $ \gamma, \beta $ be learnable. This reparametrization strategy is quite popular. It replaces the normalized activation matrix $ \dash{\mH} $ with the reparametrized matrix $ \gamma \dash{\mH} + \beta $.

It may seem counterintuitive to first remove the mean $ \vmu $ of the activations and then reintroduce it as $ \beta $, a learnable parameter. But the mean of activations, $ \vmu $, is a result of complicated interactions between many layers of the network. The parameter $ \beta $ is a learnable parameter, offering more control, and better learnable by gradient descent.

Please support us

Help us create more engaging and effective content and keep it free of paywalls and advertisements!

Please donate

Subscribe for article updates

Stay up to date with new material for free.