Introduction to regularization for deep learning

This learning module has many interactive demos. It is easier to work with them on a larger screen. Bookmark and revisit if you are currently on a small screen device.

\(\DeclareMathOperator*{\argmax}{arg\,max} \DeclareMathOperator*{\argmin}{arg\,min} \DeclareMathOperator*{\asterisk}{\ast} \newcommand{\sup}{\text{sup}} \newcommand{\inf}{\text{inf}} \newcommand{\min}{\text{min}\;} \newcommand{\max}{\text{max}\;} \newcommand{\maxunder}[1]{\underset{#1}{\max}} \newcommand{\minunder}[1]{\underset{#1}{\min}} \newcommand{\real}{\mathbb{R}} \newcommand{\natural}{\mathbb{N}} \newcommand{\integer}{\mathbb{Z}} \newcommand{\rational}{\mathbb{Q}} \newcommand{\irrational}{\mathbb{I}} \newcommand{\complex}{\mathbb{C}} \newcommand{\cardinality}[1]{|#1|} \newcommand{\vec}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\star}[1]{#1^*} \newcommand{\inv}[1]{#1^{-1}} \newcommand{\indicator}[1]{\mathcal{I}(#1)} \renewcommand{\BigO}[1]{\mathcal{O}(#1)} \renewcommand{\BigOsymbol}{\mathcal{O}} \renewcommand{\smallo}[1]{\mathcal{o}(#1)} \renewcommand{\smallosymbol}[1]{\mathcal{o}} \newcommand{\set}[1]{\mathbb{#1}} \newcommand{\complement}[1]{#1^c} \newcommand{\powerset}[1]{\mathcal{P}(#1)} \newcommand{\setdiff}{\setminus} \newcommand{\setsymmdiff}{\oplus} \newcommand{\dash}[1]{#1^{'}} \newcommand{\permutation}[2]{{}_{#1} \mathrm{ P }_{#2}} \newcommand{\combination}[2]{{}_{#1} \mathrm{ C }_{#2}} \newcommand{\prob}[1]{P(#1)} \newcommand{\pmf}[1]{P(#1)} \newcommand{\pdf}[1]{p(#1)} \newcommand{\cdf}[1]{F(#1)} \newcommand{\expect}[2]{E_{#1}\left[#2\right]} \newcommand{\entropy}[1]{\mathcal{H}\left[#1\right]} \newcommand{\expe}[1]{\mathrm{e}^{#1}} \newcommand{\textexp}[1]{\text{exp}\left(#1\right)} \def\independent{\perp\!\!\!\perp} \def\notindependent{\not\!\independent} \newcommand{\yhat}{\hat{y}} \newcommand{\vs}{\vec{s}} \newcommand{\vt}{\vec{t}} \newcommand{\vu}{\vec{u}} \newcommand{\vv}{\vec{v}} \newcommand{\vw}{\vec{w}} \newcommand{\vx}{\vec{x}} \newcommand{\vy}{\vec{y}} \newcommand{\vz}{\vec{z}} \newcommand{\va}{\vec{a}} \newcommand{\vb}{\vec{b}} \newcommand{\vc}{\vec{c}} \newcommand{\vd}{\vec{d}} \newcommand{\ve}{\vec{e}} \newcommand{\vg}{\vec{g}} \newcommand{\vh}{\vec{h}} \newcommand{\vi}{\vec{i}} \newcommand{\vk}{\vec{k}} \newcommand{\vo}{\vec{o}} \newcommand{\vp}{\vec{p}} \newcommand{\vq}{\vec{q}} \newcommand{\vr}{\vec{r}} \newcommand{\vs}{\vec{s}} \newcommand{\vmu}{\vec{\mu}} \newcommand{\vsigma}{\vec{\sigma}} \newcommand{\vphi}{\vec{\phi}} \newcommand{\vtau}{\vec{\tau}} \newcommand{\vtheta}{\vec{\theta}} \newcommand{\mA}{\mat{A}} \newcommand{\mB}{\mat{B}} \newcommand{\mC}{\mat{C}} \newcommand{\mD}{\mat{D}} \newcommand{\mE}{\mat{E}} \newcommand{\mH}{\mat{H}} \newcommand{\mK}{\mat{K}} \newcommand{\mP}{\mat{P}} \newcommand{\mQ}{\mat{Q}} \newcommand{\mR}{\mat{R}} \newcommand{\mS}{\mat{S}} \newcommand{\mU}{\mat{U}} \newcommand{\mV}{\mat{V}} \newcommand{\mW}{\mat{W}} \newcommand{\mX}{\mat{X}} \newcommand{\mY}{\mat{Y}} \newcommand{\mZ}{\mat{Z}} \newcommand{\mI}{\mat{I}} \newcommand{\mLambda}{\mat{\Lambda}} \newcommand{\mSigma}{\mat{\Sigma}} \newcommand{\mTheta}{\mat{\theta}} \newcommand{\setsymb}[1]{#1} \newcommand{\sA}{\setsymb{A}} \newcommand{\sB}{\setsymb{B}} \newcommand{\sC}{\setsymb{C}} \newcommand{\sO}{\setsymb{O}} \newcommand{\sP}{\setsymb{P}} \newcommand{\sQ}{\setsymb{Q}} \newcommand{\sH}{\setsymb{H}} \newcommand{\sX}{\setsymb{X}} \newcommand{\sY}{\setsymb{Y}} \newcommand{\norm}[2]{||{#1}||_{#2}} \newcommand{\infnorm}[1]{\norm{#1}{\infty}} \newcommand{\fillinblank}{\text{ }\underline{\text{ ? }}\text{ }} \newcommand{\lbrace}{\left\{} \newcommand{\rbrace}{\right\}} \newcommand{\set}[1]{\lbrace #1 \rbrace} \newcommand{\seq}[1]{\left( #1 \right)} \newcommand{\ndim}{N} \newcommand{\ndimsmall}{n} \newcommand{\dataset}{\mathbb{D}} \newcommand{\ndata}{D} \newcommand{\ndatasmall}{d} \newcommand{\labeledset}{\mathbb{L}} \newcommand{\nlabeled}{L} \newcommand{\nlabeledsmall}{l} \newcommand{\unlabeledset}{\mathbb{U}} \newcommand{\nunlabeled}{U} \newcommand{\nunlabeledsmall}{u} \newcommand{\nclass}{M} \newcommand{\nclasssmall}{m} \newcommand{\loss}{\mathcal{L}} \newcommand{\sign}{\text{sign}} \newcommand{\Gauss}{\mathcal{N}} \newcommand{\hadamard}{\circ} \newcommand{\doh}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\dox}[1]{\doh{#1}{x}} \newcommand{\doy}[1]{\doh{#1}{y}} \newcommand{\doxx}[1]{\doh{#1}{x^2}} \newcommand{\doyy}[1]{\doh{#1}{y^2}} \newcommand{\doxy}[1]{\frac{\partial #1}{\partial x \partial y}} \newcommand{\doyx}[1]{\frac{\partial #1}{\partial y \partial x}} \newcommand{\qed}{\tag*{$\blacksquare$}}\)

        Regularization
        Deep Learning
      

Introduction

Regularization is a collection of strategies that enable a learning algorithm to generalize better on new inputs, often times at the expense of reduced performance on the training set. In this sense, it is a strategy to reduce the possibility of overfitting the training data, and possibly reduce variance of the model by increasing some bias. Some regularization approaches are commonly used for other machine learning approaches, but as we will see in this article, some strategies are specifically designed for regularizing deep learning.

Prerequisites

To understand regularization approaches for deep learning, we recommend familiarity with the concepts in

Introduction to machine learning: An introduction to basic concepts in machine learning such as classification, training instances, features, and feature types.
Multilayer perceptrons

Follow the above links to first get acquainted with the corresponding concepts.

Parameter constraints

One method of regularizing deep neural networks is to constrain the parameter values, for example, by applying a suitable norm as a penalty on the parameters or weights of the model. If $ \loss $ denotes the unregularized loss of the neural network, then we incorporate the regularization term $ \Omega(\mTheta) $ on the parameters $ \mTheta $ of the model.

$$ \loss_{\text{regularized}} = \loss + \alpha \Omega(\mTheta) $$

where, $ \alpha \in \real $ is a hyperparameter that controls the impact of the regularization term.

A popular form of penalty on the weights is the $ L_2 $ norm, also known as weight decay in neural networks, which is applied on each weight parameter in the network.

$$ \Omega(\mTheta) = \sum_{\vw \in \mTheta} \norm{\vw}{2} $$

where, the $ L_2 $ norm is defined as $ \norm{\vw}{2} = \vw^T\vw $.

Sparsity can be enforced in the model parameters by using an $L_1$ norm instead, just like in lasso regression.

$$ \Omega(\mTheta) = \sum_{\vw \in \mTheta} \norm{\vw}{1} $$

where, the $ L_1 $ norm is defined as $ \norm{\vw}{1} = \sum_{i} |w_i| $.

For more details on the effect of $ L_2 $ and $ L_1 $ regularization in machine learning models, refer to our comprehensive article on norm-based regularization in machine learning.

Parameter tying

An enhanced form of parameter norm penalties is that of parameter tying. Intuitively, if we believe that certain parameters in the model should be close to each other, we can enforce this belief by specifically requiring them to be close in values. For example, if $ \vw_a $ and $ \vw_b $ are two weights in a network that we expect should be similar (maybe because they perform the same task or have similar input/output distributions), then we can enforce them to be closer by introducing a norm on their difference as an additional loss term.

$$ \loss_{\text{tying}} = \loss + \alpha \norm{\vw_a - \vw_b}{2} $$

The example here shows the $ L_2 $ norm, but any other suitable norm may also be used.

Such parameter tying is particularly useful in the case of multitask learning approaches that typically end up with similar versions of the weight vectors for multiple related tasks. This has an added benefit of being able to learn each task with limited amount of training data because each task also receives additional supervision from the weight vectors of other related tasks.

Early stopping

The net effect of training longer on a dataset is overfitting that dataset, as the model parameters adapt severely to conform to the training data. Such models do not generalize well to new unseen data.

An obvious solution to this is, well, training for fewer epochs. This is the main idea behind early stopping. The art here is to train for just the right number of epochs. How do we do that?

The early stopping approach works as follows. At the end of each epoch, save the corresponding validation error and a snapshot of the current model parameters, as a check point. After completing the planned maximum number of training iterations, discover the best performing epoch and apply the model parameters from that epoch as the final trained model. It is also possible to terminate the training early by checking if the best validation error so far has not reduced any further in a predefined number of iterations.

An obvious benefit of early stopping is that it requires no change to the model or the loss function. This is particularly important compared to weight decay regularization wherein the restricting the weights severely may lead to poor performance. That being said, it may be used in conjunction with other regularization strategies such as parameter norm penalties.

One drawback of early stopping is the additional storage required to save a snapshot of model parameters after each epoch. Another drawback, particularly in training data starved problems, is the need to hold out a validation set that is separate from the training set. This can be a major problem as a separate validation implies fewer examples actually used for training. This issue can be mitigated by retraining on the entire dataset (including the validation set), once the best number of epochs have been discovered. But, whether to train for the same number of epochs, or the same number of parameter updates, is a subtle art!

Sparse representations

An alternative approach to parameter norm penalties is that of sparse representations. Instead of penalizing the parameters, the sparse representation approach applies penalizing norms on activations of the layers in the model. Constraining the activations of a model layer indirectly implies a penalty on the model parameters, resulting in regularization.

If $ \vh $ denotes the activations of the various layers in the network, then the updated loss can be

$$ \loss_{\text{sparserepr}} = \loss + \alpha \sum_{\vh} \Omega(\vh) $$

We can use the now familiar $ L_2 $ or $ L_1 $ norms on these activations.

Dropout

You may remember bootstrap aggregating, also known as bagging, from our comprehensive article on random forests. Bagging is an effective strategy to reduce variance, without introducing additional bias, of a predictor by building an ensemble of such models trained on randomly sampled subsets of the training dataset. Building an ensemble using bagging is also likely to help with reducing variance or other high capacity models, including deep neural networks. But unlike training a tree, neural network training is typically more computationally demanding. Building an ensemble of such models will likely be even more computationally expensive. Can we approximate the ensemble somehow? Yes, with the dropout strategy.

The dropout strategy works by randomly dropping out some units in the network during minibatch training. A unit can be dropped out of the network by zeroing the activations or output of that unit. This is typically achieved by multiplying the activations of all units (except the output units) in the network with a randomly sampled binary mask for each input example. For example, input layer units can be dropped with 0.5 probability and hidden layer units may be dropped with 0.2 probability.

The net effect of such random muting of units in the network is the creation of an inherent ensemble of exponentially many networks, formed of all possible networks resulting from the dropout. During prediction, the ensemble prediction can be obtained by averaging over the outputs of many networks resulting from the dropout. Typically 10 to 20 randomly sampled binary masks provide good variance reduction in the outputs.

Please support us

Help us create more engaging and effective content and keep it free of paywalls and advertisements!

Please donate

Let's connect

Please share your comments, questions, encouragement, and feedback.

Regularization

Deep Learning