Introduction to logistic regression classifier

This learning module has many interactive demos. It is easier to work with them on a larger screen. Bookmark and revisit if you are currently on a small screen device.

\(\DeclareMathOperator*{\argmax}{arg\,max} \DeclareMathOperator*{\argmin}{arg\,min} \DeclareMathOperator*{\asterisk}{\ast} \newcommand{\sup}{\text{sup}} \newcommand{\inf}{\text{inf}} \newcommand{\min}{\text{min}\;} \newcommand{\max}{\text{max}\;} \newcommand{\maxunder}[1]{\underset{#1}{\max}} \newcommand{\minunder}[1]{\underset{#1}{\min}} \newcommand{\real}{\mathbb{R}} \newcommand{\natural}{\mathbb{N}} \newcommand{\integer}{\mathbb{Z}} \newcommand{\rational}{\mathbb{Q}} \newcommand{\irrational}{\mathbb{I}} \newcommand{\complex}{\mathbb{C}} \newcommand{\cardinality}[1]{|#1|} \newcommand{\vec}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\star}[1]{#1^*} \newcommand{\inv}[1]{#1^{-1}} \newcommand{\indicator}[1]{\mathcal{I}(#1)} \renewcommand{\BigO}[1]{\mathcal{O}(#1)} \renewcommand{\BigOsymbol}{\mathcal{O}} \renewcommand{\smallo}[1]{\mathcal{o}(#1)} \renewcommand{\smallosymbol}[1]{\mathcal{o}} \newcommand{\set}[1]{\mathbb{#1}} \newcommand{\complement}[1]{#1^c} \newcommand{\powerset}[1]{\mathcal{P}(#1)} \newcommand{\setdiff}{\setminus} \newcommand{\setsymmdiff}{\oplus} \newcommand{\dash}[1]{#1^{'}} \newcommand{\permutation}[2]{{}_{#1} \mathrm{ P }_{#2}} \newcommand{\combination}[2]{{}_{#1} \mathrm{ C }_{#2}} \newcommand{\prob}[1]{P(#1)} \newcommand{\pmf}[1]{P(#1)} \newcommand{\pdf}[1]{p(#1)} \newcommand{\cdf}[1]{F(#1)} \newcommand{\expect}[2]{E_{#1}\left[#2\right]} \newcommand{\entropy}[1]{\mathcal{H}\left[#1\right]} \newcommand{\expe}[1]{\mathrm{e}^{#1}} \newcommand{\textexp}[1]{\text{exp}\left(#1\right)} \def\independent{\perp\!\!\!\perp} \def\notindependent{\not\!\independent} \newcommand{\yhat}{\hat{y}} \newcommand{\vs}{\vec{s}} \newcommand{\vt}{\vec{t}} \newcommand{\vu}{\vec{u}} \newcommand{\vv}{\vec{v}} \newcommand{\vw}{\vec{w}} \newcommand{\vx}{\vec{x}} \newcommand{\vy}{\vec{y}} \newcommand{\vz}{\vec{z}} \newcommand{\va}{\vec{a}} \newcommand{\vb}{\vec{b}} \newcommand{\vc}{\vec{c}} \newcommand{\vd}{\vec{d}} \newcommand{\ve}{\vec{e}} \newcommand{\vg}{\vec{g}} \newcommand{\vh}{\vec{h}} \newcommand{\vi}{\vec{i}} \newcommand{\vk}{\vec{k}} \newcommand{\vo}{\vec{o}} \newcommand{\vp}{\vec{p}} \newcommand{\vq}{\vec{q}} \newcommand{\vr}{\vec{r}} \newcommand{\vs}{\vec{s}} \newcommand{\vmu}{\vec{\mu}} \newcommand{\vsigma}{\vec{\sigma}} \newcommand{\vphi}{\vec{\phi}} \newcommand{\vtau}{\vec{\tau}} \newcommand{\vtheta}{\vec{\theta}} \newcommand{\mA}{\mat{A}} \newcommand{\mB}{\mat{B}} \newcommand{\mC}{\mat{C}} \newcommand{\mD}{\mat{D}} \newcommand{\mE}{\mat{E}} \newcommand{\mH}{\mat{H}} \newcommand{\mK}{\mat{K}} \newcommand{\mP}{\mat{P}} \newcommand{\mQ}{\mat{Q}} \newcommand{\mR}{\mat{R}} \newcommand{\mS}{\mat{S}} \newcommand{\mU}{\mat{U}} \newcommand{\mV}{\mat{V}} \newcommand{\mW}{\mat{W}} \newcommand{\mX}{\mat{X}} \newcommand{\mY}{\mat{Y}} \newcommand{\mZ}{\mat{Z}} \newcommand{\mI}{\mat{I}} \newcommand{\mLambda}{\mat{\Lambda}} \newcommand{\mSigma}{\mat{\Sigma}} \newcommand{\mTheta}{\mat{\theta}} \newcommand{\setsymb}[1]{#1} \newcommand{\sA}{\setsymb{A}} \newcommand{\sB}{\setsymb{B}} \newcommand{\sC}{\setsymb{C}} \newcommand{\sO}{\setsymb{O}} \newcommand{\sP}{\setsymb{P}} \newcommand{\sQ}{\setsymb{Q}} \newcommand{\sH}{\setsymb{H}} \newcommand{\sX}{\setsymb{X}} \newcommand{\sY}{\setsymb{Y}} \newcommand{\norm}[2]{||{#1}||_{#2}} \newcommand{\infnorm}[1]{\norm{#1}{\infty}} \newcommand{\fillinblank}{\text{ }\underline{\text{ ? }}\text{ }} \newcommand{\lbrace}{\left\{} \newcommand{\rbrace}{\right\}} \newcommand{\set}[1]{\lbrace #1 \rbrace} \newcommand{\seq}[1]{\left( #1 \right)} \newcommand{\ndim}{N} \newcommand{\ndimsmall}{n} \newcommand{\dataset}{\mathbb{D}} \newcommand{\ndata}{D} \newcommand{\ndatasmall}{d} \newcommand{\labeledset}{\mathbb{L}} \newcommand{\nlabeled}{L} \newcommand{\nlabeledsmall}{l} \newcommand{\unlabeledset}{\mathbb{U}} \newcommand{\nunlabeled}{U} \newcommand{\nunlabeledsmall}{u} \newcommand{\nclass}{M} \newcommand{\nclasssmall}{m} \newcommand{\loss}{\mathcal{L}} \newcommand{\sign}{\text{sign}} \newcommand{\Gauss}{\mathcal{N}} \newcommand{\hadamard}{\circ} \newcommand{\doh}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\dox}[1]{\doh{#1}{x}} \newcommand{\doy}[1]{\doh{#1}{y}} \newcommand{\doxx}[1]{\doh{#1}{x^2}} \newcommand{\doyy}[1]{\doh{#1}{y^2}} \newcommand{\doxy}[1]{\frac{\partial #1}{\partial x \partial y}} \newcommand{\doyx}[1]{\frac{\partial #1}{\partial y \partial x}} \newcommand{\qed}{\tag*{$\blacksquare$}}\)

        Logistic regression
        Machine Learning
      

Introduction

The logistic regression classifier is a discriminative model for binary classification. Before the advent of deep learning and its easy-to-use libraries, the logistic regression classifier was one of the widely deployed classifiers for machine learning applications. It is still widely used for linear classification, especially in the medical domain.

Prerequisites

To understand the logistic regression classifier, we recommend familiarity with the concepts in

Introduction to machine learning: An introduction to basic concepts in machine learning such as classification, training instances, features, and feature types.
Probability: A sound understanding of conditional and marginal probabilities is desirable.
Maximum likelihood estimation: To understand the training for logistic regression classifier.

Follow the above links to first get acquainted with the corresponding concepts.

Problem setting

In classification, the goal of the predictive model is to identify the class that generated a particular instance.

Consider such an instance $ \vx \in \real^N $, a vector consisting of $ N $ features, $\vx = [x_1, x_2, \ldots, x_N] $.

We need to assign it to one of the $ 2 $ classes $ C_1 $ and $ C_2 $ depending on the values of the $ N $ features .

Predictive model

Logistic regression uses a special form for the predictive model. It defines the probability of the first class as

$$ P(C_1|\vx) = \sigma(\vw^T \vx + b) $$

where $ \vw \in \real^\ndim $ and $ b \in \real $ are parameters of the logistic regression classifier. The parameter $ \vw $ is also known as the weights of the model. In the above Equation, the $ \sigma $ is a special function known as the logistic sigmoid function. We will describe its nature shortly.

Since this is a binary classification setting, the probability of belonging to the other class is

$$ P(C_2|\vx) = 1 - P(C_1|\vx) $$

Thus, the predicted class label $ \yhat $ is the class with the highest probability.

$$ \yhat = \argmax_{c \in \set{C_1,C_2}} P(C=c | \vx) $$

Logistic sigmoid function

The logistic regression name comes from the logistic sigmoid function used in the predictive model. The sigmoid function is also quite commonly used as an activation function in neural networks.

The sigmoid function is defined as

$$ \sigma(a) = \frac{1}{1 + \expe{-a}} $$

In the case of logistic regression, it would be

$$ \sigma(\vw^T\vx + b) = \frac{1}{1 + \expe{-(\vw^T \vx + b)}} $$

Nature of the sigmoid function

Note that the predictive model uses sigmoid function to calculate probability. This is possible because the output of the sigmoid function is bounded in the range $ [0,1] $.

The sigmoid function output is $ 0.5 $ only when its input is $ 0 $. For positive inputs, the sigmoid returns values in the range $ (0.5,1] $. For negative inputs, the sigmoid returns values in the range $ [0,0.5) $.

Nature of the predictive model

Now, let us try to understand the effect of changing the weight vector $ \vw $ and the bias $ b $ on the predictive model.

The weight vector $ \vw = [w_1, w_2] $. Drag the cricle to change the vector

$ \sigma(\vw^T\vx + b) $

Classification model

Observations about the predictive model

If you have tried the interactive demo of the predictive model in the previous section, you should note a few things.

The weight vector $ \vw $ is always perpendicular to the decision boundary, the so-called separating hyperplane between the orange and blue classes. Thus, a rotation of the weight vector results in a corresponding rotation of the decision boundary.
The weight vector $ \vw $ points in the direction of increasing value of the function $ \sigma (\vw^T\vx + b) $.
Increasing the magnitude of the vector (dragging it in the same direction away from the origin) does not change the decision boundary. This intuition plays a key role in regularization, as we shall see in the norm-based regularization in machine learning.
The bias term $ b $ has the net effect of sliding the decision boundary away from the origin. When $ b = 0 $, the decision boundary passes through the origin.

Training logistic regression

Training a logistic regression classifier involves discovering suitable values for the parameters — $ \vw $ and $ b $.

The parameters are optimized for maximizing the likelihood of observed data — the training set — by (maximum likelihood estimation) (MLE).

Suppose $ \labeledset = \set{(\vx_1, y_1), \ldots, (\vx_\nlabeled, y_\nlabeled)} $ denotes the training set consisting of $ \nlabeled $ training instances. Assume that $ y_\nlabeledsmall = 1 $ if $ \vx_\nlabeledsmall $ belongs to the class $ C_1 $, and zero if it belongs to the class $ C_2 $. The likelihood in the case of logistic regression is

$$ P(\labeledset|\vw) = \prod_{\nlabeledsmall=1}^\nlabeled P(C_1|\vx_\nlabeledsmall)^{y_\nlabeledsmall} \left(1 - P(C_1|\vx_\nlabeledsmall)\right)^{1 - y_\nlabeledsmall} $$

Note that the term $P(C_1|\vx_\nlabeledsmall)^{y_\nlabeledsmall} $ is activated if the instance $ \vx_\nlabeledsmall $ belongs to the class $ C_1 $, since $y_\nlabeledsmall = 1 $ in that case.

The latter term $ \left(1 - P(C_1|\vx_\nlabeledsmall)\right)^{1 - y_\nlabeledsmall} $ is activated when the training instance $ \vx_\nlabeledsmall $ belongs to the class $C_2$, since $y_\nlabeledsmall = 0 $ in that case.

With the likelihood function defined this way, the training proceeds by MLE using popular optimization approaches. Particularly in the case of logisitc regression, there is no closed-form solution to the MLE optimization problem. Hence, iterative approaches such as minibatch stochastic gradient descent may be used.

Training demo

Let us fit a logistic regression model to some training data. Since the model parameters do not have a closed form solution, we will use an iterative approach with minibatch-SGD to minimize the negative log-likelihood.

Fitting logistic regression to training data

Training accuracy per training epoch

Being a linear classifier, the trained model comfortably separates linearly separable classes. However, it is ineffective at non-linearly separable scenarios.

Dealing with feature types

Note that the predictive model involves a dot product of the weight vector $ \vw $ and the instance vector $ \vx $. This is easy for binary and continuous features since both can be treated as real-valued features.

In the case of categorical features a direct dot product with the weight vector is not meaningful. Therefore, we need to first preprocess the categorical variables using one-hot encoding to arrive at a binary feature representation.

Please support us

Help us create more engaging and effective content and keep it free of paywalls and advertisements!

Please donate

Subscribe for article updates

Stay up to date with new material for free.