Limits and continuity

This learning module has many interactive demos. It is easier to work with them on a larger screen. Bookmark and revisit if you are currently on a small screen device.

\(\DeclareMathOperator*{\argmax}{arg\,max} \DeclareMathOperator*{\argmin}{arg\,min} \DeclareMathOperator*{\asterisk}{\ast} \newcommand{\sup}{\text{sup}} \newcommand{\inf}{\text{inf}} \newcommand{\min}{\text{min}\;} \newcommand{\max}{\text{max}\;} \newcommand{\maxunder}[1]{\underset{#1}{\max}} \newcommand{\minunder}[1]{\underset{#1}{\min}} \newcommand{\real}{\mathbb{R}} \newcommand{\natural}{\mathbb{N}} \newcommand{\integer}{\mathbb{Z}} \newcommand{\rational}{\mathbb{Q}} \newcommand{\irrational}{\mathbb{I}} \newcommand{\complex}{\mathbb{C}} \newcommand{\cardinality}[1]{|#1|} \newcommand{\vec}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\star}[1]{#1^*} \newcommand{\inv}[1]{#1^{-1}} \newcommand{\indicator}[1]{\mathcal{I}(#1)} \renewcommand{\BigO}[1]{\mathcal{O}(#1)} \renewcommand{\BigOsymbol}{\mathcal{O}} \renewcommand{\smallo}[1]{\mathcal{o}(#1)} \renewcommand{\smallosymbol}[1]{\mathcal{o}} \newcommand{\set}[1]{\mathbb{#1}} \newcommand{\complement}[1]{#1^c} \newcommand{\powerset}[1]{\mathcal{P}(#1)} \newcommand{\setdiff}{\setminus} \newcommand{\setsymmdiff}{\oplus} \newcommand{\dash}[1]{#1^{'}} \newcommand{\permutation}[2]{{}_{#1} \mathrm{ P }_{#2}} \newcommand{\combination}[2]{{}_{#1} \mathrm{ C }_{#2}} \newcommand{\prob}[1]{P(#1)} \newcommand{\pmf}[1]{P(#1)} \newcommand{\pdf}[1]{p(#1)} \newcommand{\cdf}[1]{F(#1)} \newcommand{\expect}[2]{E_{#1}\left[#2\right]} \newcommand{\entropy}[1]{\mathcal{H}\left[#1\right]} \newcommand{\expe}[1]{\mathrm{e}^{#1}} \newcommand{\textexp}[1]{\text{exp}\left(#1\right)} \def\independent{\perp\!\!\!\perp} \def\notindependent{\not\!\independent} \newcommand{\yhat}{\hat{y}} \newcommand{\vs}{\vec{s}} \newcommand{\vt}{\vec{t}} \newcommand{\vu}{\vec{u}} \newcommand{\vv}{\vec{v}} \newcommand{\vw}{\vec{w}} \newcommand{\vx}{\vec{x}} \newcommand{\vy}{\vec{y}} \newcommand{\vz}{\vec{z}} \newcommand{\va}{\vec{a}} \newcommand{\vb}{\vec{b}} \newcommand{\vc}{\vec{c}} \newcommand{\vd}{\vec{d}} \newcommand{\ve}{\vec{e}} \newcommand{\vg}{\vec{g}} \newcommand{\vh}{\vec{h}} \newcommand{\vi}{\vec{i}} \newcommand{\vk}{\vec{k}} \newcommand{\vo}{\vec{o}} \newcommand{\vp}{\vec{p}} \newcommand{\vq}{\vec{q}} \newcommand{\vr}{\vec{r}} \newcommand{\vs}{\vec{s}} \newcommand{\vmu}{\vec{\mu}} \newcommand{\vsigma}{\vec{\sigma}} \newcommand{\vphi}{\vec{\phi}} \newcommand{\vtau}{\vec{\tau}} \newcommand{\vtheta}{\vec{\theta}} \newcommand{\mA}{\mat{A}} \newcommand{\mB}{\mat{B}} \newcommand{\mC}{\mat{C}} \newcommand{\mD}{\mat{D}} \newcommand{\mE}{\mat{E}} \newcommand{\mH}{\mat{H}} \newcommand{\mK}{\mat{K}} \newcommand{\mP}{\mat{P}} \newcommand{\mQ}{\mat{Q}} \newcommand{\mR}{\mat{R}} \newcommand{\mS}{\mat{S}} \newcommand{\mU}{\mat{U}} \newcommand{\mV}{\mat{V}} \newcommand{\mW}{\mat{W}} \newcommand{\mX}{\mat{X}} \newcommand{\mY}{\mat{Y}} \newcommand{\mZ}{\mat{Z}} \newcommand{\mI}{\mat{I}} \newcommand{\mLambda}{\mat{\Lambda}} \newcommand{\mSigma}{\mat{\Sigma}} \newcommand{\mTheta}{\mat{\theta}} \newcommand{\setsymb}[1]{#1} \newcommand{\sA}{\setsymb{A}} \newcommand{\sB}{\setsymb{B}} \newcommand{\sC}{\setsymb{C}} \newcommand{\sO}{\setsymb{O}} \newcommand{\sP}{\setsymb{P}} \newcommand{\sQ}{\setsymb{Q}} \newcommand{\sH}{\setsymb{H}} \newcommand{\sX}{\setsymb{X}} \newcommand{\sY}{\setsymb{Y}} \newcommand{\norm}[2]{||{#1}||_{#2}} \newcommand{\infnorm}[1]{\norm{#1}{\infty}} \newcommand{\fillinblank}{\text{ }\underline{\text{ ? }}\text{ }} \newcommand{\lbrace}{\left\{} \newcommand{\rbrace}{\right\}} \newcommand{\set}[1]{\lbrace #1 \rbrace} \newcommand{\seq}[1]{\left( #1 \right)} \newcommand{\ndim}{N} \newcommand{\ndimsmall}{n} \newcommand{\dataset}{\mathbb{D}} \newcommand{\ndata}{D} \newcommand{\ndatasmall}{d} \newcommand{\labeledset}{\mathbb{L}} \newcommand{\nlabeled}{L} \newcommand{\nlabeledsmall}{l} \newcommand{\unlabeledset}{\mathbb{U}} \newcommand{\nunlabeled}{U} \newcommand{\nunlabeledsmall}{u} \newcommand{\nclass}{M} \newcommand{\nclasssmall}{m} \newcommand{\loss}{\mathcal{L}} \newcommand{\sign}{\text{sign}} \newcommand{\Gauss}{\mathcal{N}} \newcommand{\hadamard}{\circ} \newcommand{\doh}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\dox}[1]{\doh{#1}{x}} \newcommand{\doy}[1]{\doh{#1}{y}} \newcommand{\doxx}[1]{\doh{#1}{x^2}} \newcommand{\doyy}[1]{\doh{#1}{y^2}} \newcommand{\doxy}[1]{\frac{\partial #1}{\partial x \partial y}} \newcommand{\doyx}[1]{\frac{\partial #1}{\partial y \partial x}} \newcommand{\qed}{\tag*{$\blacksquare$}}\)

        Limits and continuity
        Calculus
      

A clear knowledge of limits and continuity is essential to understand more advanced topics in calculus, such as derivatives.

This article will prepare you for these concepts.

Prerequisites

To understand limits and continuity, we recommend familiarity with the concepts in

Follow the above links to first get acquainted with the corresponding concepts.

Limit

The sigmoid function is a popular function in machine learning. It is defined as follows

$$ \begin{align} & \sigma(x) = \frac{1}{1 + \exp^{-x}} \\ \end{align}$$

Crazy people like us, the machine learners, often wonder: what is the maximum and minimum value of $ \sigma(x) $ for $ x \in \real $?

We try various values of $ x $. Try this interactive function plot to explore the values of the sigmoid function as you change value of $ x $ by dragging the grey circle.

We observe a few things.

For positive increasing values of $ x $, the outputs are increasing. Moreover, they are getting closer to 1.0, but not quite there.
For negative decreasing values of $ x $, the outputs are decreasing and approaching 0.0, but not quite there.

How do we denote these concepts mathematically? With the limits notation.

$$ \lim_{x \to \infty} \sigma(x) = 1 $$ $$ \lim_{x \to -\infty} \sigma(x) = 0 $$

We read this to mean that as the input $ x $ moves closer to $ \infty $, the output of the function $ \sigma(x) $ approaches 1. Does it actually take on the value 1 will depend on what it means for the $ x $ to take on the value $ \infty $.

Limits are an important mathematical concept regarding the behavior of a function near a point. So, $ \lim_{x \to p} f(x) = L $, means that the output of the function $ f(x) $ gets closer to $ L $ as the input $ x $ gets closer to $ p $.

Limits of example functions

Below, we have provided some example functions and their limits.

One-sided limits

Let's take another popular function in machine learning. The Heaviside step function or the unit step function is defined as follows

$$ H(x) = \begin{cases} 1, \text{ if } x \ge 0 \\ 0, \text{ otherwise } \\ \end{cases} $$

So, we know that $ H(0) = 1 $. We can also write this in the limits notation introduced earlier as

$$ \lim_{x \to 0} H(x) = 1 $$

Or, can we? Are we coming closer to zero from the positive side or the negative side? If we are approaching from the positive side, then sure, we can claim that to be 1. But, if we are visiting zero from the negative side then a point very very very very close to zero will have a $H(x) = 0 $. Mathematically, if $ 0^+ $ and $ 0^{-} $ denote the positive and negative side of zero respectively, then we can denote this as follows

$$ \lim_{x \to 0^+} H(x) = 1 $$ $$ \lim_{x \to 0^-} H(x) = 0 $$

Such one-sided limits, that depend on the direction of approach, are known as, well, one-sided limits.

Continuity

In the case of the Heaviside function, we note that the two one-sided limits do not match. This means cannot really write $ \lim_{x \to 0} H(x) $, without specifying the direction of approach.

In other words, the limit $ \lim_{x \to 0 } H(x) $ does not exist!

It is then said that the function $ H(x) $ is discontinuous at the point 0. For any other input in the domain of $ x $, the limit exists. Thus, one says that the function $ H(x) $ is continuous on the negative interval and on the positive interval, but not continous at zero.

In general, for any function $ f(x) $, if $ \lim_{x \to p^+} f(x) \neq \lim_{x \to p^-} f(x) $, then it is said that the limit does not exist at the point $ p $.

A function $ f(x) $ is said to be continuous at a point $ p $, if the limit $ \lim_{x \to p} f(x) $ exists, and that $ \lim_{x \to p} f(x) = f(p) $.

A function $ f(x) $ is said to be continuous within an interval $ [a,b] $, if it is continuous at every point within the interval $ [a,b] $.

So, is sigmoid a continuous function on $ \real $? Figure that one out.

Examples of discontinuous functions

Below, we have provided some example functions and their limits. The points of discontinuity are highlighted in red.

Where to next?

With a sound understanding of limits and continuity, it is time to dive into derivatives.

Already a calculus expert? Check out comprehensive courses on multivariate calculus, machine learning or deep learning.

Please support us

Help us create more engaging and effective content and keep it free of paywalls and advertisements!

Please donate

Let's connect

Please share your comments, questions, encouragement, and feedback.