Jensen's inequality

Jensen's inequality generalizes the statement that a secant line of a convex function lies above its graph.

File:Convex 01.ogv

In mathematics, Jensen's inequality, named after the Danish mathematician Johan Jensen, relates the value of a convex function of an integral to the integral of the convex function. It was proved by Jensen in 1906, building on an earlier proof of the same inequality for doubly-differentiable functions by Otto Hölder in 1889. Given its generality, the inequality appears in many forms depending on the context, some of which are presented below. In its simplest form the inequality states that the convex transformation of a mean is less than or equal to the mean applied after convex transformation (or equivalently, the opposite inequality for concave transformations).

Jensen's inequality generalizes the statement that the secant line of a convex function lies above the graph of the function, which is Jensen's inequality for two points: the secant line consists of weighted means of the convex function (for t ∈ [0,1]),

$tf(x_1)+(1-t)f(x_2),$

while the graph of the function is the convex function of the weighted means,

$f(t{x}_1+(1-t)x_2).$

Thus, Jensen's inequality in this case is

$f(t{x}_1+(1-t)x_2)\le tf(x_1)+(1-t)f(x_2).$

In the context of probability theory, it is generally stated in the following form: if X is a random variable and Template:Mvar is a convex function, then

Failed to parse (Conversion error. Server ("https://wikimedia.org/api/rest_") reported: "Class "Wikibase\Client\WikibaseClient" not found"): {\displaystyle \varphi (\operatorname {E} [X])\leq \operatorname {E} \left[\varphi (X)\right].}

The difference between the two sides of the inequality, $\mathrm{E}[\phi (X)]-\phi (\mathrm{E}[X])$, is called the Jensen gap.

Statements

The classical form of Jensen's inequality involves several numbers and weights. The inequality can be stated quite generally using either the language of measure theory or (equivalently) probability. In the probabilistic setting, the inequality can be further generalized to its full strength.

Finite form

For a real convex function $\phi$, numbers $x_1,x_2,\dots ,x_n$ in its domain, and positive weights $a_i$, Jensen's inequality can be stated as:

and the inequality is reversed if $\phi$ is concave, which is

Equality holds if and only if $x_1=x_2=\cdots =x_n$ or $\phi$ is linear on a domain containing $x_1,x_2,\cdots ,x_n$.

As a particular case, if the weights $a_i$ are all equal, then (Template:EquationNote) and (Template:EquationNote) become

For instance, the function Template:Math is concave, so substituting $\phi (x)=\log (x)$ in the previous formula (Template:EquationNote) establishes the (logarithm of the) familiar arithmetic-mean/geometric-mean inequality:

$$\log \left(\frac{\sum _{i=1}^n{x}_i}{n}\right)\ge \frac{\sum _{i=1}^n\log \left(x_i\right)}{n}$$ $$\exp (\log \left(\frac{\sum _{i=1}^n{x}_i}{n}\right))\ge \exp \left(\frac{\sum _{i=1}^n\log \left(x_i\right)}{n}\right)$$ $$\frac{x_1+x_2+\cdots +x_n}{n}\ge \sqrt[n]{x_1\cdot x_2\cdots x_n}$$

A common application has Template:Mvar as a function of another variable (or set of variables) Template:Mvar, that is, $x_i=g(t_i)$. All of this carries directly over to the general continuous case: the weights Template:Math are replaced by a non-negative integrable function Template:Math, such as a probability distribution, and the summations are replaced by integrals.

Measure-theoretic form

Let $(\mathrm{\Omega },A,\mu )$ be a probability space. Let $f:\mathrm{\Omega }\to ℝ$ be a $\mu$-measurable function and $\phi :ℝ\to ℝ$ be convex. Then: $$\phi \left(\underset{\mathrm{\Omega }}{\int }f\mathrm{d}\mu \right)\le \underset{\mathrm{\Omega }}{\int }\phi \circ f\mathrm{d}\mu$$

In real analysis, we may require an estimate on

$\phi (\underset{a}{\overset{b}{\int }}f(x)dx)$

where $a,b\in ℝ$, and $f:[a,b]\to ℝ$ is a non-negative Lebesgue-integrable function. In this case, the Lebesgue measure of $[a,b]$ need not be unity. However, by integration by substitution, the interval can be rescaled so that it has measure unity. Then Jensen's inequality can be applied to get

$\phi (\frac{1}{b-a}\underset{a}{\overset{b}{\int }}f(x)dx)\le \frac{1}{b-a}\underset{a}{\overset{b}{\int }}\phi (f(x))dx.$

Probabilistic form

The same result can be equivalently stated in a probability theory setting, by a simple change of notation. Let $(\mathrm{\Omega },𝔉,\mathrm{P})$ be a probability space, X an integrable real-valued random variable and $\phi$ a convex function. Then:

$\phi (\mathrm{E}[X])\le \mathrm{E}[\phi (X)].$

In this probability setting, the measure Template:Mvar is intended as a probability $\mathrm{P}$, the integral with respect to Template:Mvar as an expected value $\mathrm{E}$, and the function $f$ as a random variable X.

Note that the equality holds if and only if $\phi$ is a linear function on some convex set $A$ such that $\mathrm{P}(X\in A)=1$ (which follows by inspecting the measure-theoretical proof below).

General inequality in a probabilistic setting

More generally, let T be a real topological vector space, and X a T-valued integrable random variable. In this general setting, integrable means that there exists an element $\mathrm{E}[X]$ in T, such that for any element z in the dual space of T: $\mathrm{E}|⟨z,X⟩|<\mathrm{\infty }$, and $⟨z,\mathrm{E}[X]⟩=\mathrm{E}[⟨z,X⟩]$. Then, for any measurable convex function Template:Mvar and any sub-σ-algebra $𝔊$ of $𝔉$:

$\phi (\mathrm{E}[X\mid 𝔊])\le \mathrm{E}[\phi (X)\mid 𝔊].$

Here $\mathrm{E}[\cdot \mid 𝔊]$ stands for the expectation conditioned to the σ-algebra $𝔊$. This general statement reduces to the previous ones when the topological vector space Template:Mvar is the real axis, and $𝔊$ is the trivial Template:Mvar-algebra Template:Math (where Template:Math is the empty set, and Template:Math is the sample space).

A sharpened and generalized form

Let X be a one-dimensional random variable with mean $\mu$ and variance ${\sigma }^2\ge 0$. Let $\phi (x)$ be a twice differentiable function, and define the function

$h(x)\triangleq \frac{\phi \left(x\right)-\phi \left(\mu \right)}{{(x-\mu )}^2}-\frac{{\phi }^{\prime }\left(\mu \right)}{x-\mu }.$

Then

${\sigma }^2\inf \frac{{\phi }^{″}(x)}{2}\le {\sigma }^2\inf h(x)\le E\left[\phi \left(X\right)\right]-\phi (E[X])\le {\sigma }^2\sup h(x)\le {\sigma }^2\sup \frac{{\phi }^{″}(x)}{2}.$

In particular, when $\phi (x)$ is convex, then ${\phi }^{″}(x)\ge 0$, and the standard form of Jensen's inequality immediately follows for the case where $\phi (x)$ is additionally assumed to be twice differentiable.

Proofs

Intuitive graphical proof

A graphical "proof" of Jensen's inequality for the probabilistic case. The dashed curve along the Template:Mvar axis is the hypothetical distribution of Template:Mvar, while the dashed curve along the Template:Mvar axis is the corresponding distribution of Template:Mvar values. Note that the convex mapping Template:Math increasingly "stretches" the distribution for increasing values of Template:Mvar.

This is a proof without words of Jensen's inequality for Template:Mvar variables. Without loss of generality, the sum of the positive weights is Template:Math. It follows that the weighted point lies in the convex hull of the original points, which lies above the function itself by the definition of convexity. The conclusion follows.

Jensen's inequality can be proved in several ways, and three different proofs corresponding to the different statements above will be offered. Before embarking on these mathematical derivations, however, it is worth analyzing an intuitive graphical argument based on the probabilistic case where Template:Mvar is a real number (see figure). Assuming a hypothetical distribution of Template:Mvar values, one can immediately identify the position of $\mathrm{E}[X]$ and its image $\phi (\mathrm{E}[X])$ in the graph. Noticing that for convex mappings Template:Math of some Template:Mvar values the corresponding distribution of Template:Mvar values is increasingly "stretched up" for increasing values of Template:Mvar, it is easy to see that the distribution of Template:Mvar is broader in the interval corresponding to Template:Math and narrower in Template:Math for any Template:Math; in particular, this is also true for $X_0=\mathrm{E}[X]$. Consequently, in this picture the expectation of Template:Mvar will always shift upwards with respect to the position of $\phi (\mathrm{E}[X])$. A similar reasoning holds if the distribution of Template:Mvar covers a decreasing portion of the convex function, or both a decreasing and an increasing portion of it. This "proves" the inequality, i.e.

$\phi (\mathrm{E}[X])\le \mathrm{E}[\phi (X)]=\mathrm{E}[Y],$

with equality when Template:Math is not strictly convex, e.g. when it is a straight line, or when Template:Mvar follows a degenerate distribution (i.e. is a constant).

The proofs below formalize this intuitive notion.

Proof 1 (finite form)

If Template:Math and Template:Math are two arbitrary nonnegative real numbers such that Template:Math then convexity of Template:Mvar implies

$\mathrm{\forall }{x}_1,x_2:\phi ({\lambda }_1{x}_1+{\lambda }_2{x}_2)\le {\lambda }_1\phi (x_1)+{\lambda }_2\phi (x_2).$

This can be generalized: if Template:Math are nonnegative real numbers such that Template:Math, then

$\phi ({\lambda }_1{x}_1+{\lambda }_2{x}_2+\cdots +{\lambda }_n{x}_n)\le {\lambda }_1\phi (x_1)+{\lambda }_2\phi (x_2)+\cdots +{\lambda }_n\phi (x_n),$

for any Template:Math.

The finite form of the Jensen's inequality can be proved by induction: by convexity hypotheses, the statement is true for n = 2. Suppose the statement is true for some n, so

$\phi \left(\sum _{i=1}^n{\lambda }_i{x}_i\right)\le \sum _{i=1}^n{\lambda }_i\phi \left(x_i\right)$

for any Template:Math such that Template:Math.

One needs to prove it for Template:Math. At least one of the Template:Math is strictly smaller than $1$, say Template:Math; therefore by convexity inequality:

$\begin{array}{cc}\phi \left(\sum _{i=1}^{n+1}{\lambda }_i{x}_i\right)& =\phi ((1-{\lambda }_{n+1})\sum _{i=1}^n\frac{{\lambda }_i}{1-{\lambda }_{n+1}}{x}_i+{\lambda }_{n+1}{x}_{n+1})\\ & \le (1-{\lambda }_{n+1})\phi \left(\sum _{i=1}^n\frac{{\lambda }_i}{1-{\lambda }_{n+1}}{x}_i\right)+{\lambda }_{n+1}\phi (x_{n+1}).\end{array}$

Since Template:Math,

$\sum _{i=1}^n\frac{{\lambda }_i}{1-{\lambda }_{n+1}}=1$,

applying the inductive hypothesis gives

$\phi \left(\sum _{i=1}^n\frac{{\lambda }_i}{1-{\lambda }_{n+1}}{x}_i\right)\le \sum _{i=1}^n\frac{{\lambda }_i}{1-{\lambda }_{n+1}}\phi (x_i)$

therefore

$\begin{array}{cc}\phi \left(\sum _{i=1}^{n+1}{\lambda }_i{x}_i\right)& \le (1-{\lambda }_{n+1})\sum _{i=1}^n\frac{{\lambda }_i}{1-{\lambda }_{n+1}}\phi (x_i)+{\lambda }_{n+1}\phi (x_{n+1})=\sum _{i=1}^{n+1}{\lambda }_i\phi (x_i)\end{array}$

We deduce the inequality is true for Template:Math, by induction it follows that the result is also true for all integer Template:Math greater than 2.

In order to obtain the general inequality from this finite form, one needs to use a density argument. The finite form can be rewritten as:

$\phi (\int xd{\mu }_n(x))\le \int \phi (x)d{\mu }_n(x),$

where μ_n is a measure given by an arbitrary convex combination of Dirac deltas:

${\mu }_n=\sum _{i=1}^n{\lambda }_i{\delta }_{x_i}.$

Since convex functions are continuous, and since convex combinations of Dirac deltas are weakly dense in the set of probability measures (as could be easily verified), the general statement is obtained simply by a limiting procedure.

Proof 2 (measure-theoretic form)

Let $g$ be a real-valued $\mu$-integrable function on a probability space $\mathrm{\Omega }$, and let $\phi$ be a convex function on the real numbers. Since $\phi$ is convex, at each real number $x$ we have a nonempty set of subderivatives, which may be thought of as lines touching the graph of $\phi$ at $x$, but which are below the graph of $\phi$ at all points (support lines of the graph).

Now, if we define

$x_0:=\underset{\mathrm{\Omega }}{\int }gd\mu ,$

because of the existence of subderivatives for convex functions, we may choose $a$ and $b$ such that

$ax+b\le \phi (x),$

for all real $x$ and

$a{x}_0+b=\phi (x_0).$

But then we have that

$\phi \circ g(\omega )\ge ag(\omega )+b$

for almost all $\omega \in \mathrm{\Omega }$. Since we have a probability measure, the integral is monotone with $\mu (\mathrm{\Omega })=1$ so that

$\underset{\mathrm{\Omega }}{\int }\phi \circ gd\mu \ge \underset{\mathrm{\Omega }}{\int }(ag+b)d\mu =a\underset{\mathrm{\Omega }}{\int }gd\mu +b\underset{\mathrm{\Omega }}{\int }d\mu =a{x}_0+b=\phi (x_0)=\phi \left(\underset{\mathrm{\Omega }}{\int }gd\mu \right),$

as desired.

Proof 3 (general inequality in a probabilistic setting)

Let X be an integrable random variable that takes values in a real topological vector space T. Since $\phi :T\to ℝ$ is convex, for any $x,y\in T$, the quantity

$\frac{\phi (x+\theta y)-\phi (x)}{\theta },$

is decreasing as Template:Mvar approaches 0⁺. In particular, the subdifferential of $\phi$ evaluated at Template:Mvar in the direction Template:Mvar is well-defined by

$(D\phi )(x)\cdot y:=\underset{\theta \downarrow 0}{\lim }\frac{\phi (x+\theta y)-\phi (x)}{\theta }=\underset{\theta \ne 0}{\inf }\frac{\phi (x+\theta y)-\phi (x)}{\theta }.$

It is easily seen that the subdifferential is linear in Template:Mvar Template:Citation needed (that is false and the assertion requires Hahn-Banach theorem to be proved) and, since the infimum taken in the right-hand side of the previous formula is smaller than the value of the same term for Template:Math, one gets

$\phi (x)\le \phi (x+y)-(D\phi )(x)\cdot y.$

In particular, for an arbitrary sub-Template:Mvar-algebra $𝔊$ we can evaluate the last inequality when $x=\mathrm{E}[X\mid 𝔊],y=X-\mathrm{E}[X\mid 𝔊]$ to obtain

$\phi (\mathrm{E}[X\mid 𝔊])\le \phi (X)-(D\phi )(\mathrm{E}[X\mid 𝔊])\cdot (X-\mathrm{E}[X\mid 𝔊]).$

Now, if we take the expectation conditioned to $𝔊$ on both sides of the previous expression, we get the result since:

$\mathrm{E}[[(D\phi )(\mathrm{E}[X\mid 𝔊])\cdot (X-\mathrm{E}[X\mid 𝔊])]\mid 𝔊]=(D\phi )(\mathrm{E}[X\mid 𝔊])\cdot \mathrm{E}[(X-\mathrm{E}[X\mid 𝔊])\mid 𝔊]=0,$

by the linearity of the subdifferential in the y variable, and the following well-known property of the conditional expectation:

$\mathrm{E}[(\mathrm{E}[X\mid 𝔊])\mid 𝔊]=\mathrm{E}[X\mid 𝔊].$

Applications and special cases

Form involving a probability density function

Suppose Template:Math is a measurable subset of the real line and f(x) is a non-negative function such that

$\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}f(x)dx=1.$

In probabilistic language, f is a probability density function.

Then Jensen's inequality becomes the following statement about convex integrals:

If g is any real-valued measurable function and $\phi$ is convex over the range of g, then

$\phi (\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}g(x)f(x)dx)\le \underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\phi (g(x))f(x)dx.$

If g(x) = x, then this form of the inequality reduces to a commonly used special case:

$\phi (\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}xf(x)dx)\le \underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\phi (x)f(x)dx.$

This is applied in Variational Bayesian methods.

Example: even moments of a random variable

If g(x) = x²ⁿ, and X is a random variable, then g is convex as

$\frac{d^{2}g}{d{x}^2}(x)=2n(2n-1)x^{2n-2}\ge 0\mathrm{\forall }x\in ℝ$

and so

$g(\mathrm{E}[X])=(\mathrm{E}[X]{)}^{2n}\le \mathrm{E}[X^{2n}].$

In particular, if some even moment 2n of X is finite, X has a finite mean. An extension of this argument shows X has finite moments of every order $l\in \mathbb{N}$ dividing n.

Alternative finite form

Let Template:Math and take Template:Mvar to be the counting measure on Template:Math, then the general form reduces to a statement about sums:

$\phi (\sum _{i=1}^{n}g(x_i){\lambda }_i)\le \sum _{i=1}^n\phi (g(x_i)){\lambda }_i,$

provided that Template:Math and

${\lambda }_1+\cdots +{\lambda }_n=1.$

There is also an infinite discrete form.

Statistical physics

Jensen's inequality is of particular importance in statistical physics when the convex function is an exponential, giving:

$e^{\mathrm{E}[X]}\le \mathrm{E}\left[e^X\right],$

where the expected values are with respect to some probability distribution in the random variable Template:Mvar.

Proof: Let $\phi (x)=e^x$ in $\phi (\mathrm{E}[X])\le \mathrm{E}[\phi (X)].$

Information theory

If Template:Math is the true probability density for Template:Mvar, and Template:Math is another density, then applying Jensen's inequality for the random variable Template:Math and the convex function Template:Math gives

$\mathrm{E}[\phi (Y)]\ge \phi (\mathrm{E}[Y])$

Therefore:

$-D(p(x)\Vert q(x))=\int p(x)\log \left(\frac{q(x)}{p(x)}\right)dx\le \log (\int p(x)\frac{q(x)}{p(x)}dx)=\log (\int q(x)dx)=0$

a result called Gibbs' inequality.

It shows that the average message length is minimised when codes are assigned on the basis of the true probabilities p rather than any other distribution q. The quantity that is non-negative is called the Kullback–Leibler divergence of q from p, where $D(p(x)\Vert q(x))=\int p(x)\log \left(\frac{p(x)}{q(x)}\right)dx$.

Since Template:Math is a strictly convex function for Template:Math, it follows that equality holds when Template:Math equals Template:Math almost everywhere.

Rao–Blackwell theorem

If L is a convex function and $𝔊$ a sub-sigma-algebra, then, from the conditional version of Jensen's inequality, we get

$L(\mathrm{E}[\delta (X)\mid 𝔊])\le \mathrm{E}[L(\delta (X))\mid 𝔊]⟹\mathrm{E}[L(\mathrm{E}[\delta (X)\mid 𝔊])]\le \mathrm{E}[L(\delta (X))].$

So if δ(X) is some estimator of an unobserved parameter θ given a vector of observables X; and if T(X) is a sufficient statistic for θ; then an improved estimator, in the sense of having a smaller expected loss L, can be obtained by calculating

${\delta }_1(X)={\mathrm{E}}_{\theta }[\delta (X^{\prime })\mid T(X^{\prime })=T(X)],$

the expected value of δ with respect to θ, taken over all possible vectors of observations X compatible with the same value of T(X) as that observed. Further, because T is a sufficient statistic, ${\delta }_1(X)$ does not depend on θ, hence, becomes a statistic.

This result is known as the Rao–Blackwell theorem.

Risk aversion

The relation between risk aversion and declining marginal utility for scalar outcomes can be stated formally with Jensen's inequality: risk aversion can be stated as preferring a certain outcome $u(E[x])$ to a fair gamble with potentially larger but uncertain outcome of $u(x)$:

$u(E[x])>E[u(x)]$.

But this is simply Jensen's inequality for a concave $u(x)$: a utility function that exhibits declining marginal utility.

Generalizations

Beyond its classical formulation for real numbers and convex functions, Jensen’s inequality has been extended to the realm of operator theory. In this non‐commutative setting the inequality is expressed in terms of operator convex functions—that is, functions defined on an interval I that satisfy

$f(\lambda x+(1-\lambda )y)\le \lambda f(x)+(1-\lambda )f(y)$

for every pair of self‐adjoint operators x and y (with spectra in I) and every scalar $\lambda \in [0,1]$. Hansen and Pedersen established a definitive version of this inequality by considering genuine non‐commutative convex combinations. In particular, if one has an n‑tuple of bounded self‐adjoint operators $x_1,\dots ,x_n$ with spectra in I and an n‑tuple of operators $a_1,\dots ,a_n$ satisfying

$\sum _{i=1}^n{a}_i^{*}{a}_i=I,$

then the following operator Jensen inequality holds:

$f\left(\sum _{i=1}^n{a}_i^{*}{x}_i{a}_i\right)\le \sum _{i=1}^n{a}_i^{*}f(x_i)a_i.$

This result shows that the convex transformation “respects” non-commutative convex combinations, thereby extending the classical inequality to operators without the need for additional restrictions on the interval of definition. A closely related extension is given by the Jensen trace inequality. For a continuous convex function f defined on I, if one considers self‐adjoint matrices $x_1,\dots ,x_n$ (with spectra in I) and matrices $a_1,\dots ,a_n$ satisfying $\sum _{i=1}^n{a}_i^{*}{a}_i=I$, then one has

$\mathrm{Tr}\left(f\right(\sum _{i=1}^n{a}_i^{*}{x}_i{a}_i\left)\right)\le \mathrm{Tr}(\sum _{i=1}^n{a}_i^{*}f(x_i)a_i).$

This inequality naturally extends to C*-algebras equipped with a finite trace and is particularly useful in applications ranging from quantum statistical mechanics to information theory. Furthermore, contractive versions of these operator inequalities are available when one only assumes $\sum _{i=1}^n{a}_i^t{a}_i\le I$, provided that additional conditions such as $f(0)\le 0$ (when 0 ∈ I) are imposed. Extensions to continuous fields of operators and to settings involving conditional expectations on C-algebras further illustrate the broad applicability of these generalizations.

See also

• Karamata's inequality for a more general inequality
• Popoviciu's inequality
• Law of averages
• A proof without words of Jensen's inequality

References

• Template:Cite book
• Tristan Needham (1993) "A Visual Explanation of Jensen's Inequality", American Mathematical Monthly 100(8):768–71.
• Template:Cite book
• Template:Cite book
• Template:Cite book
• Sam Savage (2012) The Flaw of Averages: Why We Underestimate Risk in the Face of Uncertainty (1st ed.) Wiley. ISBN 978-0471381976

External links

• Jensen's Operator Inequality of Hansen and Pedersen.
• Template:Springer
• Template:MathWorld
• Template:Cite web