Taylor expansions for the moments of functions of random variables

Template:Multiple issues In probability theory, it is possible to approximate the moments of a function f of a random variable X using Taylor expansions, provided that f is sufficiently differentiable and that the moments of X are finite.

A simulation-based alternative to this approximation is the application of Monte Carlo simulations.

First moment

Given ${\mu }_X$ and ${\sigma }_X^2$, the mean and the variance of $X$, respectively, a Taylor expansion of the expected value of $f(X)$ can be found via

$\begin{array}{cc}\mathrm{E}[f(X)]& =\mathrm{E}\left[f({\mu }_X+(X-{\mu }_X))\right]\\ & \approx \mathrm{E}[f({\mu }_X)+f^{\prime }({\mu }_X)(X-{\mu }_X)+\frac{1}{2}{f}^{″}({\mu }_X){(X-{\mu }_X)}^2]\\ & =f({\mu }_X)+f^{\prime }({\mu }_X)\mathrm{E}[X-{\mu }_X]+\frac{1}{2}{f}^{″}({\mu }_X)\mathrm{E}\left[{(X-{\mu }_X)}^2\right].\end{array}$

Since $E[X-{\mu }_X]=0,$ the second term vanishes. Also, $E[(X-{\mu }_X{)}^2]$ is ${\sigma }_X^2$. Therefore,

$\mathrm{E}[f(X)]\approx f({\mu }_X)+\frac{f^{″}({\mu }_X)}{2}{\sigma }_X^2$.

It is possible to generalize this to functions of more than one variable using multivariate Taylor expansions. For example,

$\mathrm{E}\left[\frac{X}{Y}\right]\approx \frac{\mathrm{E}\left[X\right]}{\mathrm{E}\left[Y\right]}-\frac{\mathrm{cov}[X,Y]}{\mathrm{E}{\left[Y\right]}^2}+\frac{\mathrm{E}\left[X\right]}{\mathrm{E}{\left[Y\right]}^3}\mathrm{var}\left[Y\right]$

Second moment

Similarly,

$\mathrm{var}[f(X)]\approx {(f^{\prime }(\mathrm{E}\left[X\right]))}^2\mathrm{var}\left[X\right]={(f^{\prime }({\mu }_X))}^2{\sigma }_X^2-\frac{1}{4}{(f^{″}({\mu }_X))}^2{\sigma }_X^4$

The above is obtained using a second order approximation, following the method used in estimating the first moment. It will be a poor approximation in cases where $f(X)$ is highly non-linear. This is a special case of the delta method.

Indeed, we take $\mathrm{E}[f(X)]\approx f({\mu }_X)+\frac{f^{″}({\mu }_X)}{2}{\sigma }_X^2$.

With $f(X)=g(X{)}^2$, we get $\mathrm{E}\left[Y^2\right]$. The variance is then computed using the formula $\mathrm{var}\left[Y\right]=\mathrm{E}\left[Y^2\right]-{\mu }_Y^2$.

An example is,

$\mathrm{var}\left[\frac{X}{Y}\right]\approx \frac{\mathrm{var}\left[X\right]}{\mathrm{E}{\left[Y\right]}^2}-\frac{2\mathrm{E}\left[X\right]}{\mathrm{E}{\left[Y\right]}^3}\mathrm{cov}[X,Y]+\frac{\mathrm{E}{\left[X\right]}^2}{\mathrm{E}{\left[Y\right]}^4}\mathrm{var}\left[Y\right].$

The second order approximation, when X follows a normal distribution, is:

$\mathrm{var}[f(X)]\approx {(f^{\prime }(\mathrm{E}\left[X\right]))}^2\mathrm{var}\left[X\right]+\frac{{(f^{″}(\mathrm{E}\left[X\right]))}^2}{2}{(\mathrm{var}\left[X\right])}^2={(f^{\prime }({\mu }_X))}^2{\sigma }_X^2+\frac{1}{2}{(f^{″}({\mu }_X))}^2{\sigma }_X^4+(f^{\prime }({\mu }_X))(f^{‴}({\mu }_X)){\sigma }_X^4$

First product moment

To find a second-order approximation for the covariance of functions of two random variables (with the same function applied to both), one can proceed as follows. First, note that $\mathrm{cov}[f(X),f(Y)]=\mathrm{E}[f(X)f(Y)]-\mathrm{E}[f(X)]\mathrm{E}[f(Y)]$. Since a second-order expansion for $\mathrm{E}[f(X)]$ has already been derived above, it only remains to find $\mathrm{E}[f(X)f(Y)]$. Treating $f(X)f(Y)$ as a two-variable function, the second-order Taylor expansion is as follows:

$\begin{array}{cc}f(X)f(Y)& \approx f({\mu }_X)f({\mu }_Y)+(X-{\mu }_X)f^{\prime }({\mu }_X)f({\mu }_Y)+(Y-{\mu }_Y)f({\mu }_X)f^{\prime }({\mu }_Y)+\frac{1}{2}[(X-{\mu }_X{)}^2{f}^{″}({\mu }_X)f({\mu }_Y)+2(X-{\mu }_X)(Y-{\mu }_Y)f^{\prime }({\mu }_X)f^{\prime }({\mu }_Y)+(Y-{\mu }_Y{)}^{2}f({\mu }_X)f^{″}({\mu }_Y)]\end{array}$

Taking expectation of the above and simplifying—making use of the identities $\mathrm{E}(X^2)=\mathrm{var}(X)+{[\mathrm{E}(X)]}^2$ and $\mathrm{E}(XY)=\mathrm{cov}(X,Y)+[\mathrm{E}(X)][\mathrm{E}(Y)]$—leads to $\mathrm{E}[f(X)f(Y)]\approx f({\mu }_X)f({\mu }_Y)+f^{\prime }({\mu }_X)f^{\prime }({\mu }_Y)\mathrm{cov}(X,Y)+\frac{1}{2}{f}^{″}({\mu }_X)f({\mu }_Y)\mathrm{var}(X)+\frac{1}{2}f({\mu }_X)f^{″}({\mu }_Y)\mathrm{var}(Y)$. Hence,

$\begin{array}{cc}\mathrm{cov}[f(X),f(Y)]& \approx f({\mu }_X)f({\mu }_Y)+f^{\prime }({\mu }_X)f^{\prime }({\mu }_Y)\mathrm{cov}(X,Y)+\frac{1}{2}{f}^{″}({\mu }_X)f({\mu }_Y)\mathrm{var}(X)+\frac{1}{2}f({\mu }_X)f^{″}({\mu }_Y)\mathrm{var}(Y)-[f({\mu }_X)+\frac{1}{2}{f}^{″}({\mu }_X)\mathrm{var}(X)][f({\mu }_Y)+\frac{1}{2}{f}^{″}({\mu }_Y)\mathrm{var}(Y)]\\ & =f^{\prime }({\mu }_X)f^{\prime }({\mu }_Y)\mathrm{cov}(X,Y)-\frac{1}{4}{f}^{″}({\mu }_X)f^{″}({\mu }_Y)\mathrm{var}(X)\mathrm{var}(Y)\end{array}$

Random vectors

If X is a random vector, the approximations for the mean and variance of $f(X)$ are given by

$\begin{array}{cc}\mathrm{E}(f(X))& =f({\mu }_X)+\frac{1}{2}\mathrm{trace}(H_f({\mu }_X){\mathrm{\Sigma }}_X)\\ \mathrm{var}(f(X))& =\mathrm{\nabla }f({\mu }_X{)}^t{\mathrm{\Sigma }}_X\mathrm{\nabla }f({\mu }_X)+\frac{1}{2}\mathrm{trace}(H_f({\mu }_X){\mathrm{\Sigma }}_X{H}_f({\mu }_X){\mathrm{\Sigma }}_X).\end{array}$

Here $\mathrm{\nabla }f$ and $H_f$ denote the gradient and the Hessian matrix respectively, and ${\mathrm{\Sigma }}_X$ is the covariance matrix of X.

See also

• Propagation of uncertainty
• WKB approximation
• Delta method

Further reading

• Template:Cite book