Wishart distribution

In statistics, the Wishart distribution is a generalization of the gamma distribution to multiple dimensions. It is named in honor of John Wishart, who first formulated the distribution in 1928. Other names include Wishart ensemble (in random matrix theory, probability distributions over matrices are usually called "ensembles"), or Wishart–Laguerre ensemble (since its eigenvalue distribution involves Laguerre polynomials), or LOE, LUE, LSE (in analogy with GOE, GUE, GSE).

It is a family of probability distributions defined over symmetric, positive-definite random matrices (i.e. matrix-valued random variables). These distributions are of great importance in the estimation of covariance matrices in multivariate statistics. In Bayesian statistics, the Wishart distribution is the conjugate prior of the inverse covariance-matrix of a multivariate-normal random vector.

Definition

Suppose Template:Mvar is a Template:Math matrix, each column of which is independently drawn from a [[multivariate normal distribution|Template:Mvar-variate normal distribution]] with zero mean:

$G=(g_1,\dots ,g_n)\sim {𝒩}_p(0,V).$

It means $g_i=(g_{i,1},\dots ,g_{i,p}{)}^T\stackrel{iid}{\sim }{𝒩}_p(0,V)\mathrm{\forall }i\in \{1,\dots ,n\}$

Then the Wishart distribution is the probability distribution of the Template:Math random matrix

$S=G{G}^T=\sum _{i=1}^n{g}_i{g}_i^T$

known as the scatter matrix. One indicates that Template:Mvar has that probability distribution by writing

$S\sim W_p(V,n).$

The positive integer Template:Mvar is the number of degrees of freedom. Sometimes this is written Template:Math. For Template:Math the matrix Template:Mvar is invertible with probability Template:Math if Template:Mvar is invertible.

If Template:Math then this distribution is a chi-squared distribution with Template:Mvar degrees of freedom.

Occurrence

The Wishart distribution arises as the distribution of the sample covariance matrix for a sample from a multivariate normal distribution. It occurs frequently in likelihood-ratio tests in multivariate statistical analysis. It also arises in the spectral theory of random matricesTemplate:Citation needed and in multidimensional Bayesian analysis. It is also encountered in wireless communications, while analyzing the performance of Rayleigh fading MIMO wireless channels.

Probability density function

Spectral density of Wishart-Laguerre ensemble with dimensions (8, 15). A reconstruction of Figure 1 of.

The Wishart distribution can be characterized by its probability density function as follows:

Let Template:Math be a Template:Math symmetric matrix of random variables that is positive semi-definite. Let Template:Math be a (fixed) symmetric positive definite matrix of size Template:Math.

Then, if Template:Math, Template:Math has a Wishart distribution with Template:Mvar degrees of freedom if it has the probability density function

$f_{𝐗}(𝐗)=\frac{1}{2^{np/2}{\left|𝐕\right|}^{n/2}{\mathrm{\Gamma }}_p\left(\frac{n}{2}\right)}{\left|𝐗\right|}^{(n-p-1)/2}{e}^{-\frac{1}{2}\mathrm{tr}({𝐕}^{-1}𝐗)}$

where $\left|𝐗\right|$ is the determinant of $𝐗$ and Template:Math is the multivariate gamma function defined as

${\mathrm{\Gamma }}_p\left(\frac{n}{2}\right)={\pi }^{p(p-1)/4}\prod _{j=1}^p\mathrm{\Gamma }(\frac{n}{2}-\frac{j-1}{2}).$

The density above is not the joint density of all the $p^2$ elements of the random matrix Template:Math (such Template:Nowrap density does not exist because of the symmetry constrains $X_{ij}=X_{ji}$), it is rather the joint density of $p(p+1)/2$ elements $X_{ij}$ for $i\le j$ (, page 38). Also, the density formula above applies only to positive definite matrices $𝐱;$ for other matrices the density is equal to zero.

In fact the above definition can be extended to any real Template:Math. If Template:Math, then the Wishart no longer has a density—instead it represents a singular distribution that takes values in a lower-dimension subspace of the space of Template:Math matrices.

Spectral density

The joint-eigenvalue density for the eigenvalues ${\lambda }_1,\dots ,{\lambda }_p\ge 0$ of a random matrix $𝐗\sim W_p(𝐈,n)$ is,

$c_{n,p}{e}^{-\frac{1}{2}\sum _i{\lambda }_i}\prod {\lambda }_i^{(n-p-1)/2}\prod _{i<j}|{\lambda }_i-{\lambda }_j|$

where $c_{n,p}$ is a constant. The spectral density can be marginalized to yield the density of a single eigenvalue, by evaluating a Selberg integral.

The spectral density can be integrated to give the probability that all eigenvalues of a Wishart random matrix lie within an interval.

Use in Bayesian statistics

In Bayesian statistics, in the context of the multivariate normal distribution, the Wishart distribution is the conjugate prior to the precision matrix Template:Math, where Template:Math is the covariance matrix.Template:Rp The use of Normal-Wishart conjugate priors (for mean and precision) is particularly common for vector autoregression models.

Choice of parameters

The least informative, proper Wishart prior is obtained by setting Template:Math.Template:Citation needed

A common choice for V leverages the fact that the mean of X ~W_p(V, n) is nV. Then V is chosen so that nV equals an initial guess for X. For instance, when estimating a precision matrix Σ⁻¹ ~ W_p(V, n) a reasonable choice for V would be n⁻¹Σ₀⁻¹, where Σ₀ is some prior estimate for the covariance matrix Σ.

Properties

Log-expectation

The following formula plays a role in variational Bayes derivations for Bayes networks involving the Wishart distribution. From equation (2.63),

$\mathrm{E}[\ln \left|𝐗\right|]={\psi }_p\left(\frac{n}{2}\right)+p\ln (2)+\ln |𝐕|$

where ${\psi }_p$ is the multivariate digamma function (the derivative of the log of the multivariate gamma function).

Log-variance

The following variance computation could be of help in Bayesian statistics:

$\mathrm{Var}[\ln \left|𝐗\right|]=\sum _{i=1}^p{\psi }_1\left(\frac{n+1-i}{2}\right)$

where ${\psi }_1$ is the trigamma function. This comes up when computing the Fisher information of the Wishart random variable.

Entropy

The information entropy of the distribution has the following formula:Template:Rp

$\mathrm{H}\left[𝐗\right]=-\ln (B(𝐕,n))-\frac{n-p-1}{2}\mathrm{E}[\ln \left|𝐗\right|]+\frac{np}{2}$

where Template:Math is the normalizing constant of the distribution:

$B(𝐕,n)=\frac{1}{{\left|𝐕\right|}^{n/2}{2}^{np/2}{\mathrm{\Gamma }}_p\left(\frac{n}{2}\right)}.$

This can be expanded as follows:

$\begin{array}{cc}\mathrm{H}\left[𝐗\right]& =\frac{n}{2}\ln \left|𝐕\right|+\frac{np}{2}\ln 2+\ln {\mathrm{\Gamma }}_p\left(\frac{n}{2}\right)-\frac{n-p-1}{2}\mathrm{E}[\ln \left|𝐗\right|]+\frac{np}{2}\\ & =\frac{n}{2}\ln \left|𝐕\right|+\frac{np}{2}\ln 2+\ln {\mathrm{\Gamma }}_p\left(\frac{n}{2}\right)-\frac{n-p-1}{2}({\psi }_p\left(\frac{n}{2}\right)+p\ln 2+\ln \left|𝐕\right|)+\frac{np}{2}\\ & =\frac{n}{2}\ln \left|𝐕\right|+\frac{np}{2}\ln 2+\ln {\mathrm{\Gamma }}_p\left(\frac{n}{2}\right)-\frac{n-p-1}{2}{\psi }_p\left(\frac{n}{2}\right)-\frac{n-p-1}{2}(p\ln 2+\ln \left|𝐕\right|)+\frac{np}{2}\\ & =\frac{p+1}{2}\ln \left|𝐕\right|+\frac{1}{2}p(p+1)\ln 2+\ln {\mathrm{\Gamma }}_p\left(\frac{n}{2}\right)-\frac{n-p-1}{2}{\psi }_p\left(\frac{n}{2}\right)+\frac{np}{2}\end{array}$

Cross-entropy

The cross-entropy of two Wishart distributions $p_0$ with parameters $n_0,V_0$ and $p_1$ with parameters $n_1,V_1$ is

$\begin{array}{cc}H(p_0,p_1)& ={\mathrm{E}}_{p_0}[-\log p_1]\\ & ={\mathrm{E}}_{p_0}[-\log \frac{{\left|𝐗\right|}^{(n_1-p_1-1)/2}{e}^{-\mathrm{tr}({𝐕}_1^{-1}𝐗)/2}}{2^{n_1{p}_1/2}{\left|{𝐕}_1\right|}^{n_1/2}{\mathrm{\Gamma }}_{p_1}\left(\frac{n_1}{2}\right)}]\\ & =\frac{n_1{p}_1}{2}\log 2+\frac{n_1}{2}\log \left|{𝐕}_1\right|+\log {\mathrm{\Gamma }}_{p_1}(\frac{n_1}{2})-\frac{n_1-p_1-1}{2}{\mathrm{E}}_{p_0}[\log \left|𝐗\right|]+\frac{1}{2}{\mathrm{E}}_{p_0}[\mathrm{tr}\left({𝐕}_1^{-1}𝐗\right)]\\ & =\frac{n_1{p}_1}{2}\log 2+\frac{n_1}{2}\log \left|{𝐕}_1\right|+\log {\mathrm{\Gamma }}_{p_1}(\frac{n_1}{2})-\frac{n_1-p_1-1}{2}({\psi }_{p_0}(\frac{n_0}{2})+p_0\log 2+\log \left|{𝐕}_0\right|)+\frac{1}{2}\mathrm{tr}\left({𝐕}_1^{-1}{n}_0{𝐕}_0\right)\\ & =-\frac{n_1}{2}\log \left|{𝐕}_1^{-1}{𝐕}_0\right|+\frac{p_1+1}{2}\log \left|{𝐕}_0\right|+\frac{n_0}{2}\mathrm{tr}\left({𝐕}_1^{-1}{𝐕}_0\right)+\log {\mathrm{\Gamma }}_{p_1}\left(\frac{n_1}{2}\right)-\frac{n_1-p_1-1}{2}{\psi }_{p_0}(\frac{n_0}{2})+\frac{n_1(p_1-p_0)+p_0(p_1+1)}{2}\log 2\end{array}$

Note that when $p_0=p_1$ and $n_0=n_1$ we recover the entropy.

KL-divergence

The Kullback–Leibler divergence of $p_1$ from $p_0$ is

$\begin{array}{cc}{D}_{KL}(p_0\Vert p_1)& =H(p_0,p_1)-H(p_0)\\ & =-\frac{n_1}{2}\log |{𝐕}_1^{-1}{𝐕}_0|+\frac{n_0}{2}(\mathrm{tr}({𝐕}_1^{-1}{𝐕}_0)-p)+\log \frac{{\mathrm{\Gamma }}_p\left(\frac{n_1}{2}\right)}{{\mathrm{\Gamma }}_p\left(\frac{n_0}{2}\right)}+\frac{n_0-n_1}{2}{\psi }_p\left(\frac{n_0}{2}\right)\end{array}$

Characteristic function

The characteristic function of the Wishart distribution is

$\mathrm{\Theta }\mapsto \mathrm{E}[\exp (i\mathrm{tr}\left(𝐗𝜣\right))]={|1-2i𝜣𝐕|}^{-n/2}$

where Template:Math denotes expectation. (Here Template:Math is any matrix with the same dimensions as Template:Math, Template:Math indicates the identity matrix, and Template:Mvar is a square root of Template:Math). Properly interpreting this formula requires a little care, because noninteger complex powers are multivalued; when Template:Mvar is noninteger, the correct branch must be determined via analytic continuation.

Theorem

If a Template:Math random matrix Template:Math has a Wishart distribution with Template:Mvar degrees of freedom and variance matrix Template:Math — write $𝐗\sim {𝒲}_p(𝐕,m)$ — and Template:Math is a Template:Math matrix of rank Template:Mvar, then

$𝐂𝐗{𝐂}^T\sim {𝒲}_q(𝐂𝐕{𝐂}^T,m).$

Corollary 1

If Template:Math is a nonzero Template:Math constant vector, then:

${\sigma }_z^{-2}{𝐳}^T𝐗𝐳\sim {\chi }_m^2.$

In this case, ${\chi }_m^2$ is the chi-squared distribution and ${\sigma }_z^2={𝐳}^T𝐕𝐳$ (note that ${\sigma }_z^2$ is a constant; it is positive because Template:Math is positive definite).

Corollary 2

Consider the case where Template:Math (that is, the Template:Mvar-th element is one and all others zero). Then corollary 1 above shows that

${\sigma }_{jj}^{-1}{w}_{jj}\sim {\chi }_m^2$

gives the marginal distribution of each of the elements on the matrix's diagonal.

George Seber points out that the Wishart distribution is not called the “multivariate chi-squared distribution” because the marginal distribution of the off-diagonal elements is not chi-squared. Seber prefers to reserve the term multivariate for the case when all univariate marginals belong to the same family.

Estimator of the multivariate normal distribution

The Wishart distribution is the sampling distribution of the maximum-likelihood estimator (MLE) of the covariance matrix of a multivariate normal distribution. A derivation of the MLE uses the spectral theorem.

Bartlett decomposition

The Bartlett decomposition of a matrix Template:Math from a Template:Mvar-variate Wishart distribution with scale matrix Template:Math and Template:Mvar degrees of freedom is the factorization:

$𝐗=\mathbf{\text{𝐋}}\mathbf{\text{𝐀}}{\mathbf{\text{𝐀}}}^T{\mathbf{\text{𝐋}}}^T,$

where Template:Math is the Cholesky factor of Template:Math, and:

$𝐀=\left(\begin{array}{ccccc}{c}_1& 0& 0& \cdots & 0\\ n_{21}& c_2& 0& \cdots & 0\\ n_{31}& n_{32}& c_3& \cdots & 0\\ ⋮& ⋮& ⋮& \ddots & ⋮\\ n_{p1}& n_{p2}& n_{p3}& \cdots & c_p\end{array}\right)$

where $c_i^2\sim {\chi }_{n-i+1}^2$ and Template:Math independently. This provides a useful method for obtaining random samples from a Wishart distribution.

Marginal distribution of matrix elements

Let Template:Math be a Template:Math variance matrix characterized by correlation coefficient Template:Math and Template:Math its lower Cholesky factor:

$𝐕=\left(\begin{array}{cc}{\sigma }_1^2& \rho {\sigma }_1{\sigma }_2\\ \rho {\sigma }_1{\sigma }_2& {\sigma }_2^2\end{array}\right),𝐋=\left(\begin{array}{cc}{\sigma }_1& 0\\ \rho {\sigma }_2& \sqrt{1-{\rho }^2}{\sigma }_2\end{array}\right)$

Multiplying through the Bartlett decomposition above, we find that a random sample from the Template:Math Wishart distribution is

$𝐗=\left(\begin{array}{cc}{\sigma }_1^2{c}_1^2& {\sigma }_1{\sigma }_2(\rho c_1^2+\sqrt{1-{\rho }^2}{c}_1{n}_{21})\\ {\sigma }_1{\sigma }_2(\rho c_1^2+\sqrt{1-{\rho }^2}{c}_1{n}_{21})& {\sigma }_2^2((1-{\rho }^2)c_2^2+{(\sqrt{1-{\rho }^2}{n}_{21}+\rho c_1)}^2)\end{array}\right)$

The diagonal elements, most evidently in the first element, follow the Template:Math distribution with Template:Mvar degrees of freedom (scaled by Template:Math) as expected. The off-diagonal element is less familiar but can be identified as a normal variance-mean mixture where the mixing density is a Template:Math distribution. The corresponding marginal probability density for the off-diagonal element is therefore the variance-gamma distribution

$f(x_{12})=\frac{{\left|x_{12}\right|}^{\frac{n-1}{2}}}{\mathrm{\Gamma }\left(\frac{n}{2}\right)\sqrt{2^{n-1}\pi (1-{\rho }^2){\left({\sigma }_1{\sigma }_2\right)}^{n+1}}}\cdot K_{\frac{n-1}{2}}\left(\frac{\left|x_{12}\right|}{{\sigma }_1{\sigma }_2(1-{\rho }^2)}\right)\exp \left(\frac{\rho x_{12}}{{\sigma }_1{\sigma }_2(1-{\rho }^2)}\right)$

where Template:Math is the modified Bessel function of the second kind. Similar results may be found for higher dimensions. In general, if $X$ follows a Wishart distribution with parameters, $\mathrm{\Sigma },n$, then for $i\ne j$, the off-diagonal elements

$X_{ij}\sim \text{VG}(n,{\mathrm{\Sigma }}_{ij},({\mathrm{\Sigma }}_{ii}{\mathrm{\Sigma }}_{jj}-{\mathrm{\Sigma }}_{ij}^2{)}^{1/2},0)$.

It is also possible to write down the moment-generating function even in the noncentral case (essentially the nth power of Craig (1936) equation 10) although the probability density becomes an infinite sum of Bessel functions.

The range of the shape parameter

It can be shown that the Wishart distribution can be defined if and only if the shape parameter Template:Math belongs to the set

${\mathrm{\Lambda }}_p:=\{0,\dots ,p-1\}\cup (p-1,\mathrm{\infty }).$

This set is named after Simon Gindikin, who introduced it in the 1970s in the context of gamma distributions on homogeneous cones. However, for the new parameters in the discrete spectrum of the Gindikin ensemble, namely,

${\mathrm{\Lambda }}_p^{\ast }:=\{0,\dots ,p-1\},$

the corresponding Wishart distribution has no Lebesgue density.

Wishart–Laguerre ensembles and β-extensions

In random matrix theory, the Wishart family is often studied through its Laguerre ensembles. For the real case (orthogonal symmetry, Template:Math=1), the joint density of the eigenvalues ${\lambda }_1,\dots ,{\lambda }_p\ge 0$ of $𝐗\sim W_p(𝐈,n)$ is

$c_{n,p}\exp (-\frac{1}{2}\sum _{i=1}^p{\lambda }_i)\prod _{i=1}^p{\lambda }_i^{\frac{n-p-1}{2}}\prod _{1\le i<j\le p}|{\lambda }_i-{\lambda }_j|,$

which is the Laguerre orthogonal ensemble (LOE). The complex and quaternion analogues are the Laguerre unitary (LUE, $\beta =2$) and Laguerre symplectic (LSE, $\beta =4$) ensembles, respectively.

β-Laguerre ensemble (general $\beta >0$)

A further generalization, the β-Laguerre ensemble, allows the Dyson index $\beta >0$ to vary continuously. Its joint eigenvalue density has the Coulomb gas form

$p({\lambda }_1,\dots ,{\lambda }_p)\propto \left(\prod _{i=1}^p{\lambda }_i^{\alpha }{e}^{-\frac{\beta }{2}{\lambda }_i}\right)(\prod _{i<j}|{\lambda }_i-{\lambda }_j{|}^{\beta }),\alpha >-1,$

which reduces to LOE/LUE/LSE for $\beta =1,2,4$. For the classical Gaussian Wishart case (scale $𝐈$), one has the identification

$\alpha =\frac{\beta }{2}(n-p+1)-1.$

A concrete probabilistic construction for any $\beta >0$ is provided by the Dumitriu–Edelman bidiagonal model. One samples a random bidiagonal matrix $𝐁$ with independent chi variables and sets $𝐋=𝐁{𝐁}^T$; the eigenvalues of $𝐋$ follow the β-Laguerre law with parameter $\alpha$.

Sampling

• Classical $\beta =1,2,4$. Bartlett's decomposition gives $𝐗=𝐋𝐀{𝐀}^T{𝐋}^T$ with $𝐀$ lower-triangular having chi/normal entries, yielding LOE/LUE/LSE exactly.
• General $\beta >0$. In the Dumitriu–Edelman model, for matrix size $p$ and parameter $\alpha >-1$, one samples independently

$B_{ii}\sim \frac{1}{\sqrt{\beta }}{\chi }_{\beta (\alpha +p-i+1)},B_{i,i-1}\sim \frac{1}{\sqrt{\beta }}{\chi }_{\beta (p-i+1)}(i=2,\dots ,p),$

and sets $𝐋=𝐁{𝐁}^T$. Then the eigenvalues of $𝐋$ have the β-Laguerre joint density above.

Rectangular data matrices

If one wishes to realize these spectra as the singular values of a rectangular $M\times N$ matrix (with $M\ge N=p$), draw independent Haar-distributed $𝐔\in O(M)$ (or $U(M)$) and $𝐕\in O(N)$, and set

$X=U\left[\begin{array}{c}\mathrm{diag}(\sqrt{{\lambda }_1},\dots ,\sqrt{{\lambda }_p})\\ 0\end{array}\right]V^{\ast },$

so that $X^{\ast }X$ has eigenvalues $\{{\lambda }_i\}$. For i.i.d. Gaussian entries (true Wishart), $\beta$ is fixed by the field (real, complex, quaternion) and $\alpha$ reduces to $\frac{\beta }{2}(n-p+1)-1$.

Hard-edge behavior and universality

At the “hard edge” (near $\lambda =0$), β-Laguerre ensembles exhibit Bessel-kernel correlations and level repulsion of order $\beta$. In particular, with $p\to \mathrm{\infty }$ and fixed $\alpha$, the distribution of the smallest eigenvalue converges to a universal hard-edge (Bessel) law that depends only on $\alpha$ and $\beta$.

Macroscopic limit (Marchenko–Pastur law)

Under proportional growth $p/n\to \gamma \in (0,\mathrm{\infty })$, the empirical spectral distribution of $n^{-1}𝐆{𝐆}^T$ (with i.i.d. entries of variance ${\sigma }^2$) converges almost surely to the Marchenko–Pastur law, supported on ${\sigma }^2(1\pm \sqrt{\gamma }{)}^2$. When $\gamma =1$, the density diverges as $x^{-1/2}$ at the origin (a hard-edge singularity).

Relationships to other distributions

• The Wishart distribution is related to the inverse-Wishart distribution, denoted by $W_p^{-1}$, as follows: If Template:Math and if we do the change of variables Template:Math, then $𝐂\sim W_p^{-1}({𝐕}^{-1},n)$. This relationship may be derived by noting that the absolute value of the Jacobian determinant of this change of variables is Template:Math, see for example equation (15.15) in.
• In Bayesian statistics, the Wishart distribution is a conjugate prior for the precision parameter of the multivariate normal distribution, when the mean parameter is known.
• A generalization is the multivariate gamma distribution.
• A different type of generalization is the normal-Wishart distribution, essentially the product of a multivariate normal distribution with a Wishart distribution.

See also

• Chi-squared distribution
• Complex Wishart distribution
• F-distribution
• Gamma distribution
• Hotelling's T-squared distribution
• Inverse-Wishart distribution
• Multivariate gamma distribution
• Student's t-distribution
• Wilks' lambda distribution

External links

• A C++ library for random matrix generator