Normal-inverse-Wishart distribution

Template:Short description Template:Probability distribution In probability theory and statistics, the normal-inverse-Wishart distribution (or Gaussian-inverse-Wishart distribution) is a multivariate four-parameter family of continuous probability distributions. It is the conjugate prior of a multivariate normal distribution with unknown mean and covariance matrix (the inverse of the precision matrix).

Definition

Suppose

$𝝁|{𝝁}_0,\lambda ,𝜮\sim 𝒩\left(𝝁\right|{𝝁}_0,\frac{1}{\lambda }𝜮)$

has a multivariate normal distribution with mean ${𝝁}_0$ and covariance matrix $\frac{1}{\lambda }𝜮$, where

$𝜮|𝜳,\nu \sim {𝒲}^{-1}(𝜮|𝜳,\nu )$

has an inverse Wishart distribution. Then $(𝝁,𝜮)$ has a normal-inverse-Wishart distribution, denoted as

$(𝝁,𝜮)\sim \mathrm{N}\mathrm{I}\mathrm{W}({𝝁}_0,\lambda ,𝜳,\nu ).$

Characterization

Probability density function

$f(𝝁,𝜮|{𝝁}_0,\lambda ,𝜳,\nu )=𝒩\left(𝝁\right|{𝝁}_0,\frac{1}{\lambda }𝜮){𝒲}^{-1}(𝜮|𝜳,\nu )$

The full version of the PDF is as follows:

$f(𝝁,𝜮|{𝝁}_0,\lambda ,𝜳,\nu )=\frac{{\lambda }^{D/2}|𝜳{|}^{\nu /2}|𝜮{|}^{-\frac{\nu +D+2}{2}}}{(2\pi {)}^{D/2}{2}^{\frac{\nu D}{2}}{\Gamma }_D(\frac{\nu }{2})}\text{exp}\{-\frac{1}{2}Tr({𝜳𝜮}^{-1})-\frac{\lambda }{2}(𝝁-{𝝁}_0{)}^T{𝜮}^{-1}(𝝁-{𝝁}_0)\}$

Here ${\Gamma }_D[\cdot ]$ is the multivariate gamma function and $Tr(𝜳)$ is the Trace of the given matrix.

Properties

Marginal distributions

By construction, the marginal distribution over $𝜮$ is an inverse Wishart distribution, and the conditional distribution over $𝝁$ given $𝜮$ is a multivariate normal distribution. The marginal distribution over $𝝁$ is a multivariate t-distribution.

Posterior distribution of the parameters

Suppose the sampling density is a multivariate normal distribution

${𝒚}_{𝒊}|𝝁,𝜮\sim {𝒩}_p(𝝁,𝜮)$

where $𝒚$ is an $n\times p$ matrix and ${𝒚}_{𝒊}$ (of length $p$) is row $i$ of the matrix .

With the mean and covariance matrix of the sampling distribution is unknown, we can place a Normal-Inverse-Wishart prior on the mean and covariance parameters jointly

$(𝝁,𝜮)\sim \mathrm{N}\mathrm{I}\mathrm{W}({𝝁}_0,\lambda ,𝜳,\nu ).$

The resulting posterior distribution for the mean and covariance matrix will also be a Normal-Inverse-Wishart

$(𝝁,𝜮|y)\sim \mathrm{N}\mathrm{I}\mathrm{W}({𝝁}_n,{\lambda }_n,{𝜳}_n,{\nu }_n),$

where

${𝝁}_n=\frac{\lambda {𝝁}_0+n\overline{𝒚}}{\lambda +n}$

${\lambda }_n=\lambda +n$

${\nu }_n=\nu +n$

${𝜳}_n=𝜳\mathit{+}𝑺+\frac{\lambda n}{\lambda +n}(\overline{𝒚}\mathit{-}{𝝁}_0)(\overline{𝒚}\mathit{-}{𝝁}_0{)}^T\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}𝑺=\sum _{i=1}^n({𝒚}_{𝒊}\mathit{-}\overline{𝒚})({𝒚}_{𝒊}\mathit{-}\overline{𝒚}{)}^T$.

To sample from the joint posterior of $(𝝁,𝜮)$, one simply draws samples from $𝜮|𝒚\sim {𝒲}^{-1}({𝜳}_n,{\nu }_n)$, then draw $𝝁|𝜮,𝒚\sim {𝒩}_p({𝝁}_n,𝜮/{\lambda }_n)$. To draw from the posterior predictive of a new observation, draw $\widetilde{𝒚}|𝝁,𝜮,𝒚\sim {𝒩}_p(𝝁,𝜮)$ , given the already drawn values of $𝝁$ and $𝜮$.

Generating normal-inverse-Wishart random variates

Generation of random variates is straightforward:

1. Sample $𝜮$ from an inverse Wishart distribution with parameters $𝜳$ and $\nu$
2. Sample $𝝁$ from a multivariate normal distribution with mean ${𝝁}_0$ and variance $\frac{1}{\lambda }𝜮$

Related distributions

• The normal-Wishart distribution is essentially the same distribution parameterized by precision rather than variance. If $(𝝁,𝜮)\sim \mathrm{N}\mathrm{I}\mathrm{W}({𝝁}_0,\lambda ,𝜳,\nu )$ then $(𝝁,{𝜮}^{-1})\sim \mathrm{N}\mathrm{W}({𝝁}_0,\lambda ,{𝜳}^{-1},\nu )$ .
• The normal-inverse-gamma distribution is the one-dimensional equivalent.
• The multivariate normal distribution and inverse Wishart distribution are the component distributions out of which this distribution is made.

References

• Bishop, Christopher M. (2006). Pattern Recognition and Machine Learning. Springer Science+Business Media.
• Murphy, Kevin P. (2007). "Conjugate Bayesian analysis of the Gaussian distribution." [1]