Asymptotic theory (statistics)

Template:Short descriptionIn statistics, asymptotic theory, or large sample theory, is a framework for assessing properties of estimators and statistical tests. Within this framework, it is often assumed that the sample size Template:Math may grow indefinitely; the properties of estimators and tests are then evaluated under the limit of Template:Math. In practice, a limit evaluation is considered to be approximately valid for large finite sample sizes too.

Overview

Most statistical problems begin with a dataset of size Template:Math. The asymptotic theory proceeds by assuming that it is possible (in principle) to keep collecting additional data, thus that the sample size grows infinitely, i.e. Template:Math. Under the assumption, many results can be obtained that are unavailable for samples of finite size. An example is the weak law of large numbers. The law states that for a sequence of independent and identically distributed (IID) random variables Template:Math, if one value is drawn from each random variable and the average of the first Template:Math values is computed as Template:Math, then the Template:Math converge in probability to the population mean Template:Math as Template:Math.

In asymptotic theory, the standard approach is Template:Math. For some statistical models, slightly different approaches of asymptotics may be used. For example, with panel data, it is commonly assumed that one dimension in the data remains fixed, whereas the other dimension grows: Template:Math and Template:Math, or vice versa.

Besides the standard approach to asymptotics, other alternative approaches exist:

• Within the local asymptotic normality framework, it is assumed that the value of the "true parameter" in the model varies slightly with Template:Math, such that the Template:Math-th model corresponds to Template:Math. This approach lets us study the regularity of estimators.
• When statistical tests are studied for their power to distinguish against the alternatives that are close to the null hypothesis, it is done within the so-called "local alternatives" framework: the null hypothesis is Template:Math and the alternative is Template:Math. This approach is especially popular for the unit root tests.
• There are models where the dimension of the parameter space Template:Math slowly expands with Template:Math, reflecting the fact that the more observations there are, the more structural effects can be feasibly incorporated in the model.
• In kernel density estimation and kernel regression, an additional parameter is assumed—the bandwidth Template:Math. In those models, it is typically taken that Template:Math as Template:Math. The rate of convergence must be chosen carefully, though, usually Template:Math.

In many cases, highly accurate results for finite samples can be obtained via numerical methods (i.e. computers); even in such cases, though, asymptotic analysis can be useful. This point was made by Template:Harvtxt, as follows. Template:Quote

Asymptotic properties

Point estimators

In the following, $({\hat{\theta }}_n{)}_{n\in \mathbb{N}}$ will be an arbitrary sequence of estimators.

Consistency

The sequence $({\hat{\theta }}_n{)}_{n\in \mathbb{N}}$ is said to be (weakly) consistent, if it converges in probability to the true value of the parameter being estimated:

$${\hat{\theta }}_n\stackrel{\stackrel{}{p}}{\to }{\theta }_0.$$

This means that the probability that the estimated value gets arbitrarily close to ${\theta }_0$ converges to zero as $n$ tends to infinity. More specifically, if the convergence of $({\hat{\theta }}_n{)}_{n\in \mathbb{N}}$ to ${\theta }_0$ is of stochastic order $1/a_n$ (also written ${\mathit{\text{𝑂}}}_p(1/a_n)$) for some sequence $(a_n{)}_{n\in \mathbb{N}}$ of real numbers diverging to infinity, that is,

$$\underset{n\to \mathrm{\infty }}{lim}\mathbb{P}(|a_n({\hat{\theta }}_n-{\theta }_0)|>K)=0\text{for all }K>0,$$

then $({\hat{\theta }}_n{)}_{n\in \mathbb{N}}$ is called $a_n$-consistent.

The sequence of estimates is said be strongly consistent, if it even converges almost surely to the true parameter:

$${\hat{\theta }}_n\stackrel{\mathrm{a}\mathrm{.}\mathrm{s}\mathrm{.}}{⟶}{\theta }_0.$$

The subtle difference to a weakly consistent estimator is that a strongly consistent estimator is guaranteed to get arbitrarily close to ${\theta }_0$ as $n$ tends to infinity, eventually.

Asymptotic distribution

If it is possible to find sequences of non-random constants $(a_n{)}_{n\in \mathbb{N}}$, $(b_n{)}_{n\in \mathbb{N}}$ (possibly depending on the value ${\theta }_0$), and a non-degenerate distribution $\mu$ such that

$$b_n({\hat{\theta }}_n-a_n)\stackrel{d}{\to }\mu ,$$

then the sequence of estimators ${\hat{\theta }}_n$ is said to have the asymptotic distribution $\mu$.

Most often, the asymptotic distribution of estimators encountered in practice is the normal distribution $\mu =𝒩(0,V)$ for some covariance matrix $V$, in which case the estimator is called asymptotically normal. Some authors require specifically $a_n={\theta }_0$ and $b_n=\sqrt{n}$ and then call the sequence of estimators asymptotically normal for ${\theta }_0$.

Unbiasedness in the limit

The sequence is called unbiased in the limit, if $\underset{n\to \mathrm{\infty }}{lim}𝔼[{\hat{\theta }}_n]={\theta }_0$. An estimator with this property is also sometimes called approximately unbiased.

Asymptotic unbiasedness

The sequence is called asymptotically unbiased for the parameter ${\theta }_0$, if ${\sigma }_n({\hat{\theta }}_n-{\theta }_0)$ converges in distribution to a random variable $G\ne 0$ with $𝔼[G]=0$ for some sequence $({\sigma }_n{)}_{n\in \mathbb{N}}$ of real numbers.

Limiting variance

If there exists a sequence $({\tau }_n{)}_{n\in \mathbb{N}}$ of real numbers such that $\underset{n\to \mathrm{\infty }}{lim}{\tau }_n\mathrm{V}\mathrm{a}\mathrm{r}({\hat{\theta }}_n)={\sigma }^2$, then ${\sigma }^2$ is called the limiting variance of $({\hat{\theta }}_n{)}_{n\in \mathbb{N}}$. Typically, it is chosen ${\tau }_n=n$.

Asymptotic variance

If there exists a sequence $(b_n{)}_{n\in \mathbb{N}}$ such that

$$b_n({\hat{\theta }}_n-{\theta }_0)\stackrel{d}{\to }G$$

with $0<𝔼[G^2]<\mathrm{\infty }$, then $V=\mathrm{V}\mathrm{a}\mathrm{r}(G)$ is called the asymptotic variance of $({\hat{\theta }}_n{)}_{n\in \mathbb{N}}$. In particular, if $({\hat{\theta }}_n{)}_{n\in \mathbb{N}}$ is asymptotically unbiased, then $V=𝔼[G^2]$.

Asymptotic efficiency

The sequence is called asymptotically efficient, if it is asymptotically normal for ${\theta }_0$ and its asymptotic variance is $V=I({\theta }_0{)}^{-1}$ with the Fisher information $I({\theta }_0)$, that is, the asymptotic variance achieves the Cramér–Rao bound.

General relations between asymptotic properties

As above, let $({\hat{\theta }}_n{)}_{n\in \mathbb{N}}$ be an arbitrary sequence of estimators.

• If $({\hat{\theta }}_n{)}_{n\in \mathbb{N}}$ is asymptotically normal for ${\theta }_0$, then it is consistent (by Slutsky's theorem).
• If $({\hat{\theta }}_n{)}_{n\in \mathbb{N}}$ is consistent, then it is asymptotically unbiased.
• If $({\hat{\theta }}_n{)}_{n\in \mathbb{N}}$ is asymptotically unbiased and uniformly integrable, then it is unbiased in the limit. This is the case, for example, when the second moments are uniformly bounded, i.e., $\underset{n\in \mathbb{N}}{sup}𝔼[|{\hat{\theta }}_n{|}^2]<\mathrm{\infty }$.
• In the univariate case, if an estimator is asymptotically unbiased and has asymptotic variance ${\tau }^2$, and if the limiting variance ${\sigma }^2$ exists, then ${\tau }^2\le {\sigma }^2$. Equility holds if and only if the normalized sequence of estimators, that is $b_n({\hat{\theta }}_n-{\theta }_0)$ for suitable $b_n$, is uniformly integrable (which is often the case in practice).

Asymptotic theorems

Asymptotic properties of a sequence of estimators $({\hat{\theta }}_n{)}_{n\in \mathbb{N}}$ (for a parameter $\theta$) often result from general theorems from probability theory. In the following, let $X_1,X_2,\dots$ be independent and identically distributed observations from the true distribution ${\mathbb{P}}_{\theta }$.

For consistency

Strong law of large numbers

If the estimators are of the form ${\hat{\theta }}_n=\frac{1}{n}\sum _{i=1}^{n}f(X_i)$ for some measurable function $f$ such that $𝔼[|f(X_1)|]<\mathrm{\infty }$, then by the strong law of large numbers

$${\hat{\theta }}_n\stackrel{a.s.}{\to }𝔼[f(X_1)].$$

Thus, if $\theta =𝔼[f(X_1)]$, then $({\hat{\theta }}_n{)}_{n\in \mathbb{N}}$ is strongly consistent.

Continuous mapping theorem

Suppose $\theta =f(\tau )$ for some other parameter $\tau$ and a continuous function $f$. Given a (strongly) consistent sequence of estimators $({\hat{\tau }}_n{)}_{n\in \mathbb{N}}$ for the parameter $\tau$, then by the continuous mapping theorem, the sequence of estimators $({\hat{\theta }}_n{)}_{n\in \mathbb{N}}=(f({\hat{\tau }}_n){)}_{n\in \mathbb{N}}$ is (strongly) consistent for the parameter $\theta$.

For asymptotic distribution

Central limit theorem

If the estimators are of the form ${\hat{\theta }}_n=\frac{1}{n}\sum _{i=1}^{n}f(X_i)$ for some real-valued measurable function $f$ such that $𝔼[|f(X_1){|}^2]<\mathrm{\infty }$, then by the central limit theorem, $({\hat{\theta }}_n{)}_{n\in \mathbb{N}}$ is asymptotically normal in the sense

$$\sqrt{n}\frac{{\hat{\theta }}_n-\mu }{\sigma }\stackrel{d}{⟶}𝒩(0,1)$$

with the definitions $𝔼[f(X_1)]=\mu$ and $\mathrm{V}\mathrm{a}\mathrm{r}(f(X_1))={\sigma }^2$.

Fisher–Tippet–Gnedenko theorem

If the estimators are of the form ${\hat{\theta }}_n=\underset{i=1,\dots ,n}{max}f(X_i)$ for some real-valued measurable function $f$ and if there exists sequences $(a_n{)}_{n\in \mathbb{N}}$ and $(b_n{)}_{n\in \mathbb{N}}$ and a non-degenerate distribution $G$ such that

$$\frac{{\hat{\theta }}_n-a_n}{b_n}\stackrel{d}{⟶}G,$$

then, by the Fisher–Tippet–Gnedenko theorem, the asymptotic distribution $G$ of $({\hat{\theta }}_n{)}_{n\in \mathbb{N}}$ is a generalized extreme value distribution.

Delta method

Suppose $\theta =f(\tau )$ for some other parameter $\tau$ and a differentiable function $f$ with $\mathrm{\nabla }f(\tau )\ne 0$. Given a sequence of estimators $({\hat{\tau }}_n{)}_{n\in \mathbb{N}}$ for the parameter $\tau$ having asymptotic distribution $G$ in the sense

$$\frac{{\hat{\tau }}_n-\tau }{{\sigma }_n}\stackrel{d}{⟶}Z\sim G$$

for some sequence of real numbers $({\sigma }_n{)}_{n\in \mathbb{N}}$, then by the delta method, the sequence of estimators $({\hat{\theta }}_n{)}_{n\in \mathbb{N}}=(f({\hat{\tau }}_n){)}_{n\in \mathbb{N}}$ has asymptotically the distribution of $\mathrm{\nabla }f(\tau {)}^{\mathrm{\top }}Z$.

In the special case, where $({\hat{\tau }}_n{)}_{n\in \mathbb{N}}$ is asymptotically normal for $\tau$ with asymptotic variance $V$, the sequence $({\hat{\theta }}_n{)}_{n\in \mathbb{N}}$ is asymptotically normal with asymptotic variance $\mathrm{\nabla }f(\tau {)}^{\mathrm{\top }}V\mathrm{\nabla }f(\tau )$.

See also

• Asymptotic analysis
• Exact statistics
• Large deviations theory

Bibliography

• Template:Citation
• Template:Citation
• Template:Citation
• Template:Citation
• Template:Citation
• Template:Citation
• Template:Citation
• Template:Citation
• Template:Citation
• Template:Citation
• Template:Citation
• Template:Citation
• Template:Citation
• Template:Cite book
• Template:Citation
• Template:Citation
• Template:Citation