Draft:Constrained minimum criterion

---

In statistics, the Constrained Minimum Criterion (CMC) is a criterion for selecting regression models founded on the classical theory of likelihood based inference. It is a frequentist alternative to the Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC) with certain advantages.

Geometric motivation

For a full regression model with $p$ predictor variables and an intercept, the unknown vector of regression parameters ${\mathit{\beta }}^t$ is a $(p+1)$-vector. Elements of ${\mathit{\beta }}^t$ corresponding to active variables are non-zero, and elements corresponding to inactive variables are all zero. The likelihood ratio confidence region for ${\mathit{\beta }}^t$ is centred on its maximum likelihood estimator $\hat{\mathit{\beta }}$. As the sample size $n$ goes to infinity, the confidence region shrinks in size and degenerates into $\hat{\mathit{\beta }}$, and $\hat{\mathit{\beta }}$ converges to ${\mathit{\beta }}^t$. It follows that the whole confidence region converges to ${\mathit{\beta }}^t$, so for sufficiently large $n$, elements of vectors in the confidence region corresponding to active variables are all non-zero. This implies that when ${\mathit{\beta }}^t$ is captured by the confidence region, it is a vector in the region having the most zeros in its elements. Because of this, the CMC chooses from the confidence region a vector with the most zeros in its elements as an estimate of ${\mathit{\beta }}^t$, thereby selecting the model defined by variables corresponding to non-zero elements of the chosen vector.

Definition

Let $ℳ=\{M_j{\}}_{j=1}^{2^p}$ be the collection of $2^p$ subsets of the $p$ variables where each $M_j$ represents a subset. Denote by ${\hat{\mathit{\beta }}}_j$ the maximum likelihood estimator for the vector of regression parameters of the reduced model defined by $M_j$. Augment ${\hat{\mathit{\beta }}}_j$ to be of dimension $(p+1)$ by adding zeros to its elements to represent variables not in $M_j$. For a fixed $\alpha \in (0,1)$, denote by ${𝒞}_{1-\alpha }$ the $100(1-\alpha )\mathrm{\%}$ likelihood ratio confidence region for ${\mathit{\beta }}^t$ which is a region in the $(p+1)$-dimensional space centred on $\hat{\mathit{\beta }}$. The CMC chooses the model represented by the solution vector of the following constrained minimization problem,

$\underset{ℳ}{\min }\Vert {\hat{\mathit{\beta }}}_j{\Vert }_0\text{ subject to }{\hat{\mathit{\beta }}}_j\in {𝒞}_{1-\alpha },$

where $\Vert \cdot {\Vert }_0$ denotes the $L_0$ norm. The solution vector is called the CMC solution, which is a sparse estimator of ${\mathit{\beta }}^t$. Its corresponding model is called the CMC selection. When there are two or more solution vectors to the minimization problem, the one with the highest likelihood is chosen to be the CMC solution.

Asymptotic properties

Let ${\hat{\mathit{\beta }}}_{\alpha }$ be the CMC solution and ${\hat{M}}_{\alpha }$ be the corresponding CMC selection. Under regularity conditions for the asymptotic normality of the maximum likelihood estimator $\hat{\mathit{\beta }}$, ($i$) the CMC solution is consistent in that

${\hat{\mathit{\beta }}}_{\alpha }\stackrel{p}{⟶}{\mathit{\beta }}^t$

as $n\to \mathrm{\infty }$, and ($ii$) the probability that ${\hat{M}}_{\alpha }$ is the true model has an asymptotic lower bound

$\underset{n\to +\mathrm{\infty }}{\lim }P({\hat{M}}_{\alpha }=M_j^t)\ge 1-\alpha ,$

where $M_j^t$ denotes the unknown true model containing only and all active variables.

Tuning parameter

The tuning parameter $\alpha$ controls the balance between the false active rate and false inactive rate of the selected model which is also the balance between the fit and the sparsity of the selected model. When the sample size $n$ is large, the asymptotic lower bound in ($ii$) shows that setting $\alpha$ to a small value will lead to a high probability that the CMC selection is the true model. When $n$ is not large, a small $\alpha$ will lead to a high false inactive rate, so a larger value should be used. The recommended default value is $\alpha =0.5$. At this default value, the CMC is often more accurate than the AIC and BIC in terms of false active rate and false inactive rate.

The tuning parameter $\alpha$ makes it easy to adapt the CMC to special situations such as when $n$ is small. The AIC and BIC both require special adjustments to their penalty terms for small $n$ situations. The CMC can handle such situations with a simple change of the $\alpha$ level. In asymptotic properties ($i$) and ($ii$) above, the $\alpha$ level is fixed. Stronger results may be obtained by allowing $\alpha$ to vary with $n$. For selecting Gaussian linear models, one may let $\alpha$ go to zero at a certain speed depending on $n$ as $n$ goes to infinity so that the CMC selection is consistent; that is, one may find a sequence of tuning parameter values ${\alpha }_n\to 0$ such that

$\underset{n\to +\mathrm{\infty }}{\lim }P({\hat{M}}_{{\alpha }_n}=M_j^t)=1.$

Computation

For the best subset selection, the AIC and BIC require the computation of the maximum likelihood of all $2^p$ models. The CMC may require far fewer. Denote by $M_{-i}$ the model containing all variables except the $i$th variable ${𝐱}_i$. Denote by ${\hat{\mathit{\beta }}}_{-i}$ the maximum likelihood estimator and by $\lambda ({\hat{\mathit{\beta }}}_{-i})$ the maximum log-likelihood ratio of this model. In some cases, the value of $\lambda ({\hat{\mathit{\beta }}}_{-i})$ alone is sufficient to determine if ${𝐱}_i$ will be selected by the CMC. One can first compute $\lambda ({\hat{\mathit{\beta }}}_{-i})$ and use it to determine if ${𝐱}_i$ will be selected for $i=1,2,\dots ,p$. Suppose this has identified $p^{\prime }$ variables that will be selected. Then, one only needs to select from the remaining $p-p^{\prime }$ variables. The total number of models that need to be computed by the CMC is thus $p+2^{p-p^{\prime }}$ which could be substantially smaller than $2^p$ required by the AIC and BIC.

Remarks

Comprehensive discussions of model selection philosophies and criteria can be found in the literature. In other model selection strategies such as the AIC and BIC, the sparsity of the selected model comes as a by-product of the model selection process. By directly minimizing the size of the model subject to a lower bound constraint on the likelihood ratio, the CMC is the first model selection method to explicitly pursue the sparsity of the selected model.

See also

• Akaike information criterion
• Bayesian information criterion
• Hannan–Quinn information criterion
• Deviance information criterion
• Minimum message length

Further readings

• Template:Citation.
• Template:Citation.
• Template:Citation.
• Template:Citation.
• Template:Citation.