Ordinal regression

Template:Short description Template:Regression bar In statistics, ordinal regression, also called ordinal classification, is a type of regression analysis used for predicting an ordinal variable, i.e. a variable whose value exists on an arbitrary scale where only the relative ordering between different values is significant. It can be considered an intermediate problem between regression and classification. Examples of ordinal regression are ordered logit and ordered probit. Ordinal regression turns up often in the social sciences, for example in the modeling of human levels of preference (on a scale from, say, 1–5 for "very poor" through "excellent"), as well as in information retrieval. In machine learning, ordinal regression may also be called ranking learning.Template:Efn

Linear models for ordinal regression

Ordinal regression can be performed using a generalized linear model (GLM) that fits both a coefficient vector and a set of thresholds to a dataset. Suppose one has a set of observations, represented by length-Template:Mvar vectors Template:Math through Template:Math, with associated responses Template:Math through Template:Mvar, where each Template:Mvar is an ordinal variable on a scale Template:Math. For simplicity, and without loss of generality, we assume Template:Mvar is a non-decreasing vector, that is, Template:Mvar. To this data, one fits a length-Template:Mvar coefficient vector Template:Math and a set of thresholds Template:Math with the property that Template:Math. This set of thresholds divides the real number line into Template:Mvar disjoint segments, corresponding to the Template:Mvar response levels.

The model can now be formulated as

$\mathrm{Pr}(y\le i\mid 𝐱)=\sigma ({\theta }_i-𝐰\cdot 𝐱)$

or, the cumulative probability of the response Template:Mvar being at most Template:Mvar is given by a function Template:Mvar (the inverse link function) applied to a linear function of Template:Math. Several choices exist for Template:Mvar; the logistic function

$\sigma ({\theta }_i-𝐰\cdot 𝐱)=\frac{1}{1+e^{-({\theta }_i-𝐰\cdot 𝐱)}}$

gives the ordered logit model, while using the CDF of the standard normal distribution gives the ordered probit model. A third option is to use an exponential function

$\sigma ({\theta }_i-𝐰\cdot 𝐱)=1-\exp (-\exp ({\theta }_i-𝐰\cdot 𝐱))$

which gives the proportional hazards model.

Latent variable model

The probit version of the above model can be justified by assuming the existence of a real-valued latent variable (unobserved quantity) Template:Mvar, determined by

$y^{*}=𝐰\cdot 𝐱+\epsilon$

where Template:Mvar is normally distributed with zero mean and unit variance, conditioned on Template:Math. The response variable Template:Mvar results from an "incomplete measurement" of Template:Mvar, where one only determines the interval into which Template:Mvar falls:

$y=\{\begin{array}{cc}1& \text{if}{y}^{*}\le {\theta }_1,\\ 2& \text{if}{\theta }_1<y^{*}\le {\theta }_2,\\ 3& \text{if}{\theta }_2<y^{*}\le {\theta }_3\\ ⋮\\ K& \text{if}{\theta }_{K-1}<y^{*}.\end{array}$

Defining Template:Math and Template:Math, the above can be summarized as Template:Math if and only if Template:Math.

From these assumptions, one can derive the conditional distribution of Template:Mvar asTemplate:R

$\begin{array}{cc}P(y=k\mid 𝐱)& =P({\theta }_{k-1}<y^{*}\le {\theta }_k\mid 𝐱)\\ & =P({\theta }_{k-1}<𝐰\cdot 𝐱+\epsilon \le {\theta }_k)\\ & =\mathrm{\Phi }({\theta }_k-𝐰\cdot 𝐱)-\mathrm{\Phi }({\theta }_{k-1}-𝐰\cdot 𝐱)\end{array}$

where Template:Math is the cumulative distribution function of the standard normal distribution, and takes on the role of the inverse link function Template:Mvar. The log-likelihood of the model for a single training example Template:Math, Template:Mvar can now be stated asTemplate:R

$\log ℒ(𝐰,\mathit{\theta }\mid {𝐱}_i,y_i)=\sum _{k=1}^K[y_i=k]\log [\mathrm{\Phi }({\theta }_k-𝐰\cdot {𝐱}_i)-\mathrm{\Phi }({\theta }_{k-1}-𝐰\cdot {𝐱}_i)]$

(using the Iverson bracket Template:Math.) The log-likelihood of the ordered logit model is analogous, using the logistic function instead of Template:Math.

Alternative models

In machine learning, alternatives to the latent-variable models of ordinal regression have been proposed. An early result was PRank, a variant of the perceptron algorithm that found multiple parallel hyperplanes separating the various ranks; its output is a weight vector Template:Math and a sorted vector of Template:Math thresholds Template:Math, as in the ordered logit/probit models. The prediction rule for this model is to output the smallest rank Template:Mvar such that Template:Math.

Other methods rely on the principle of large-margin learning that also underlies support vector machines.

Another approach is given by Rennie and Srebro, who, realizing that "even just evaluating the likelihood of a predictor is not straight-forward" in the ordered logit and ordered probit models, propose fitting ordinal regression models by adapting common loss functions from classification (such as the hinge loss and log loss) to the ordinal case.

Software

ORCA (Ordinal Regression and Classification Algorithms) is an Octave/MATLAB framework including a wide set of ordinal regression methods.

R packages that provide ordinal regression methods include MASS and Ordinal.

See also

• Logistic regression

Further reading

• Template:Cite book
• Template:Cite book
• Template:Cite book