Frisch–Waugh–Lovell theorem

Ragnar Frisch, econometrician and founding member of the Econometric Society

In statistics and econometrics, the Frisch–Waugh–LovellTemplate:Efn (FWL) theorem is a theorem of ordinary least squares estimators which states that estimates of multiple regression coefficients depend only on the unique variation of each explanatory variable. The theorem is named for econometricians Ragnar Frisch, Frederick V. Waugh, and Michael C. Lovell.

Ordinary least squares is a method of estimating coefficients in a linear regression where a single dependent variable is modeled as a linear function of one or more explanatory variables. The Frisch–Waugh–Lovell theorem states that, in a linear regression model fitted by least squares, each explanatory variable's coefficient reflects the relationship between the dependent variable and the part of that explanatory variable which is not linearly related to the other explanatory variables. Specifically, each explanatory variable can be decomposed into two parts: the part linearly related to the other explanatory variables, and a residual component. Then, that variable's coefficient can be found by regressing the dependent variable on the residual component. As a result, the theorem is sometimes called the regression anatomy theorem.

Coefficients in least squares-estimated regressions are often interpreted as the effect of the respective variable controlling for or holding constant the set of other explanatory variables. However, given 'high' multicollinearity, "Interpreting coefficients [...] should be carried out with caution". The Frisch–Waugh–Lovell theorem shows that coefficients represent effects due to unique variation in each explanatory variable.

An initial version of the theorem was introduced by Udny Yule in 1907, though it was not popularized in economics until a 1933 paper by Ragnar Frisch and Frederick Waugh in the first volume of Econometrica. At the time, there was debate among economists over the proper way to adjust statistical models for the effect of time trends. Frisch and Waugh used the theorem to show that the two leading methods of adjustment were numerically equivalent, resolving the debate. Michael Lovell contributed to the theorem's development through a 1963 paper generalizing Frisch and Waugh's result beyond time trends to arbitrary sets of explanatory variables.

Background

The Frisch–Waugh–Lovell theorem is a result for regressions estimated by ordinary least squares, the most commonly used estimator in applied econometrics.Template:Sfn A regression is a statistical model where a dependent variable is modeled as a function of one or more explanatory variables plus some residual term. Ordinary least squares can be used when a dependent variable is modeled as a linear combination of one or more explanatory variables plus some residual term. For example, an individual's wages may be modeled as a linear function of a constant term, education, and parental income, with a residual term that encompasses deviations from the model's prediction. Ordinary least squares sets the values of the coefficients to minimize the sum of squared residuals.Template:Sfn Under a certain set of assumptions, the hypotheses of the Gauss–Markov theorem, least squares estimation is the best linear unbiased estimator.Template:Sfn

Let $y$ be any dependent variable and $x_1,x_2,\dots ,x_k$ a set of $k$ explanatory variables, and suppose $n$ observations of $(y_i,x_{1i},x_{2i},\dots ,x_{ki})$ are obtained. If $y$ is modeled as a linear function of the explanatory variables and a constant, the estimated model can be written as $y_i={\hat{\beta }}_0+{\hat{\beta }}_1{x}_{1i}+{\hat{\beta }}_2{x}_{2i}+\cdots +{\hat{\beta }}_k{x}_{ki}+{\hat{e}}_i$, where the hats denote estimates of the respective parameters. The least squares estimator sets the coefficients ${\hat{\beta }}_j,j\in \{0,\dots ,k\}$ to minimize the sum of squared residuals${\sum }_{i=1}^n{\hat{e}}_i^2$. With $n$ observations this involves minimizing across $n$ equations, and is typically written in matrix form as $y=X\hat{\beta }+\hat{e}$, where $y$ and $\hat{e}$ are $n\times 1$ vectors of dependent variable observations and residuals, respectively, $X$ is an $n\times k$ matrix of explanatory variables' observations, and $\hat{\beta }$ is a coefficient vector. Then, the least squares solution is $\hat{\beta }=(X^{\prime }X{)}^{-1}{X}^{\prime }y$ yielding $\hat{e}=y-X\hat{\beta }$.Template:Sfn

In regressions estimated by least squares, it is common to refer to an explanatory variable's coefficient as the effect of that variable "holding constant" the other explanatory variables.Template:Sfn For example, if wage is modeled as a function of education and work experience, the coefficient on education is interpreted as the difference in the expectation of wage for a unit difference in education, "holding constant" work experience. Econometrician Arthur Goldberger frames the Frisch–Waugh–Lovell theorem as "giving content to th[is] language".Template:Sfn

Definition and interpretation

The Frisch–Waugh–Lovell theorem states that in a least squares-estimated regression of the form

$y_i={\hat{\beta }}_0+{\hat{\beta }}_1{x}_{1i}+{\hat{\beta }}_2{x}_{2i}+\cdots +{\hat{\beta }}_k{x}_{ki}+{\hat{e}}_i$

any coefficient ${\hat{\beta }}_j,j\in \{1,\dots ,k\}$ can be obtained by the two-step process of:

1. Regress $x_j$ on the set of other explanatory variables, obtaining residuals ${\tilde{x}}_j$
2. Regress $y$ on ${\tilde{x}}_j$, obtaining ${\hat{\beta }}_j=\frac{\text{cov}(y,{\tilde{x}}_j)}{\text{var}({\tilde{x}}_j)}$

This two-step process is referred to as the residual regression or equivalently the regression anatomy theorem.Template:SfnTemplate:Sfn This result is a numerical property of least squares estimation and does not depend on statistical properties of the data.Template:SfnTemplate:Sfn

The variation in each explanatory variable can be decomposed into two parts: the part which is linearly related to the set of other explanatory variables and the remaining, 'residual' variation. By the theorem, an explanatory variable's coefficient can be found from the simple regression of the dependent variable on the residual part.Template:SfnTemplate:Sfn This result is the basis for interpreting the impact of including additional variables in a regression: it is equivalent to removing from the existing variables the component of shared variance with the new variables.Template:SfnTemplate:Sfn

Further, the Frisch–Waugh–Lovell theorem can be applied to understand how multicollinearity affects parameter estimation. When most of the variation in an explanatory variable is linearly explained by the other explanatory variables, the residual part has relatively very little variation. Consequently, the estimate of the explanatory variable's coefficient may be less precise than if fewer variables were controlled for.Template:SfnTemplate:Sfn

Example

Consider the regression of wage on education and parental income:

${\text{wage}}_i={\hat{\beta }}_0+{\hat{\beta }}_1{\text{education}}_i+{\hat{\beta }}_2{\text{parental income}}_i+{\hat{e}}_i$

While the least squares estimates for ${\hat{\beta }}_0,{\hat{\beta }}_1$ and ${\hat{\beta }}_2$ can be obtained by minimizing ${\sum }_{i=1}^n{\hat{e}}_i^2$ directly, each can be equivalently obtained by the two-step residual regression. In the case of education:

1. Regress education on parental income, saving the residuals from this regression: the part of education not linearly related to parental income
2. Regress wages on the residuals, obtaining the least squares estimate for ${\hat{\beta }}_1$

This illustrates how ${\hat{\beta }}_1$ reflects the effect of education on wages controlling for parental income: it is the relationship between wages and the part of education not linearly related to parental income.Template:Sfn

Double residual regression

The double residual regression is the three-step process:

1. Regress $x_j$ on the set of other explanatory variables, obtaining residuals ${\tilde{x}}_j$
2. Regress $y$ on the set of explanatory variables excluding $x_j$, obtaining residuals $\tilde{y}$
3. Regress $\tilde{y}$ on ${\tilde{x}}_j$, estimating ${\hat{\beta }}_j=\frac{\text{cov}(\tilde{y},{\tilde{x}}_j)}{\text{var}({\tilde{x}}_j)}$ and ${\hat{e}}_i={\tilde{y}}_i-{\hat{\beta }}_j{\tilde{x}}_{ji}$

Like the two-step process, this yields an identical coefficient to the full regression.Template:SfnTemplate:Sfn It includes the additional feature that the residuals from the regression in step 3 equal the residuals in the full regression.Template:SfnTemplate:Sfn

Multivariate definition

Consider the least squares-estimated regression $y=X\hat{\beta }+Z\hat{\delta }+\hat{e}$, where $y$ and $\hat{e}$ are $n\times 1$ vectors of dependent variable observations and residuals, respectively, $X$ is an $n\times k$ matrix of $k$ explanatory variables' observations, $Z$ is an $n\times p$ matrix of $p$ explanatory variables' observations, and $\hat{\beta }$ and $\hat{\delta }$ are $k\times 1$ and $p\times 1$ coefficient vectors for $X$ and $Z$, respectively. Then, the Frisch–Waugh–Lovell theorem states that

$\hat{\beta }=({\tilde{X}}^{\prime }\tilde{X}{)}^{-1}{\tilde{X}}^{\prime }y=({\tilde{X}}^{\prime }\tilde{X}{)}^{-1}{\tilde{X}}^{\prime }\tilde{y}$

$\hat{e}=\tilde{y}-\tilde{X}\hat{\beta }$

where $\tilde{X}=X-Z(Z^{\prime }Z{)}^{-1}{Z}^{\prime }X$, the residuals from the least squares regression of $X$ on $Z$, and $\tilde{y}=y-Z(Z^{\prime }Z{)}^{-1}{Z}^{\prime }y$, the residuals from the least squares regression of $y$ on $Z$. The first expression of $\hat{\beta }$ is the residual regression, while the second is the double residual regression.Template:SfnTemplate:SfnTemplate:Sfn

Geometric interpretation

With a linear regression of the form $y=X\hat{\beta }+Z\hat{\delta }+\hat{e}$, the fitted values $X\hat{\beta }+Z\hat{\delta }$ can be interpreted as the orthogonal projection of $y$ onto the column space of $[XZ]$, $\text{Col}(X,Z)$.Template:Sfn The Frisch–Waugh–Lovell theorem is then (in the double residual regression case) the three step process:

1. Project $X$ onto the orthogonal complement of $\text{Col}(Z)$, obtaining residuals $\tilde{X}$
2. Project $y$ onto the orthogonal complement of $\text{Col}(Z)$, obtaining residual vector $\tilde{y}$
3. Project $\tilde{y}$ onto $\text{Col}(\tilde{X})$, obtaining projection $\tilde{X}\beta$ and residuals $e=\tilde{y}-\tilde{X}\beta$

The resulting $\hat{\beta }$ and residuals are identical to those in the full regression of $y$ on $X$ and $Z$.Template:SfnTemplate:SfnTemplate:Sfn

Proof

Consider the least squares-estimated regression $y=X\hat{\beta }+Z\hat{\delta }+\hat{e}$ and annihilator matrix $M_Z=I-Z(Z^{\prime }Z{)}^{-1}{Z}^{\prime }$. Premultiplying both sides of the regression equation by the annihilator matrix removes from $y$ and $X$ the part linearly explained by $Z$:

$\begin{array}{cc}{M}_{Z}y& =M_Z(X\hat{\beta }+Z\hat{\delta }+\hat{e})\\ \tilde{y}& =M_{Z}X\hat{\beta }+M_{Z}Z\hat{\delta }+M_Z\hat{e}\\ \tilde{y}& =\tilde{X}\hat{\beta }+\hat{e}\end{array}$

Where the third line follows from $M_{Z}Z=0$ by construction and $M_Z\hat{e}=\hat{e}$ from $Z^{\prime }\hat{e}=0$, a property of least squares where $\hat{e}$ is in the orthogonal complement of the column space of $Z$. Then, by the least squares solution, $\hat{\beta }=({\tilde{X}}^{\prime }\tilde{X}{)}^{-1}{\tilde{X}}^{\prime }\tilde{y}$ and $\hat{e}=\tilde{y}-\tilde{X}\hat{\beta }$. This concludes the proof of the double residual regression form of the Frisch-Waugh-Lovell theorem.Template:SfnTemplate:SfnTemplate:Sfn

The residual regression – where $Z$ is projected out of $X$ but not $y$ – returns a coefficient vector equal to the double residual regression. Their equivalence can be proved:

$({\tilde{X}}^{\prime }\tilde{X}{)}^{-1}{\tilde{X}}^{\prime }y=({\tilde{X}}^{\prime }\tilde{X}{)}^{-1}{X}^{\prime }{M}_{Z}y=({\tilde{X}}^{\prime }\tilde{X}{)}^{-1}{X}^{\prime }{M}_Z{M}_{Z}y=({\tilde{X}}^{\prime }\tilde{X}{)}^{-1}{\tilde{X}}^{\prime }\tilde{y}$

Where the intermediary inequalities follow from the idempotency and symmetry of the annihilator matrix.Template:SfnTemplate:Sfn

History

Yule

In 1907, statistician Udny Yule introduced a new system of notation for, and derived a number of algebraic results of, least squares-estimated regression coefficients. Among his results was an early form of the Frisch–Waugh–Lovell theorem.Template:SfnTemplate:SfnTemplate:Sfn In Yule's notation, where $x_{1\cdot 34\dots n}$ represents the residuals from the regression of $x_1$ on $x_3$ through $x_n$, and $x_{2\cdot 34\dots n}$ the residuals from the regression of $x_2$ on $x_3$ through $x_n$, he finds that the regression of $x_{1\cdot 34\dots n}$ on $x_{2\cdot 34\dots n}$ yields the coefficient for $x_2$ in the full regression of $x_1$ on $x_2$ through $x_n$. He notes that this relationship holds "quite generally and without reference to the form of the [variables'] frequency distribution."Template:Sfn With this result, Yule defines the multiple regression coefficient $b_{12\cdot 34\dots n}$ – the coefficient on $x_2$ in the regression of $x_1$ on $x_2$ through $x_n$ – as the simple regression of $x_{1\cdot 34\dots n}$ on $x_{2\cdot 34\dots n}$:

$\begin{array}{c}{b}_{12\cdot 34\dots n}=\frac{\sum x_{1\cdot 34\dots n}{x}_{2\cdot 34\dots n}}{\sum x_{2\cdot 34\dots n}^2}\end{array}$

Having related simple regression coefficients to multiple regression coefficients, Yule describes his result as filling a gap in the interpretation of least squares coefficients and partial correlations by showing that they reflect "an actual correlation between determinate variables."Template:SfnTemplate:Sfn

Using Yule's notation, in a 1968 text econometrician Arthur Goldberger states the residual regression form of the Frisch–Waugh–Lovell theorem as $b_{1(2\cdot 3)}=b_{12\cdot 3}$ and the double residual regression form as $b_{(1\cdot 3)(2\cdot 3)}=b_{12\cdot 3}$.Template:Sfn

Frisch and Waugh

In the early 20th century, there was debate among economists over the correct approach to adjusting time series data used in regressions for the influence of linear trends. The two primary methods in question were the direct de-trending of each time series and the inclusion of a time trend in the regression. In 1933 and using the notation introduced by Yule, a paper in the first volume of Econometrica by econometricians Ragnar Frisch and Frederick V. Waugh proved the equivalence between the two methods.Template:SfnTemplate:SfnTemplate:Sfn

Prior to Frisch and Waugh's result, much of the debate around the optimal time trend adjustment concerned estimates of static demand equations whose observations had been taken over time. Because the theoretical demand equations did not depend on time, economists sought to adjust for the effect of time trends in order to bring the statistical model closer in line with theory. Advocates of including time trends in regressions argued it improved the model's fit, where opponents argued that time trends may violate ceteris paribus assumptions of the underlying theoretical model.Template:Sfn In proving the equivalence of the two methods, addressing the difference in model fit, and formalizing the distinction between estimated coefficients and theoretical models, Frisch and Waugh's paper resolved the debate around trend adjustments.Template:Sfn

Economist and historian Mary S. Morgan contextualizes Frisch and Waugh's result, as it pertains to a greater understanding of regression coefficients, as having "paved the way for a more generous use of the other factors in the demand equation."Template:Sfn Their results were extended to the instrumental variables estimator by Reirsøl, a former research assistant of Frisch, in 1945.Template:Sfn In 1952, Frisch and Waugh's results were applied by Gerhard Tintner to polynomial trend adjustment, an extension from the 1933 paper's focus on linear trend adjustment.Template:SfnTemplate:Sfn In a 1953 textbook on demand analysis, econometrician Herman Wold references Frisch and Waugh's paper as a special case applied to trend adjustments.Template:Sfn

Generalization and later development

In 1963, econometrician Michael C. Lovell generalized Frisch and Waugh's results, providing a proof of the theorem using projection matrices and applying it to seasonality adjustments.Template:SfnTemplate:SfnTemplate:Sfn Rather than focusing on certain types of variables, as Frisch and Waugh did with time trends, Lovell proves the result with arbitrary sets of explanatory variables.Template:SfnTemplate:Sfn Lovell presents 7 regression specifications and proves how their coefficients relate, among them both the residual and double residual forms of the theorem.Template:Sfn Among Lovell's results was the equivalence of the residual vector between the full and double residual regressions, which had not been proved in previous works.Template:Sfn Lovell published an additional proof in 2008 using only simple algebra.Template:Sfn

In 1964, economist Richard Stone published a generalized proof of the theorem using inverses of partitioned matrices. Unlike Lovell's paper, Stone's two-page note did not discuss residual vectors or other implications of the theorem.Template:SfnTemplate:Sfn

The Frisch–Waugh–Lovell theorem is included in most intermediate to advanced econometrics textbooks, and most treatments follow Lovell's framing.Template:SfnTemplate:Sfn

Naming

The theorem has been referred to under a number of names, including the Yule–Frisch–Waugh–Lovell theorem, Frisch–Waugh–Lovell theorem, Frisch-Waugh theorem, partitioned regression theorem, residual regression, and the regression anatomy theorem.Template:SfnTemplate:SfnTemplate:Sfn

While Frisch and Waugh's paper was not the first introduction of the result, it was the first proof in econometrics.Template:SfnTemplate:Sfn Recognizing the generalization by Lovell, the theorem was presented as the Frisch–Waugh–Lovell theorem in a 1993 econometrics textbook by Russell Davidson and James G. MacKinnon.Template:Sfn

Extensions

Where the Frisch–Waugh–Lovell theorem states that the full and residual regressions have the same coefficients, relationships between the coefficients' standard errors can also be shown. Lovell's 1963 paper finds that the homoskedastic standard errors of coefficients in the double residual regression differ from those of the full regression by a degrees of freedom adjustment.Template:Sfn In 2021, statistician Peng Ding presented a proof of Lovell's results and found comparable results for other estimates of standard errors, including heteroskedasticity-consistent and clustered standard errors.Template:SfnTemplate:Sfn

Analogues to the Frisch–Waugh–Lovell theorem have been shown for a number of other estimators, including generalized least squares,Template:Sfn ridge regression and the LASSO,Template:Sfn and k-class estimators, including limited information maximum likelihood.Template:Sfn

See also

• Helmert–Wolf blocking
• Gram–Schmidt process

References

Journal articles

• Template:Cite journal
• Template:Cite journal
• Template:Cite journal
• Template:Cite journal
• Template:Cite journal
• Template:Cite journal
• Template:Cite journal
• Template:Cite journal
• Template:Cite journal
• Template:Cite journal
• Template:Cite journal
• Template:Cite journal

Books

• Template:Cite book
• Template:Cite book
• Template:Cite book
• Template:Cite book
• Template:Cite book
• Template:Cite book
• Template:Cite book
• Template:Cite book
• Template:Cite book
• Template:Cite book
• Template:Cite book