View source for Holm–Bonferroni method

{{Short description|Statistical method}}
In [[statistics]], the '''Holm&ndash;Bonferroni method''',<ref>{{cite journal
|last=Holm |first=S.
|year=1979
|title=A simple sequentially rejective multiple test procedure
|journal=Scandinavian Journal of Statistics
|volume=6 |issue=2 |pages=65&ndash;70
|mr=538597 | jstor = 4615733
}}</ref> also called the '''Holm method''' or '''Bonferroni–Holm method''', is used to counteract the problem of [[multiple comparisons]]. It is intended to control the [[family-wise error rate]] (FWER) and offers a simple test [[uniformly more powerful]] than the [[Bonferroni correction]]. It is named after [[Sture Holm]], who codified the method, and [[Carlo Emilio Bonferroni]].

==Motivation==
When considering several hypotheses, the problem of [[Multiple comparisons#The problem|multiplicity]] arises: the more hypotheses are tested, the higher the probability of obtaining [[Type I error]]s ([[false positive]]s). The Holm–Bonferroni method is one of many approaches for controlling the FWER, i.e., the probability that one or more Type I errors will occur, by adjusting the rejection criterion for each of the individual hypotheses.{{citation needed|date=March 2020}}

==Formulation==
The method is as follows:
* Suppose you have <math>m</math> [[p-value]]s, sorted into order lowest-to-highest <math>P_1,\ldots,P_m</math>, and their corresponding hypotheses <math>H_1,\ldots,H_m</math>(null hypotheses). You want the FWER to be no higher than a certain pre-specified [[significance level]] <math> \alpha </math>.
* Is <math>P_1 \leq \alpha / m</math>? If so, reject <math>H_1</math> and continue to the next step, otherwise EXIT.
* Is <math>P_2 \leq \alpha / (m-1)</math>? If so, reject <math>H_2</math> also, and continue to the next step, otherwise EXIT.
* And so on: for each P value, test whether <math>P_k \leq \frac{\alpha}{m+1-k}</math>. If so, reject <math>H_k</math> and continue to examine the larger P values, otherwise EXIT.

This method ensures that the FWER is at most <math>\alpha</math>, in the strong sense.

===Rationale===
The simple [[Bonferroni correction]] rejects only null hypotheses with ''p''-value less than or equal to <math>\frac{\alpha}{m}</math>, in order to ensure that the FWER, i.e., the risk of rejecting one or more true null hypotheses (i.e., of committing one or more type I errors) is at most <math>\alpha</math>. The cost of this protection against type I errors is an increased risk of failing to reject one or more false null hypotheses (i.e., of committing one or more type II errors).

The Holm–Bonferroni method also controls the FWER at <math>\alpha</math>, but with a lower increase of type II error risk than the classical Bonferroni method. The Holm–Bonferroni method sorts the ''p''-values from lowest to highest and compares them to nominal alpha levels of <math>\frac{\alpha}{m}</math> to <math>\alpha</math> (respectively), namely the values <math>\frac{\alpha}{m},  \frac{\alpha}{m-1}, \ldots ,  \frac{\alpha}{2},  \frac{\alpha}{1}</math>.
* The index <math>k</math> identifies the first ''p''-value that is ''not'' low enough to validate rejection. Therefore, the null hypotheses <math>H_{(1)}, \ldots , H_{(k-1)}</math> are rejected, while the null hypotheses <math> H_{(k)}, ... , H_{(m)}</math> are not rejected. 
* If <math>k = 1</math> then no ''p''-values were low enough for rejection, therefore no null hypotheses are rejected.
* If no such index <math>k</math> could be found then all ''p''-values were low enough for rejection, therefore all null hypotheses are rejected (none are accepted).

===Proof===
Let <math>H_{(1)}\ldots H_{(m)}</math> be the family of hypotheses sorted by their p-values <math>P_{(1)}\leq P_{(2)}\leq\cdots\leq P_{(m)}</math>. Let <math>I_0</math> be the set of indices corresponding to the (unknown) true null hypotheses, having <math>m_0</math> members.

<blockquote>
'''Claim''': If we wrongly reject some true hypothesis, there is a true hypothesis <math>H_{(\ell)}</math> for which <math>P_{(\ell)}</math> at most <math>\frac{\alpha}{m_0}</math>.

First note that, in this case, there is at least one true hypothesis, so <math>m_0 \geq 1</math>. Let <math>\ell</math> be such that <math>H_{(\ell)}</math> is the first rejected true hypothesis. Then <math>H_{(1)}, \ldots, H_{(\ell-1)}</math>are all rejected false hypotheses. It follows that <math>\ell-1 \leq m - m_0</math> and, hence, <math>\frac{1}{m-\ell+1}\leq \frac{1}{m_0}</math> (1). Since <math>H_{(\ell)}</math> is rejected, it must be <math>P_{(\ell)} \leq \frac{\alpha}{m-\ell+1}</math> by definition of the testing procedure. Using (1), we conclude that <math>P_{(\ell)} \leq \frac{\alpha}{m_0}</math>, as desired.
</blockquote>

So let us define the random event <math>A= \bigcup_{i \in I_0} \left\{ P_i \leq \frac{\alpha}{m_0}\right\}</math>. Note that, for <math>i \in I_o</math>, since <math>H_i</math> is a true null hypothesis, we have that <math>P\left(\left\{ P_i \leq \frac{\alpha}{m_0}\right\}\right) = \frac{\alpha}{m_0}</math>. Subadditivity of the probability measure implies that <math>\Pr(A)\leq\sum_{i\in I_0}P\left(\left\{ P_i \leq \frac{\alpha}{m_0}\right\}\right) = \sum_{i \in I_0} \frac{\alpha}{m_0} =  \alpha</math>. Therefore, the probability to reject a true hypothesis is at most <math>\alpha</math>.

===Alternative proof===
The Holm–Bonferroni method can be viewed as a [[closed testing procedure]],<ref>{{cite journal |last=Marcus |first=R. |last2=Peritz |first2=E. |last3=Gabriel |first3=K. R. |year=1976 |title=On closed testing procedures with special reference to ordered analysis of variance |journal=[[Biometrika]] |volume=63 |issue=3 |pages=655–660 |doi=10.1093/biomet/63.3.655 }}</ref> with the Bonferroni correction applied locally on each of the intersections of null hypotheses.

The closure principle states that a hypothesis <math>H_i</math> in a family of hypotheses <math>H_1,\ldots,H_m</math> is rejected – while controlling the FWER at level <math>\alpha</math> – if and only if all the sub-families of the intersections with <math>H_i</math> are rejected at level <math>\alpha</math>.

The Holm–Bonferroni method is a ''shortcut procedure'', since it makes <math>m</math> or less comparisons, while the number of all intersections of null hypotheses to be tested is of order <math>2^m</math>.
It controls the FWER in the strong sense.

In the Holm–Bonferroni procedure, we first test <math>H_{(1)}</math>. If it is not rejected then the intersection of all null hypotheses <math>\bigcap\nolimits_{i = 1}^m H_i</math> is not rejected too, such that there exists at least one intersection hypothesis for each of elementary hypotheses <math>H_1,\ldots,H_m</math> that is not rejected, thus we reject none of the elementary hypotheses.

If <math>H_{(1)}</math> is rejected at level <math>\alpha/m</math> then all the intersection sub-families that contain it are rejected too, thus <math>H_{(1)}</math> is rejected.
This is because <math>P_{(1)}</math> is the smallest in each one of the intersection sub-families and the size of the sub-families is at most <math>m</math>, such that the Bonferroni threshold larger than <math>\alpha/m</math>.

The same rationale applies for <math>H_{(2)}</math>. However, since <math>H_{(1)}</math> already rejected, it sufficient to reject all the intersection sub-families of <math>H_{(2)}</math> without <math>H_{(1)}</math>. Once <math>P_{(2)}\leq\alpha/(m-1)</math> holds all the intersections that contains <math>H_{(2)}</math> are rejected.

The same applies for each <math>1\leq i \leq m</math>.

== Example ==
Consider four null hypotheses <math>H_1,\ldots,H_4</math> with unadjusted p-values <math>p_1=0.01</math>, <math>p_2=0.04</math>, <math>p_3=0.03</math> and <math>p_4=0.005</math>, to be tested at significance level <math>\alpha=0.05</math>. Since the procedure is step-down, we first test <math>H_4=H_{(1)}</math>, which has the smallest p-value <math>p_4=p_{(1)}=0.005</math>. The p-value is compared to <math>\alpha/4=0.0125</math>, the null hypothesis is rejected and we continue to the next one. Since <math>p_1=p_{(2)}=0.01<0.0167=\alpha/3</math> we reject <math>H_1=H_{(2)}</math> as well and continue. The next hypothesis <math>H_3</math> is not rejected since <math>p_3=p_{(3)}=0.03 > 0.025=\alpha/2</math>. We stop testing and conclude that <math>H_1</math> and <math>H_4</math> are rejected and  <math>H_2</math> and <math>H_3</math> are not rejected while controlling the family-wise error rate at level <math>\alpha=0.05</math>. Note that even though <math>p_2=p_{(4)}=0.04 < 0.05=\alpha</math> applies, <math>H_2</math> is '''not''' rejected. This is because the testing procedure stops once a failure to reject occurs.

==Extensions==

===Holm–Šidák method===
{{further|Šidák correction}}
When the hypothesis tests are not negatively dependent, it is possible to replace <math>\frac{\alpha}{m},\frac{\alpha}{m-1}, \ldots, \frac{\alpha}{1}</math> with:
: <math>1-(1-\alpha)^{1/m},1-(1-\alpha)^{1/(m-1)},\ldots,1-(1-\alpha)^{1}</math>
resulting in a slightly more powerful test.

===Weighted version===
Let <math>P_{(1)},\ldots,P_{(m)}</math> be the ordered unadjusted p-values. Let <math>H_{(i)}</math>, <math>0\leq w_{(i)}</math> correspond to <math>P_{(i)}</math>. Reject <math>H_{(i)}</math> as long as

: <math>P_{(j)}\leq \frac{w_{(j)}}{\sum^m_{k=j} w_{(k)}} \alpha,\quad j=1,\ldots,i</math>

===Adjusted ''p''-values===
The adjusted [[P-values|''p''-values]] for Holm–Bonferroni method are:
: <math>\widetilde{p}_{(i)}=\max_{j\leq i}\left\{ (m-j+1)p_{(j)}\right\}_1, \text{ where } \{x\}_1 \equiv \min(x,1).</math>

In the earlier example, the adjusted ''p''-values are <math>\widetilde{p}_1 = 0.03</math>, <math>\widetilde{p}_2 = 0.06</math>, <math>\widetilde{p}_3 = 0.06</math> and <math>\widetilde{p}_4 = 0.02</math>. Only hypotheses <math>H_1</math> and <math>H_4</math> are rejected at level <math>\alpha=0.05</math>.

Similar adjusted ''p''-values for Holm-Šidák method can be defined recursively as <math>\widetilde{p}_{(i)}=\max\left\{\widetilde{p}_{(i-1)}, 1 - (1-p_{(i)})^{m-i+1} \right\}</math>, where <math>\widetilde{p}_{(1)} = 1 - (1 - p_{(1)})^m</math>. Due to the inequality <math>1 - (1 - \alpha)^{1/n} < \alpha/n</math> for <math>n \geq 2</math>, the Holm-Šidák method will be more powerful than Holm–Bonferroni method.

The weighted adjusted ''p''-values are:{{citation needed|date=June 2012}}
:<math>\widetilde{p}_{(i)}=\max_{j\leq i} \left\{\frac{\sum^m_{k=j}{w_{(k)}}}{w_{(j)}} p_{(j)}\right\}_1, \text{ where } \{x\}_1 \equiv \min(x,1).</math>
A hypothesis is rejected at level α if and only if its adjusted ''p''-value is less than α. In the earlier example using equal weights, the adjusted ''p''-values are 0.03, 0.06, 0.06, and 0.02. This is another way to see that using α = 0.05, only hypotheses one and four are rejected by this procedure.

==Alternatives and usage==
{{main|Family-wise error rate#Controlling procedures}}
The Holm–Bonferroni method is "uniformly" more powerful than the classic [[Bonferroni correction]], meaning that it is always at least as powerful.

There are other methods for controlling the FWER that are more powerful than Holm–Bonferroni. For instance, in the [[Family-wise error rate#Hochberg.27s step-up procedure|Hochberg procedure]], rejection of <math>H_{(1)} \ldots H_{(k)}</math> is made after finding the ''maximal'' index <math>k</math> such that <math>P_{(k)} \leq \frac{\alpha}{m+1-k}</math>. Thus, The Hochberg procedure is uniformly more powerful than the Holm procedure. However, the Hochberg procedure requires the hypotheses to be [[Independence (probability theory)|independent]] or under certain forms of positive dependence, whereas Holm–Bonferroni can be applied without such assumptions. A similar step-up procedure is the Hommel procedure, which is uniformly more powerful than the Hochberg procedure.<ref name="Hommel1988">{{cite journal|last1=Hommel|first1=G.|title=A stagewise rejective multiple test procedure based on a modified Bonferroni test|journal=Biometrika|volume=75|issue=2|year=1988|pages=383–386|issn=0006-3444|doi=10.1093/biomet/75.2.383|hdl=2027.42/149272|hdl-access=free}}</ref>

==Naming==
Carlo Emilio Bonferroni did not take part in inventing the method described here. Holm originally called the method the "sequentially rejective Bonferroni test", and it became known as Holm–Bonferroni only after some time. Holm's motives for naming his method after Bonferroni are explained in the original paper: "The use of the Boole inequality within multiple inference theory is usually called the Bonferroni technique, and for this reason we will call our test the sequentially rejective Bonferroni test."

==References==
<references/>

{{DEFAULTSORT:Holm-Bonferroni Method}}
[[Category:Statistical hypothesis testing]]
[[Category:Statistical tests]]
[[Category:Multiple comparisons]]