Mfcttrf

From the Wikipedia entry:

Suppose there are k hypotheses $H_1,..., H_k$ to be tested and the overall
type I error rate is $alpha$. The closed testing principle allows the
rejection of any one of these elementary hypotheses, say $H_i$, if all
possible intersection hypotheses involving $H_i$ can be rejected by using
valid local level $alpha$ tests. It controls the familywise error rate for
all the k hypotheses at level α in the strong sense.

By using the closed testing principle, we ensure all the false positive rate of all hypothesis tests in a family will be controlled at $alpha$. This seems too good to be true. Why do we bother with Bonferonni or other corrections for multiple comparisons, when we can just test all the intersection hypotheses? The only downside seems to be that you need to conduct $2^n$ tests.

EDIT: I am thinking about a relatively small number of individual hypothesis tests, say $n = 3 textor 5$. Then you would have a reasonable number of tests.

It is actually used quite often (in fact, Bonferroni and Bonferroni-Holm can be shown to be valid tests using the closed testing principle - see below). Part of the reason why some simple procedures like Bonferroni are still so popular is of course, that they are very easy to implement and it is easy to communicate what you did.

E.g. even for the simple Bonferroni or Bonferroni-Holm writing down the tests for the intersection null hypotheses is a bit tedious. That's why the graphical approach to constructing closed testing procedures is very popular - at least in the clinical trials setting (where there is usually a trained statistician at hand that can help pre-specify a tailored testing procedure for a specific trial). This graphical approach provides a really easy way to construct valid testing procedures and makes communication easier. Various variants, extensions and improvements such as exploiting correlations, prioritization of certain hypotheses depending on which hypotheses are already rejected etc. have been proposed over the years.

Bonferroni vs. Bonferroni-Holm as closed testing procedures example

The Bonferroni test can be easily written down in terms of the closed testing principle, e.g. in the case of two null hypotheses as

• Test for $H_1 cap H_2$: $p_1 leq alpha/2$ and $p_2 leq alpha/2$
  


• Test for elementary null hypothesis $H_1$: $p_1 leq alpha/2$
  


• Test for elementary null hypothesis $H_2$: $p_2 leq alpha/2$
  

with $H_1$ rejected overall, if both $H_1$ and $H_1 cap H_2$ are rejected. Thus, we know it is a valid test. You also immediately see that for the elementary null hypotheses $H_1$ and $H_2$ you are using a test that is not exhausting the significance level and that the Bonferroni-Holm test

• Test for $H_1$ and $H_2$: $p_1 leq alpha/2$ and $p_2 leq alpha/2$
  


• Test for $H_1$: $p_1 leq alpha$
  


• Test for $H_2$: $p_2 leq alpha$
  

is uniformly more powerful.

Figure 1: Illustration of the graphical approach for displaying these two procedures

Power considerations and other considerations

Usually, there's no downside in terms of power to using a closed testing procedure - if anything it tends to be easier to make sure the test fully exhausts the significance level. It can sometimes be a bit tricky to build in some desired features (e.g. exploiting a known correlation) while keep some other desired contraints (e.g. that the closed testing procedure can be easily displayed graphically due to the consistency requirement on the testing procedure).