Multiple comparisons
Two different meanings of a false-positive target
Simulate 20 independent tests: six effects are real and fourteen nulls are true. Because this is a simulation, we can check each discovery against the truth.
Compare Benjamini–Hochberg with Bonferroni on the same simulated tests. Read both the chance of any false discovery (FWER) and the average false fraction among discoveries (FDR).
Compare false-positive targets
The first bar gives the calculated probability of at least one false rejection without adjustment. The second gives the proportion of simulated families with a false rejection under your chosen method. The vertical axis runs from 0 to 1.
Correct and false discoveries
Bars count families by their number of false or correct discoveries. A false discovery rejects a true null. A correct discovery detects one of the simulated real effects.
Worked comparison
Follow the calculation
What the symbols mean
For m independent tests of true null hypotheses, each at significance level α, FWER is the chance of at least one false rejection without adjustment.
Assumptions and full calculation
With the real-effect proportion set to zero, every null hypothesis is true. Null p-values are independent uniform values. Non-null p-values come from independent two-sided known-SD z tests at the chosen true effect and sample size. This represents continuous tests whose null p-values follow that distribution. The family-wise error rate, or FWER, is the chance of any false rejection in a family. Without adjustment it is 1 − (1 − α)ᵐ. Bonferroni and Holm control FWER even for dependent tests. Benjamini-Hochberg controls the false discovery rate, or FDR, under independence or suitable positive dependence. FDR is the expected proportion of false rejections among all rejections, counting the proportion as zero if there are none. With every null true, FDR equals FWER. With real effects, the simulation counts true and false discoveries separately and estimates FDR by averaging the false-discovery proportion across all simulated families. All tests are independent.