Book contents
- Frontmatter
- Contents
- Preface
- 1 Introduction
- 2 Theory of Tests, p-Values, and Confidence Intervals
- 3 From Scientific Theory to Statistical Hypothesis Test
- 4 One-Sample Studies with Binary Responses
- 5 One-Sample Studies with Ordinal or Numeric Responses
- 6 Paired Data
- 7 Two-Sample Studies with Binary Responses
- 8 Assumptions and Hypothesis Tests
- 9 Two-Sample Studies with Ordinal or Numeric Responses
- 10 General Methods for Frequentist Inferences
- 11 k-Sample Studies and Trend Tests
- 12 Clustering and Stratification
- 13 Multiplicity in Testing
- 14 Testing from Models
- 15 Causality
- 16 Censoring
- 17 Missing Data
- 18 Group Sequential and Related Adaptive Methods
- 19 Testing Fit, Equivalence, and Noninferiority
- 20 Power and Sample Size
- 21 Bayesian Hypothesis Testing
- References
- Notation Index
- Concept Index
13 - Multiplicity in Testing
Published online by Cambridge University Press: 17 April 2022
- Frontmatter
- Contents
- Preface
- 1 Introduction
- 2 Theory of Tests, p-Values, and Confidence Intervals
- 3 From Scientific Theory to Statistical Hypothesis Test
- 4 One-Sample Studies with Binary Responses
- 5 One-Sample Studies with Ordinal or Numeric Responses
- 6 Paired Data
- 7 Two-Sample Studies with Binary Responses
- 8 Assumptions and Hypothesis Tests
- 9 Two-Sample Studies with Ordinal or Numeric Responses
- 10 General Methods for Frequentist Inferences
- 11 k-Sample Studies and Trend Tests
- 12 Clustering and Stratification
- 13 Multiplicity in Testing
- 14 Testing from Models
- 15 Causality
- 16 Censoring
- 17 Missing Data
- 18 Group Sequential and Related Adaptive Methods
- 19 Testing Fit, Equivalence, and Noninferiority
- 20 Power and Sample Size
- 21 Bayesian Hypothesis Testing
- References
- Notation Index
- Concept Index
Summary
This chapter addresses multiplicity in testing, the problem that if many hypotheses are tested then unless some multiplicity adjustment is made, the probability of falsely rejecting at least one hypothesis can be unduly high. We define familywise error rate (FWER), the probability that at least one null hypothesis in the family of hypotheses is rejected. We discuss which sets of hypotheses should be grouped into families. We define the false discovery rate (FDR). We describe simple adjustments based only on p-values of the hypotheses in the family, such the Bonferroni, Holm, and Hochberg procedures for FWER control and the Benjamini–Hochberg adjustment for FDR control. We discuss max-t type inferences for controlling FWER in linear models, or other models with asymptotically normal estimators. We describe resampling-based multiplicity adjustments. We demonstrate graphical methods, showing, for example, gatekeeping and fallback methods, and allowing for more complicated methods. We briefly present logical constraints for hypotheses and the theoretically important closure method.
- Type
- Chapter
- Information
- Statistical Hypothesis Testing in ContextReproducibility, Inference, and Science, pp. 235 - 252Publisher: Cambridge University PressPrint publication year: 2022