Book contents
- Trustworthy Online Controlled Experiments
- Reviews
- Trustworthy Online Controlled Experiments
- Copyright page
- Contents
- Preface
- Acknowledgments
- Part I Introductory Topics for Everyone
- Part II Selected Topics for Everyone
- Part III Complementary and Alternative Techniques to Controlled Experiments
- Part IV Advanced Topics for Building an Experimentation Platform
- Part V Advanced Topics for Analyzing Experiments
- 17 The Statistics behind Online Controlled Experiments
- 18 Variance Estimation and Improved Sensitivity: Pitfalls and Solutions
- 19 The A/A Test
- 20 Triggering for Improved Sensitivity
- 21 Sample Ratio Mismatch and Other Trust-Related Guardrail Metrics
- 22 Leakage and Interference between Variants
- 23 Measuring Long-Term Treatment Effects
- References
- Index
18 - Variance Estimation and Improved Sensitivity: Pitfalls and Solutions
from Part V - Advanced Topics for Analyzing Experiments
Published online by Cambridge University Press: 13 March 2020
- Trustworthy Online Controlled Experiments
- Reviews
- Trustworthy Online Controlled Experiments
- Copyright page
- Contents
- Preface
- Acknowledgments
- Part I Introductory Topics for Everyone
- Part II Selected Topics for Everyone
- Part III Complementary and Alternative Techniques to Controlled Experiments
- Part IV Advanced Topics for Building an Experimentation Platform
- Part V Advanced Topics for Analyzing Experiments
- 17 The Statistics behind Online Controlled Experiments
- 18 Variance Estimation and Improved Sensitivity: Pitfalls and Solutions
- 19 The A/A Test
- 20 Triggering for Improved Sensitivity
- 21 Sample Ratio Mismatch and Other Trust-Related Guardrail Metrics
- 22 Leakage and Interference between Variants
- 23 Measuring Long-Term Treatment Effects
- References
- Index
Summary
Why you care: What is the point of running an experiment if you cannot analyze it in a trustworthy way? Variance is the core of experiment analysis. Almost all the key statistical concepts we have introduced are related to variance, such as statistical significance, p-value, power, and confidence interval. It is imperative to not only correctly estimate variance, but also to understand how to achieve variance reduction to gain sensitivity of the statistical hypothesis tests.
- Type
- Chapter
- Information
- Trustworthy Online Controlled ExperimentsA Practical Guide to A/B Testing, pp. 193 - 199Publisher: Cambridge University PressPrint publication year: 2020