Book contents
- Trustworthy Online Controlled Experiments
- Reviews
- Trustworthy Online Controlled Experiments
- Copyright page
- Contents
- Preface
- Acknowledgments
- Part I Introductory Topics for Everyone
- Part II Selected Topics for Everyone
- 5 Speed Matters
- 6 Organizational Metrics
- 7 Metrics for Experimentation and the Overall Evaluation Criterion
- 8 Institutional Memory and Meta-Analysis
- 9 Ethics in Controlled Experiments
- Part III Complementary and Alternative Techniques to Controlled Experiments
- Part IV Advanced Topics for Building an Experimentation Platform
- Part V Advanced Topics for Analyzing Experiments
- References
- Index
7 - Metrics for Experimentation and the Overall Evaluation Criterion
from Part II - Selected Topics for Everyone
Published online by Cambridge University Press: 13 March 2020
- Trustworthy Online Controlled Experiments
- Reviews
- Trustworthy Online Controlled Experiments
- Copyright page
- Contents
- Preface
- Acknowledgments
- Part I Introductory Topics for Everyone
- Part II Selected Topics for Everyone
- 5 Speed Matters
- 6 Organizational Metrics
- 7 Metrics for Experimentation and the Overall Evaluation Criterion
- 8 Institutional Memory and Meta-Analysis
- 9 Ethics in Controlled Experiments
- Part III Complementary and Alternative Techniques to Controlled Experiments
- Part IV Advanced Topics for Building an Experimentation Platform
- Part V Advanced Topics for Analyzing Experiments
- References
- Index
Summary
Why you care: To design and run a good online controlled experiment, you need metrics that meet certain characteristics. They must be measurable in the short term (experiment duration) and computable, as well as sufficiently sensitive and timely to be useful for experimentation. If you use multiple metrics to measure success for an experiment, ideally you may want to combine them into an Overall Evaluation Criterion (OEC), which is believed to causally impact long-term objectives. It often requires multiple iterations to adjust and refine the OEC, but as the quotation above, by Eliyahu Goldratt, highlights, it provides a clear alignment mechanism to the organization.
- Type
- Chapter
- Information
- Trustworthy Online Controlled ExperimentsA Practical Guide to A/B Testing, pp. 102 - 110Publisher: Cambridge University PressPrint publication year: 2020