Book contents
- Frontmatter
- Contents
- Figures
- Preface
- 1 Introduction to Observational Methods
- 2 Coding Schemes and Observational Measurement
- 3 Recording Observational Data
- 4 Representing Observational Data
- 5 Observer Agreement and Cohen’s Kappa
- 6 Kappas for Point-by-Point Agreement
- 7 The Intraclass Correlation Coefficient (ICC) for Summary Measures
- 8 Summary Statistics for Individual Codes
- 9 Cell and Summary Statistics for Contingency Tables
- 10 Preparing for Sequential and Other Analyses
- 11 Time-Window and Log-Linear Sequential Analysis
- 12 Recurrence Analysis and Permutation Tests
- Epilogue
- Appendix A Expected Values for Kappa Comparing Two Observers
- Appendix B Expected Values for Kappa Comparing with a Gold Standard
- References
- Index
7 - The Intraclass Correlation Coefficient (ICC) for Summary Measures
Published online by Cambridge University Press: 05 June 2012
- Frontmatter
- Contents
- Figures
- Preface
- 1 Introduction to Observational Methods
- 2 Coding Schemes and Observational Measurement
- 3 Recording Observational Data
- 4 Representing Observational Data
- 5 Observer Agreement and Cohen’s Kappa
- 6 Kappas for Point-by-Point Agreement
- 7 The Intraclass Correlation Coefficient (ICC) for Summary Measures
- 8 Summary Statistics for Individual Codes
- 9 Cell and Summary Statistics for Contingency Tables
- 10 Preparing for Sequential and Other Analyses
- 11 Time-Window and Log-Linear Sequential Analysis
- 12 Recurrence Analysis and Permutation Tests
- Epilogue
- Appendix A Expected Values for Kappa Comparing Two Observers
- Appendix B Expected Values for Kappa Comparing with a Gold Standard
- References
- Index
Summary
As noted early in Chapter 5, point-by-point agreement is often accepted as evidence that summary measures derived from sequential data will be reliable, probably because point-by-point agreement seems the more stringent approach. Kappa-based demonstrations of observer agreement are essential for observer training and for checking throughout coding; but once data collection is complete and summary measures have been derived from the sequential data, their reliability can be assessed directly with an intraclass correlation coefficient (ICC).
Relative Versus Absolute Agreement
Summary measures in observational research are typically of two kinds: first are frequencies, rates, and proportions for individual codes (see Chapter 8); and second are contingency indices and other contingency table statistics involving two or more codes (see Chapter 9). Both are continuous (i.e., measured on an interval or ratio scale), and so an ICC measure of reliability, which requires continuous or arguably ordinal measurement, is appropriate. The general question is: When values are derived from records of the same events coded independently by different observers, do the observers agree sufficiently for our purposes? Can we regard the observers as essentially interchangeable? In this regard, one important distinction concerns relative, as opposed to absolute, judgment (i.e., relative consistency vs. absolute agreement, or norm-referenced vs. criterion-referenced; see Suen, 1988). Do we want scores derived from two observers’ records to be rank-ordered the same way – that is, to be in relative agreement – or do we want them to agree regarding absolute magnitude?
- Type
- Chapter
- Information
- Publisher: Cambridge University PressPrint publication year: 2011