Hostname: page-component-669899f699-ggqkh Total loading time: 0 Render date: 2025-04-26T00:48:05.513Z Has data issue: false hasContentIssue false
  • English
  • Français

Observer reliability for working equine welfare assessment: problems with high prevalences of certain results

Published online by Cambridge University Press:  01 January 2023

CC Burn ,
JC Pritchard  and
HR Whay
CC Burn*
Affiliation:
University of Bristol, Department of Clinical Veterinary Science, Langford, Bristol BS40 5DU, UK
JC Pritchard
Affiliation:
University of Bristol, Department of Clinical Veterinary Science, Langford, Bristol BS40 5DU, UK The Brooke Hospital for Animals, Broadmead House, 21 Panton Street, London SW1Y 4DR, UK
HR Whay
Affiliation:
University of Bristol, Department of Clinical Veterinary Science, Langford, Bristol BS40 5DU, UK
*
* Contact for correspondence and requests for reprints: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

Welfare issues relevant to equids working in developing countries may differ greatly to those of sport and companion equids in developed countries. In this study, we test the observer reliability of a working equine welfare assessment, demonstrating how prevalence of certain observations reduces reliability ratings. The assessment included behaviour, general health, wounds, and limb and foot pathologies. In Study 1, agreement between five observers and their trainer (the ‘gold standard’) was assessed using 80 horses and 80 donkeys in India. Intra-observer agreement was later tested on 40 of each species. Study 2 took place in Egypt, using nine observers, their trainer, 30 horses and 30 donkeys, adjusting some scoring systems and providing observers with more detailed guidelines than in Study 1. Percentage agreements, Fleiss kappa (with a weighted version for ordinal scores) and prevalence indices were calculated for each variable. Reliability was similar across both studies, but was significantly poorer for donkeys than horses. Age, sex, certain wounds and (for horses alone) body condition, consistently attained clinically-useful reliability. Hoofhorn quality, point-of-hock lesions, mucous membrane abnormalities, limb-tether lesions, and skin tenting showed poor reliability. Reporting the prevalence index alongside the percentage agreement showed that, for many variables, the populations were too homogenous for conclusive reliability ratings. Suggestions are made for improving scoring systems showing poor reliability, but future testing will require deliberate selection of a more diverse equine population. This could prove challenging given that, in both populations of horses and donkeys studied here, many pathologies apparently showed 90-100% prevalence.

Keywords

animal welfaredonkeyshorsesmethodologyobserver agreementprevalence
Type
Research Article
Information
Animal Welfare , Volume 18 , Issue 2 , May 2009 , pp. 177 - 187
Copyright
© 2009 Universities Federation for Animal Welfare

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Broster, CE, Burn, CC, Barr, ARS and Whay, HR 2009 The range and prevalence of pathological abnormalities associated with lameness in working horses from developing countries. The Equine Veterinary Journal, in pressCrossRefGoogle Scholar
Burn, CC, Pritchard, JC, Farajat, M, Twaissi, AAM and Whay, HR 2008 Risk factors for strap-related lesions in working donkeys at the World Heritage site of Petra in Jordan. The Veterinary Journal 178: 261–269CrossRefGoogle ScholarPubMed
Burn, CC and Weir, AAS Using prevalence indices to aid interpretation and comparison of agreement ratings between two or more observers. The Veterinary Journal, submittedGoogle Scholar
Byrt, T, Bishop, J and Carlin, JB 1993 Bias, prevalence and kappa. Journal of Clinical Epidemiology 46: 423–429CrossRefGoogle ScholarPubMed
FAOSTAT 2005 FAO statistical database website. Food and Agricultural Organisation of the United Nations. http://faostat.fao.org/site/409/default.aspx. Date accessed: 7 July 2006.Google Scholar
Hoehler, FK 2000 Bias and prevalence effects on kappa viewed in terms of sensitivity and specificity. Journal of Clinical Epidemiology 53: 499–503CrossRefGoogle ScholarPubMed
Johnsen, PF, Johannesson, T and Sandøe, P 2001 Assessment of farm animal welfare at herd level: Many goals, many methods. Acta Agriculturae Scandinavica Section A, Animal Science S30: 26–33Google Scholar
Kraemer, HC, Periyakoil, VS and Noda, A 2004 Agreement Statistics. Kappa coefficients in medical research. In: D’Agostino, RB (ed) Tutorials in Biostatistics Volume 1: Statistical Methods in Clinical Studies pp 85–105. John Wiley & Sons, Ltd: Queensland, AustraliaGoogle Scholar
Landis, JR and Koch, GG 1977 The measurement of observer agreement for categorical data. Biometrics 33: 159–174CrossRefGoogle ScholarPubMed
Lindberg, AC, Leeb, C, Pritchard, JC, Whay, HR and Main, DCJ 2004 Determination of welfare problems and their perceived causes in working equines. Animal Welfare 13: S247Google Scholar
Maclure, M and Willett, WC 1987 Misinterpretation and misuse of the kappa statistic. American Journal of Epidemiology 126: 161–169CrossRefGoogle ScholarPubMed
Main, DCJ, Whay, HR, Leeb, C and Webster, AJF 2007 Formal animal-based welfare assessment in UK certification schemes. Animal Welfare 16: 233–236Google Scholar
Maranhão, RPA, Palhares, MS, Melo, UP, Rezende, HHC, Braga, CE, Silva Filho, JM and Vasconcelos, MNF 2006 Most frequent pathologies of the locomotor system in equids used for wagon traction in Belo Horizonte. Arquivo Brasileiro de Medicina Veterinária e Zootecnia 58: 21–27CrossRefGoogle Scholar
Pearson, RA and Ouassat, M 1996 Estimation of the liveweight and body condition of working donkeys in Morocco. Veterinary Record 138: 229–233CrossRefGoogle ScholarPubMed
Pritchard, JC and Whay, HR 2003 Guidance notes to accompany working equine welfare assessment. University of Bristol: Bristol, UK, unpublishedGoogle Scholar
Pritchard, JC and Whay, HR 2004 Guidance notes to accompany working equine welfare assessment. University of Bristol: Bristol, UK, unpublishedGoogle Scholar
Pritchard, JC, Lindberg, AC, Main, DCJ and Whay, HR 2005 Assessment of the welfare of working horses, mules and donkeys, using health and behaviour parameters. Preventive Veterinary Medicine 69: 265–283CrossRefGoogle ScholarPubMed
Pritchard, JC, Barr, ARS and Whay, HR 2006 Validity of a behavioural measure of heat stress and a skin tent test for dehydration in working horses and donkeys. Equine Veterinary Journal 38: 433–438CrossRefGoogle Scholar
Pritchard, JC, Barr, ARS and Whay, HR 2007 Repeatability of a skin tent test for dehydration in working horses and donkeys. Animal Welfare 16: 181–183Google Scholar
Pritchard, JC, Burn, CC, Barr, ARS and Whay, HR 2008 Validity of indicators of dehydration in working horses: a longitudinal study of changes in skin tent duration, mucous membrane dryness and drinking behaviour. Equine Veterinary Journal 40: 558–564CrossRefGoogle ScholarPubMed
Sim, J and Wright, CC 2005 The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Physical Therapy 85: 257–268CrossRefGoogle ScholarPubMed
Svendsen, ED 1997 The Professional Handbook of the Donkey, Third Edition. Whittet Books Limited: London, UKGoogle Scholar
Tesfaye, A and Curran, MM 2005 A longitudinal survey of market donkeys in Ethiopia. Tropical Animal Health and Production 37: 87–100CrossRefGoogle ScholarPubMed
Vach, W 2005 The dependence of Cohen's kappa on the prevalence does not matter. Journal of Clinical Epidemiology 58: 655–661CrossRefGoogle Scholar
Whay, HR, Main, DCJ, Green, LE and Webster, AJF 2003 Animal-based measures for the assessment of welfare state of dairy cattle, pigs and laying hens: Consensus of expert opinion. Animal Welfare 12: 205–217Google Scholar