Hostname: page-component-cd9895bd7-7cvxr Total loading time: 0 Render date: 2024-12-23T23:43:01.343Z Has data issue: false hasContentIssue false
  • English
  • Français

Concordance between clinician, supervisor and observer ratings of therapeutic competence in CBT and treatment as usual: does clinician competence or supervisor session observation improve agreement?

Published online by Cambridge University Press:  06 December 2019

EB Caron Open the ORCID record for EB Caron [Opens in a new window] ,
Michela A. Muggeo ,
Heather R. Souer ,
Jeffrey E. Pella  and
Golda S. Ginsburg
EB Caron*
Affiliation:
Department of Psychological Science, Fitchburg State University, Fitchburg, MA 01420, USA
Michela A. Muggeo
Affiliation:
Clarus Health Alliance, Norwich, CT 06360, USA
Heather R. Souer
Affiliation:
Department of Psychiatry, University of Connecticut School of Medicine, West Hartford, CT 06119, USA
Jeffrey E. Pella
Affiliation:
Department of Psychiatry, University of Connecticut School of Medicine, West Hartford, CT 06119, USA
Golda S. Ginsburg
Affiliation:
Department of Psychiatry, University of Connecticut School of Medicine, West Hartford, CT 06119, USA
*
*Corresponding author. Email: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

Background:

Lowering the cost of assessing clinicians’ competence could promote the scalability of evidence-based treatments such as cognitive behavioral therapy (CBT).

Aims:

This study examined the concordance between clinicians’, supervisors’ and independent observers’ session-specific ratings of clinician competence in school-based CBT and treatment as usual (TAU). It also investigated the association between clinician competence and supervisory session observation and rater agreement.

Method:

Fifty-nine school-based clinicians (90% female, 73% Caucasian) were randomly assigned to implement TAU or modular CBT for youth anxiety. Clinicians rated their confidence after each therapy session (n = 1898), and supervisors rated clinicians’ competence after each supervision session (n = 613). Independent observers rated clinicians’ competence from audio recordings (n = 395).

Results:

Patterns of rater discrepancies differed between the TAU and CBT groups. Correlations with independent raters were low across groups. Clinician competence and session observation were associated with higher agreement among TAU, but not CBT, supervisors and clinicians.

Conclusions:

These results support the gold standard practice of obtaining independent ratings of adherence and competence in implementation contexts. Further development of measures and/or rater training methods for clinicians and supervisors is needed.

Keywords

cognitive behavioral therapycompetenceimplementationself-assessmentsupervisiontreatment as usual
Type
Main
Information
Behavioural and Cognitive Psychotherapy , Volume 48 , Issue 3 , May 2020 , pp. 350 - 363
Copyright
© British Association for Behavioural and Cognitive Psychotherapies 2019

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Breitenstein, S. M., Gross, D., Garvey, C. A., Hill, C., Fogg, L., & Resnick, B. (2010). Implementation fidelity in community‐based interventions. Research in Nursing & Health, 33, 164–173. doi: 10.1002/nur.20373Google ScholarPubMed
Brosan, L., Reynolds, S., & Moore, R. G. (2008). Self-evaluation of cognitive therapy performance: do therapists know how competent they are? Behavioural and Cognitive Psychotherapy, 36, 581–587. doi: 10.1017/S1352465808004438CrossRefGoogle Scholar
Caron, E., & Dozier, M. (2019). Effects of fidelity-focused consultation on clinicians’ implementation: an exploratory multiple baseline design. Administration and Policy in Mental Health and Mental Health Services Research, 46, 445–457. doi: 10.1007/s5488-019-00924-3CrossRefGoogle ScholarPubMed
Carroll, K., Nich, C., & Rounsaville, B. (1998). Utility of therapist session checklists to monitor delivery of coping skills treatment for cocaine abusers. Psychotherapy Research, 8, 307–320. doi: 10.1080/10503309812331332407CrossRefGoogle Scholar
Dennhag, I., Gibbons, M. B. C., Barber, J. P., Gallop, R., & Crits-Christoph, P. (2012). Do supervisors and independent judges agree on evaluations of therapist adherence and competence in the treatment of cocaine dependence? Psychotherapy Research, 22, 720–730. doi: 10.1080/10503307.2012.716528CrossRefGoogle ScholarPubMed
Farmer, E. M., Burns, B. J., Phillips, S. D., Angold, A., & Costello, E. J. (2003). Pathways into and through mental health services for children and adolescents. Psychiatric Services, 54, 60–66. doi: 10.1176/appi.ps.54.1.60CrossRefGoogle ScholarPubMed
Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: a synthesis of the literature (FMHI Publication No. 231). Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network. Retrieved from: https://fpg.unc.edu/node/4445Google Scholar
Ginsburg, G. S., Becker, K. D., Drazdowski, T. K., & Tein, J. Y. (2012). Treating anxiety disorders in inner city schools: results from a pilot randomized controlled trial comparing CBT and usual care. Child & Youth Care Forum, 42, 1–19. doi: 10.1007/s5566-011-9156-4CrossRefGoogle Scholar
Ginsburg, G. S., Muggeo, M., Caron, E., Souer, H. R., & Pikulski, P. J. (2019). Exploring treatment as usual for pediatric anxiety disorders among school-based clinicians. School Mental Health, 11, 719–727.CrossRefGoogle Scholar
Ginsburg, G. S., Pella, J. E., Pikulski, P. J., Tein, J. T., & Drake, K. L. (in press). School-based treatment for anxiety research study (STARS): a randomized controlled effectiveness trial. Journal of Abnormal Child Psychology.Google Scholar
Hogue, A., Dauber, S., Lichvar, E., Bobek, M., & Henderson, C. E. (2015). Validity of therapist self-report ratings of fidelity to evidence-based practices for adolescent behavior problems: correspondence between therapists and observers. Administration and Policy in Mental Health and Mental Health Services Research, 42, 229–243. doi: 10.1007/s5488-014-0548-2CrossRefGoogle ScholarPubMed
Hulleman, C. S., & Cordray, D. S. (2009). Moving from the lab to the field: the role of fidelity and achieved relative intervention strength. Journal of Research on Educational Effectiveness, 2, 88–110. doi: 10.1080/19345740802539325CrossRefGoogle Scholar
Hurlburt, M. S., Garland, A. F., Nguyen, K., & Brookman-Frazee, L. (2010). Child and family therapy process: concordance of therapist and observational perspectives. Administration and Policy in Mental Health and Mental Health Services Research, 37, 230–244. doi: 10.1007/s5488-009-0251-xCrossRefGoogle ScholarPubMed
Kazdin, A. E. (2017). Addressing the treatment gap: a key challenge for extending evidence-based psychosocial interventions. Behaviour Research and Therapy, 88, 7–18. doi: 10.1016/j.brat.2016.06.004CrossRefGoogle ScholarPubMed
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15, 155–163. doi: 10.1016/j.jcm.2016.02.012CrossRefGoogle ScholarPubMed
Loades, M. E., & Armstrong, P. (2016). The challenge of training supervisors to use direct assessments of clinical competence in CBT consistently: a systematic review and exploratory training study. The Cognitive Behaviour Therapist, 9. doi: 10.1017/S1754470X15000288CrossRefGoogle Scholar
Loades, M. E., & Myles, P. J. (2016). Does a therapist’s reflective ability predict the accuracy of their self-evaluation of competence in cognitive behavioural therapy? The Cognitive Behaviour Therapist, 9. doi: 10.1017/S1754470X16000027CrossRefGoogle Scholar
Lowry, R. (2001–2018). Significance of the difference between two correlation coefficients. VassarStats: Website for Statistical Computation. Retrieved from: vassarstats.net/index.htmlGoogle Scholar
Martino, S., Ball, S., Nich, C., Frankforter, T. L., & Carroll, K. M. (2009). Correspondence of motivational enhancement treatment integrity ratings among therapists, supervisors, and observers. Psychotherapy Research, 19, 181–193. doi: 10.1080/10503300802688460CrossRefGoogle ScholarPubMed
Masia, C. W., Brice, C., Esseling, P. G., Stewart, C. E., Mufson, L., & Herzig, K. (2013). Consultants' perceptions of school counselors' ability to implement an empirically-based intervention for adolescent social anxiety disorder. Administration and Policy in Mental Health, 40, 541–554.CrossRefGoogle Scholar
McManus, F., Rakovshik, S., Kennerley, H., Fennell, M., & Westbrook, D. (2012). An investigation of the accuracy of therapists’ self‐assessment of cognitive‐behaviour therapy skills. British Journal of Clinical Psychology, 51, 292–306. doi: 10.1111/j.2044-8260.2011.02028.xCrossRefGoogle ScholarPubMed
Peavy, K. M., Guydish, J., Manuel, J. K., Campbell, B. K., Lisha, N., Le, T., … & Garrett, S. (2014). Treatment adherence and competency ratings among therapists, supervisors, study-related raters and external raters in a clinical trial of a 12-step facilitation for stimulant users. Journal of Substance Abuse Treatment, 47, 222–228. doi: 10.1016/j.jsat.2014.05.008CrossRefGoogle Scholar
Pellecchia, M., Connell, J. E., Beidas, R. S., Xie, M., Marcus, S. C., & Mandell, D. S. (2015). Dismantling the active ingredients of an intervention for children with autism. Journal of Autism and Developmental Disorders, 45, 2917–2927. doi: 10.1007/s5803-015-2455-0.CrossRefGoogle ScholarPubMed
Reichelt, F. K., James, I. A., & Blackburn, I. M. (2003). Impact of training on rating competence in cognitive therapy. Journal of Behavior Therapy and Experimental Psychiatry, 34, 87–99. doi: 10.1016/S0005-7916(03)00022-3CrossRefGoogle ScholarPubMed
Rozek, D. C., Serrano, J. L., Marriott, B. R., Scott, K. S., Hickman, L. B., Brothers, B. M., … & Simons, A. D. (2018). Cognitive behavioural therapy competency: pilot data from a comparison of multiple perspectives. Behavioural and Cognitive Psychotherapy, 46, 244–250. doi: 10.1017/S1352465817000662CrossRefGoogle ScholarPubMed
Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam-Gerow, M. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38, 32–43. doi: 10.1007/s5488-010-0321-0CrossRefGoogle ScholarPubMed
Wain, R. M., Kutner, B. A., Smith, J. L., Carpenter, K. M., Hu, M. C., Amrhein, P. C., & Nunes, E. V. (2015). Self-report after randomly assigned supervision does not predict ability to practice Motivational Interviewing. Journal of Substance Abuse Treatment, 57, 96–101. doi: 10.1016/j.jsat.2015.04.006CrossRefGoogle Scholar
Ward, A. M., Regan, J., Chorpita, B. F., Starace, N., Rodriguez, A., Okamura, K., … & Research Network On Youth Mental Health (2013). Tracking evidence based practice with youth: validity of the MATCH and Standard Manual Consultation Records. Journal of Clinical Child & Adolescent Psychology, 42, 44–55. doi: 10.1080/15374416.2012.700505CrossRefGoogle ScholarPubMed
Submit a response

Comments

No Comments have been published for this article.