There is a scarcity of evaluated tools to assess whether non-specialist providers achieve minimum levels of competency to effectively and safely deliver psychological interventions in low- and middle-income countries. The objective of this study was to evaluate the reliability and utility of the newly developed Working with children – Assessment of Competencies Tool (WeACT) to assess service providers’ competencies in Gaza, Palestine.

The study evaluated; (1) psychometric properties of the WeACT based on observed role-plays by trainers/supervisors (N = 8); (2) sensitivity to change among service provider competencies (N = 25) using pre-and-post training WeACT scores on standardized role-plays; (3) in-service competencies among experienced service providers (N = 64) using standardized role-plays.

We demonstrated moderate interrater reliability [intraclass correlation coefficient, single measures, ICC = 0.68 (95% CI 0.48–0.86)] after practice, with high internal consistency (α = 0.94). WeACT assessments provided clinically relevant information on achieved levels of competencies (55% of the competencies were scored as adequate pre-training; 71% post-training; 62% in-service). Pre-post training assessment saw significant improvement in competencies (W = −3.64; p < 0.001).

This study demonstrated positive results on the reliability and utility of the WeACT, with sufficient inter-rater agreement, excellent internal consistency, sensitivity to assess change, and providing insight needs for remedial training. The WeACT holds promise as a tool for monitoring quality of care when implementing evidence-based care at scale.