There are chronic shortages of psychiatrists in both the UK and internationally,^{Reference Brown and Ryland1} affecting access to mental healthcare.^{Reference Rimmer2} Selection and recruitment processes will influence the quality and quantity of practising psychiatrists. In the UK, core psychiatry training is undertaken after completion of the two Foundation Years, following graduation from medical school. Selection processes need to be fair and valid, evaluating attributes relevant to future role performance. Therefore, prior to the COVID-19 pandemic, selection for psychiatry training was nationally organised and standardised using three stages:

1. (1) Stage 1 – Administrative: proof of eligibility for UK specialty training is checked.

2. (2) Stage 2 – The Multi-Specialty Recruitment Assessment (MSRA), a computer-delivered test, is taken.

3. (3) Stage 3 – The Selection Centre, involving two face-to-face assessment stations.

The MSRA has been implemented for a range of medical specialties, including psychiatry. It uses a selected response (multiple choice) question format and consists of two elements: a Situational Judgement Test (SJT, known as the Professional Dilemmas paper) and a clinical knowledge test (the Clinical Problem Solving (CPS) paper). Boxes 1 and 2 show example SJT and CPS questions.

The MSRA aims to assess the clinical and interpersonal knowledge expected of a doctor completing their Foundation Years training. Before the pandemic candidates were nationally ranked on their combined MSRA and Selection Centre scores and this determined a training place offer. From 2018 to 2020 doctors scoring above an agreed threshold (the ‘bypass score’) on the MSRA were exempt from the Selection Centre assessments. During the COVID-19 pandemic the cost/benefit ratio of maintaining face-to-face selection processes was deemed unfavourable and the Selection Centre stage was suspended. Thus, offer decisions were informed solely by the MSRA scores.

The UK-based core training of psychiatrists involves a 3-year programme with a combination of hospital and community-based placements across a variety of psychiatric specialties. During core training the trainee must pass all components of the Royal College of Psychiatrists membership (MRCPsych) examination to progress to higher specialty training. The MRCPsych has written and practical components, the latter being the Clinical Assessment of Skills and Competencies (CASC). Over the past decade or so some changes have been made to the structure of the MRCPsych (for details see the Supplementary Material, available at https://dx.doi.org/10.1192/bjb.2024.9).³

The CASC uses the format of an Objective Structured Clinical Examination (OSCE), having 16 stations designed to evaluate a candidate's clinical skills.⁴ The scoring system and pass standard for the CASC employs the borderline regression method.^{Reference Kramer, Muijtjens, Jansen, Dusman, Tan and van der Vleuten5} In this approach, a rater evaluates a student's performance at each station by completing a checklist and a global rating scale. The checklist marks for all examinees at each station are then regressed on the attributed global rating scores, producing a linear regression equation. The global score representing borderline performance is then substituted into the equation to predict the pass/fail score for the checklist marks. To pass the CASC a candidate must achieve a passing score in at least 12 of the 16 stations and meet or exceed the pass score set. During the COVID-19 pandemic the CASC was moved to an online format. If all elements of the MRCPsych are passed, and a candidate can evidence that the specified core competencies have been achieved, the doctor can apply for higher specialty training. Once this stage of training is completed a doctor can apply for a Certificate of Completion of Training (CCT) and be placed on the specialist register held by the General Medical Council (GMC) for the appropriate branch of psychiatry. The training process for UK psychiatry is shown in Supplementary Fig. 1.

To date there is no published research relating to the effectiveness of selection into core psychiatry training. However, the validity of the UK general practice (GP) training selection process, also using the MSRA and, until the COVID-19 pandemic, a bespoke Selection Centre, has previously been explored. The findings from these studies have raised questions about the added value of Selection Centres in GP recruitment. These reported that only a small amount of variance (around 3–4%) in the clinical GP ‘exit exam’ scores is explained by the Selection Centre ratings, once the influence of the MSRA scores is controlled for.^{Reference Davison, McManus and Taylor6–Reference Tiffin, Morley, Paton and Patterson9}

Moreover, the sensitivity of selection assessments to demographic characteristics will determine the characteristics, and quantity, of the workforce selected. This is especially important in GP selection, where traditionally the workforce has had a high proportion of female, minority ethnic doctors and those who obtained their primary medical qualification outside the UK. For example, selection assessments that especially disadvantage females could have an impact on the subsequent fill rates for GP posts, in an already underserved specialty. In general, previous evidence suggests that females tend to outperform males on SJTs used for personnel selection^{Reference Whetzel, McDaniel and Nguyen10} and postgraduate clinical examinations for GP training.^{Reference Esmail and Roberts11} There is no current published evidence regarding the sensitivity of the MSRA components to candidate characteristics.

Thus, the aims of the present study were:

1. (1) to explore the predictive validity of the differing components of the psychiatry training selection process, including among candidates with relatively low MSRA scores

2. (2) to evaluate the impact of key demographic characteristics (world region of qualification, gender and ethnicity) on performance in the three elements of the selection process.

In relation to aim 1, the outcome of interest was the CASC score at first attempt, as a proxy for eventual clinical performance. In relation to aim 2, these analyses were intended to highlight how use of the various selection components may influence the demographics of the selected population of trainees.

Method

Ethics

As the study used routinely collected, de-identified data, ethical approval was not required. This was confirmed in writing by the chair of the University of York Health Sciences Ethics Committee. Moreover, it used data from the UK Medical Education Database (UKMED) and use of UKMED data is not reliant on individual consent. However, any findings published from the UKMED must be presented in blunted form.¹² Thus, all frequencies are rounded to the nearest multiple of 5.

Data processing and management

Data were available from the UK Medical Education Database (UKMED) for 3510 applicants to psychiatry training with MSRA scores from the years 2015 to 2021 inclusive, with 2445 doctors appointed. The complete flow of data through the study is shown in Fig. 1.

Fig. 1 Flow of data through the study. MSRA, Multi-Specialty Recruitment Assessment; MRCPsych, Royal College of Psychiatrists membership examination; CASC, Clinical Assessment of Skills and Competencies.

Measures

The SJT comprises 50 questions linked to interpersonally oriented, workplace-based scenarios. The CPS paper is made up of 97 questions using either a single best answer or enhanced matching question format, assessing medical knowledge, presenting clinical scenarios to provide context. The scores from the SJT and CPS are standardised across cohorts and summed to provide an overall MSRA score.

The Selection Centre consisted of a face-to-face ‘Presentation of Portfolio Station’ and a ‘Communications in a Clinical Setting Station’, with the scores from these summed. Details of the Selection Centre are provided in the Supplementary Material.

The outcome (CASC score) was not observable in unsuccessful applicants or those not sitting the CASC within the study time frame. This ‘range restriction’ in personnel selection studies is a special case of missing data.^{Reference Dahlke and Wiernik13} Consequently, missing values for both predictors and outcomes were imputed using chained equations. This is a valid approach to addressing this issue.^{Reference Zimmermann, Klusmann and Hampe14–Reference van Ginkel, Linting, Rippe and van der Voort16} As a sensitivity analysis, imputed and non-imputed results were compared (Supplementary Table 3 and Supplementary Fig. 6). To help ensure equivalence across time, the CASC scores relative to the pass mark for that sitting were used. The pass marks were not consistently available for written papers 1, 2 and 3 so the average raw scores for the first sitting of any MRCPsych written exams taken were used instead.

Analysis approach

A series of univariable regression analyses were conducted to estimate the unadjusted relationship between the scores from the MSRA elements. The correlations between the educational assessment scores were also calculated.

The potential impact of various educational and demographic characteristics (ethnicity, gender, place of qualification) on the elements of the MSRA and the Selection Centre score were estimated. Cohen's d or Glass's delta were used, depending on whether or not the variances were significantly equal across groups on testing. For the Selection Centre ratings (which included imputed values owing to the bypass score system) effect sizes were derived using the miesize package for Stata 17.0 MP version for Windows.^{Reference Tiffin17} Only the selection assessment scores were entered into the multivariable model, as demographic variables are not used in selection decisions. Receiver operator characteristic (ROC) curves were generated from the imputed data, treating the MSRA scores to ‘screen’ for those passing the CASC at first attempt.

Path analysis

Path analysis was conducted to model the unique relationships between the assessment scores and mitigate against the ‘Table 2 fallacy’.^{Reference Westreich and Greenland18} A theoretical path model for the prediction of CASC performance from the educational and selection assessment scores was developed. This was informed by prior research on predictors of postgraduate clinical exam performance^{Reference Patterson, Tiffin, Lopes and Zibarras19}(Fig. 2).

Fig. 2 The a priori theoretical model hypothesising the causal relationships between the selection assessment scores and the outcome of interest (performance on the Clinical Assessment of Skills and Competencies, CASC). MSRA, Multi-Specialty Recruitment Assessment; SJT, Situational Judgement Test; CPS, Clinical Problem Solving; MRCPsych, Royal College of Psychiatrists membership examination.

The Stata and Mplus code used to manage and analyse the data is publicly available (github.com/pat512-star/P155).

Results

A summary of the demographic and academic details for applicants and entrants are shown in Tables 1 and 2.

Table 1 Demographic variables for applicants to psychiatry training with and without the primary outcome of interest (CASC score at first attempt)

CASC, Clinical Assessment of Skills and Competencies; SES, socioeconomic status; BAME, Black and minority ethnic; EEA, European Economic Area; IMG, international medical graduate.

Table 2 Educational variables for applicants to psychiatry training with and without the primary outcome of interest (CASC score at first attempt)

CASC, Clinical Assessment of Skills and Competencies; CPS, Clinical Problem Solving; SJT, Situational Judgement Test; MSRA, Multi-Specialty Recruitment Assessment.

Applicants who entered training and had at least one attempt at the CASC recorded were more likely to be female, be a UK graduate and identify as of White ethnicity (all P < 0.001 on χ² testing).

Please note that all the results shown here are derived from the imputed data (m = 10) unless otherwise indicated.

Univariable results

Univariable relationship between the measures

Both components of the MSRA demonstrated significant univariable relationships with Selection Centre ratings. Similarly, the CPS and MRCPsych written examination scores were relatively highly correlated (rho = 0.52). The correlations (Spearman's rho) between the key measures are shown in Table 3 for the non-imputed data.

Table 3 Ranked correlations (rho values) between the metrics of interest in the non-imputed study data

The results of the univariable analyses evaluating the relationship between the selection assessment and CASC scores are shown in Table 4. As can be seen, all three selection measures are significantly predictive of CASC performance.

Table 4 Univariable regression analyses conducted in multiply imputed data (m = 10) predicting Clinical Assessment of Skills and Competencies (CASC) performance (relative to the pass mark) at first attempt from scores on the three selection measures

The results of the ROC analysis are presented in the Supplementary Material.

Sensitivity to demographic characteristics

The effect sizes for the three selection measures (CPS, SJT and Selection Centre scores) are shown in Fig. 3. In general, the measures are most sensitive to world region of primary medical qualification. Compared with the MSRA elements, the Selection Centre scores were relatively insensitive to ethnicity.

Fig. 3 Effect sizes based on demographic characteristics. The Selection Centre effect sizes are calculated from the analysis of multiply-imputed data (m = 10). Socioeconomic background (professional versus non-professional parental occupation) were available only for UK graduates. CPS, Clinical Problem Solving; SJT, Situational Judgement Test; Prof., professional; EEA, European Economic Area; IMG, international medical graduate.

Multivariable results

The results of the multiple linear regression analysis are shown in Table 5. All the assessment scores make a significant and relatively meaningful contribution to predicting CASC performance. The predictive ability of the MSRA elements reduces, and the relative contribution of the Selection Centre scores increases, for those with low scores on the MSRA. The imputed and non-imputed data results are almost identical.

Table 5 Multivariable linear regression analysis predicting Clinical Assessment of Skills and Competencies (CASC) performance at first sitting from the scores from the three selection measures on the multiply imputed study data (m = 10)^a

a. These results are provided for the whole sample and for those achieving relatively low scores on the Multi-Specialty Recruitment Assessment (MSRA).

Path analysis results

A path analysis, based on the a priori hypothesised model (see earlier) was estimated in the imputed data-sets (m = 10). When evaluating model fit a confirmatory fit index (CFI) and Tucker–Lewis index (TLI) exceeding 0.90 is deemed indicative of acceptable fit, with values exceeding 0.95 consistent with ‘good’ fit.^{Reference Hu and Bentler20}

The path model for all applicants is shown in Fig. 4 and that for those with low MSRA scores (<484) is shown in Fig. 5. Both models generally showed an acceptable fit to the data (Table 6 and Figs 4 and 5). The exception was that the model for low scorers had a TLI of only 0.87. It can be seen that the relationship between the CPS and Selection Centre score is close to zero (Fig. 5). When this redundant path was omitted the TLI increased to 0.90, indicating satisfactory fit (the CFI remained at 0.96). The results of the mediational analyses are reported in the Supplementary Material.

Fig. 4 Path model A, testing the relationship between the predictors and outcome (score on the Clinical Assessment of Skills and Competencies, CASC) in the multiply imputed dataset (n = 3510). MSRA, Multi-Specialty Recruitment Assessment; SJT, Situational Judgement Test; CPS, Clinical Problem Solving; MRCPsych, Royal College of Psychiatrists membership examination.

Fig. 5 Path model B, for those scoring below 484 in the Multi-Specialty Recruitment Assessment (MSRA), using multiply imputed data (n = 1315). SJT, Situational Judgement Test; CPS, Clinical Problem Solving; MRCPsych, Royal College of Psychiatrists membership examination; CASC, Clinical Assessment of Skills and Competencies.

Table 6 Fit indices for the two path models tested^a

CFI, confirmatory fit index; TLI, Tucker–Lewis index; RMSEA, root mean square error of approximation; MSRA, Multi-Specialty Recruitment Assessment.

a. For imputed data analyses the index shown is that averaged for the results over the imputations (m = 10)