Trainees’ opinions on why WPBA was introduced

Trainees’ perceptions of the main reasons and the most important driving forces behind the introduction of WPBA are presented in Table 1.

Table 1. Trainees’ perceptions about the reasons and driving forces behind the introduction of workplace-based assessment

	%	n
Main reasons
To improve training	43	35
Politically driven	41	33
To improve patient care	9	7
Failure of the previous training system	5	4
Do not know	2	2
Most important driving forces
PMETB	44	36
Government/political	44	36
Royal College of Psychiatrists	6	5
Do not know	4	3
Public demand	1	1
Trainers	0	0

The two most prominent perceived reasons are ‘to improve training’ (43%) and ‘politically driven’ (41%). Though disparate, they account for almost 84% of the reasons given. Only 9% of trainees considered that WPBA was introduced ‘to improve patient care’.

The two most commonly perceived driving forces are the Postgraduate Medical Education and Training Board (PMETB) and the government, identified by almost 89% of trainees.

Attitudes to and perceptions of WPBA

The majority of trainees are unimpressed with WPBA as an educational/assessment tool in terms of its reliability, validity and the evidence that underpins it (Table 2). More than three in four trainees expressed concerns about the perceived impact of assessors’ personal preferences, the provision of training and the manner of introduction of WPBA.

Table 2. Trainees’ attitudes and perceptions to WPBA, its effects, practicalities of assessments and the Royal College of Psychiatrists’ assessment tools^a

	Strongly disagree	Disagree	No opinion	Agree	Strongly agree
	% (n)
Workplace-based assessment:
is backed by good evidenceb	17 (13)	42 (33)	30 (24)	11 (9)	0 (0)
is validb	16 (13)	35 (28)	26 (21)	23 (18)	0 (0)
is reliable	24 (19)	44 (36)	17 (14)	14 (11)	1 (1)
is independent of the assessor's personal preferences	28 (23)	53 (43)	9 (7)	5 (4)	5 (4)
has appropriately been made compulsory	28 (23)	41 (33)	17 (14)	12 (10)	1 (1)
has been introduced in a well thought-out manner	31 (25)	52 (42)	11 (9)	5 (4)	1 (1)
has been accompanied by sufficient and relevant information and guidance	30 (24)	38 (31)	11 (9)	19 (15)	3 (2)
has been accompanied by sufficient trainingc	36 (29)	39 (31)	18 (14)	5 (4)	3 (2)
Workplace-based assessment has improved:
supervision	14 (11)	40 (32)	16 (13)	27 (22)	4 (3)
training	19 (15)	42 (34)	21 (17)	16 (13)	3 (2)
clinical practice	17 (14)	44 (36)	20 (16)	16 (13)	3 (2)
confidence	21 (17)	41 (33)	17 (14)	19 (15)	3 (2)
Assessments:
are easy to organise	28 (23)	48 (39)	11 (9)	12 (10)	0 (0)
have no impact on time available for clinical duties	21 (17)	49 (40)	15 (12)	14 (11)	1 (1)
are facilitated by easy access to computers	38 (29)	41 (33)	12 (10)	11 (9)	0 (0)
receive the necessary support from supervisors and colleagues	14 (11)	30 (24)	21 (17)	33 (27)	3 (2)
Assessment tools adopted by the College:b
are easy to use	14 (11)	32 (25)	20 (16)	32 (25)	3 (2)
are relevant to psychiatry	5 (4)	27 (21)	24 (19)	43 (34)	1 (1)
facilitate training well	14 (11)	44 (35)	28 (22)	13 (10)	1 (1)
Overall, workplace-based assessment in its current form:c
is being used appropriately	14 (11)	40 (32)	28 (22)	18 (14)	1 (1)
is acceptable to you as a trainee	19 (15)	35 (28)	24 (19)	21 (17)	1 (1)
accurately reflects trainees’ progress	25 (20)	43 (34)	18 (14)	13 (10)	3 (2)
is a better system than the previous one	24 (19)	35 (28)	24 (19)	15 (12)	3 (2)
is the way forward and should be retained	30 (24)	28 (22)	29 (23)	11 (9)	3 (2)

The majority of trainees feel that WPBA has had no real beneficial effects on supervision, training, clinical practice and confidence.

Over 70% of trainees feel that there are difficulties organising assessments and that assessments have a negative impact upon time for clinical duties, 44% of trainees harbour concerns about lack of support from supervisors and colleagues, and 79% have concerns about the lack of access to computer facilities in the context of assessments.

Trainees’ perceptions of their assessors

For the purpose of this survey, ‘acceptable’, ‘good’ and ‘very good’ were considered as adequate standards.

Consultant assessors

Educational (consultant) supervisors were rated well in terms of ability to assess accurately and impartially (93% of trainees), willingness to complete assessments (91%), ability to provide constructive feedback (89%), computer literacy (88%), and availability (85%) (Table 3). They were rated moderately on knowledge of WPBA (66%) and understanding of the online system (70%).

Table 3. Trainees’ perceptions of their assessors

	Consultant assessors (N = 80)	Non-consultant assessors, (N = 77)	Non-doctor assessors, (N = 81)
	% (n)
Availability	69 (85)a	51 (66)	50 (62)
Willingness to complete assessments	73 (91)	58 (75)	58 (72)
Knowledge of workplace-based assessments	53 (66)	44 (57)	5 (6)
Computer literacy	70 (88)	69 (90)	44 (54)
Understanding of the online system	56 (70)	49 (64)	15 (19)
Ability to assess accurately and impartially	74 (93)	57 (75)b	50 (62)
Ability to provide constructive feedback	71 (89)	56 (73)	45 (56)

Trainees’ perceptions about recording assessments

Once again, ‘acceptable’, ‘good’ and ‘very good’ were considered adequate standards. Overall, trainees did not rate the online system very highly (Table 4). The percentages of trainees who found it unacceptable were: 78% in terms of ease of use, 75% in terms of reliability, 70% in terms of time consumption, and 96 % in terms of the appropriateness of the fee that trainees will have to pay. Overall, from the trainees’ perspective, recording assessments online would not render the process of WPBA any easier compared with a paper-based system.

Table 4. Trainees’ perception of the online system of recording assessments (n = 78)

	Very poor	Poor	Acceptable	Good	Very good
	% (n)
Ease of use	51 (40)	27 (21)	15 (12)	5 (4)	1 (1)
Reliability	44 (34)	31 (24)	19 (15)	5 (4)	1 (1)
Time consumption	35 (27)	35 (27)	21 (16)	9 (7)	1 (1)
Appropriateness of the fee	68 (53)	28 (22)	3 (2)	0 (0)	1 (1)

The majority of trainees (51%) would prefer assessments to be recorded on paper (Table 5), although 38% would be happy for a combination system (i.e. largely online, provided assessments could be done on paper if an assessor could not be brought to a computer). Only 6% of individuals would prefer an online system (Table 6).

Table 5. Trainees’ perception of recording of assessments online in comparison with on paper (n = 80)

	Strongly disagree	Disagree	No opinion	Agree	Strongly agree
	% (n)
Will be easier	40 (32)	39 (31)	8 (6)	11 (9)	3 (2)
Will promote better compliance	25 (20)	44 (35)	11 (9)	16 (13)	4 (3)
Will be more reliable	26 (21)	40 (32)	18 (14)	13 (10)	4 (3)
Will consume less time	34 (27)	45 (36)	13 (10)	6 (5)	3 (2)
Will be more cost-effective	30 (24)	35 (28)	16 (13)	15 (12)	4 (3)
Will be easier for assessors to complete forms	41 (33)	45 (36)	10 (8)	0 (0)	4 (3)

Table 6. Trainees’ preference with regard to a system for recording assessments

	% (n)
On paper	51 (41)
Online	6 (5)
Either (both are equally good)	5 (4)
Combination (largely online, but on paper if an assessor cannot be brought to a computer)	38 (31)

Trainees’ concerns about the WPBA

The survey also incorporated the option for respondents to express in free text their views on WPBA. The most common comments and concerns were, in summary:

1. • WPBA proving to be a ‘tick-box’ exercise to fulfil annual review of competence progress requirements with few real training benefits, particularly for post-membership trainees

2. • lack of established standards for various training grades (ST1-6)

3. • significant difficulties organising and completing assessments (particularly online)

4. • lack of enthusiasm from assessors

5. • poor quality of trainers

6. • time spent performing WPBAs detracts from clinical work

7. • inconsistencies in assessments - high degrees of subjectivity, assessor bias

8. • patchy and inadequate training to assessors

9. • unreliable and unpopular online system.

Recommendations by trainees

Respondents were also invited to offer their recommendations with regard to remedial measures that might improve the WPBA system. The most common recommendations included:

1. • targeted training for assessors, particularly non-medical, will be essential (this was a consistent theme);

2. • due to current unreliability the online system needs major overhauling or should be abolished - if retained, there should be easy access to computers and no fee for the online system;

3. • methods should be devised to standardise assessments between assessors;

4. • there should be an option of recording WPBA on paper;

5. • skills that are appropriate for assessment by non-doctors should be reviewed and defined, since such assessments are relevant to some skills but not others;

6. • the assessment tools should be rendered relevant to the individual specialty and should be more flexible;

7. • the responsibility for completing assessments should not rest solely with the trainee;

8. • trainees should be given protected time to organise WPBA;

9. • the number of assessments should be reduced and the relevance of each one should be improved: the results of assessments should be interpreted and summarised in a meaningful manner;

10. • WPBA should be tailored to the level of the trainee (e.g. higher trainees should be assessed by suitable and appropriate members of staff).

Other changes introduced with WPBA

It is of relevance that this completely novel assessment system was introduced simultaneously with radical changes to medical training (MMC), medical recruitment (Medical Training Application Service, MTAS) and the core curriculum. The resultant confusion and uncertainty with regard to both the present and the future has had a significant impact on trainees’ confidence and morale. ^{Reference Tooke3,Reference Lydall, Malik and Bhugra4} Although any change tends to provoke resistance, changes imposed under such circumstances further heighten reluctance to participate. Moreover, rather than a phased and gradual introduction, WPBA was made mandatory from the outset, for all specialty trainees appointed under MMC. Changes were rushed through, with little time for comprehension, acceptance and adaptation by both trainees and trainers.

Time constraints and other pressures

It is also relevant that the entire responsibility for maintaining a performance-based portfolio has been placed on the trainee. This combines with time constraints (and perhaps lack of faith in a new unproven process) on the part of assessors in creating significant difficulties for the trainee. To further complicate the picture, it is relevant that non-fulfilment of the new requirements could result in trainees being failed at their annual appraisals, with potential denial of progress up the training ladder. The process is therefore at considerable risk of degenerating into a tick-box/paper-pushing exercise, which nevertheless, by its very nature, is likely to take priority over the acquisition of essential clinical experience.

Apart from this, WPBA is intrinsically time and resource intensive. Increased levels of supervision and assessment of trainees, integral to the organisation and completion of the new performance-based portfolio, will undoubtedly detract from time available for the performance of clinical duties, and in combination with the recent reductions in junior doctors’ working hours could paradoxically run counter to the stated intention of improving patient care. Surprisingly, despite the extra work involved for both trainers and trainees, time implications have received little official consideration, there having been no formal allocation of time for WPBA in either consultant contracts or trainee timetables. ^{Reference Grant5}

Problems with HcAT

A further source of frustration to trainees involved the introduction of an online system (Healthcare Assessment and Training, HcAT), with a view to permanently storing assessments and generating summary reports from completed assessments. ⁶ It was intended that HcAT would be rendered compulsory and also that it would constitute the only acceptable form of evidence of training. Apart from the fact that the benefits of online documentation are debatable, the resultant difficulties faced by both trainees and assessors, in terms of awareness of the new process, computer literacy, access to computers and indeed time were immense.

The majority of the trainees found the online system of recording assessments unacceptable in terms of time consumption (70%), reliability (75%) and ease of use (78%) (Table 4). The HcAT system was plagued by numerous difficulties, so much so that it was abandoned by the Royal College of Psychiatrists in favour of ‘Assessments Online’, an updated system which appears to have addressed some of the technical problems associated with HcAT, although some difficulties remain. Any online system, however, is intrinsically prone to user factors, not the least of which involves adherence from assessors (particularly non-medical).

The fee for the online assessment system also carries potential for further resentment (for 96% of trainees this is unacceptable), since registration with the College is now mandatory for training to be recognised. ⁷

Comparison with the previous appraisal system

A small but significant proportion of trainees (17%) had no experience under the previous training system and their views about WPBA may therefore be overly optimistic or pessimistic. Nevertheless, given the overall current opinion expressed by trainees, it is evident that much work remains for WPBA to effectively address the deficiencies of the previous RITA appraisal system that were identified by PMETB: regional variability, lack of quality assurance and governance mechanisms, a perception by doctors that it was a bureaucratic and form-filling exercise, and paucity of time and resources allocated for training. ⁸ Advocates of WPBA may argue that the current difficulties are merely ‘teething problems’ inherent to any new system which will resolve in time. Although this may evoke instant scepticism, if there is indeed substance to this point of view, it is certainly unacceptable for current trainees to become casualties of yet another new initiative introduced, from the trainees’ perspective, without adequate thought and consideration from policy makers.

The way forward

It would appear that a great deal of essential work remains with regard to validating the assessment tools adopted, determining the optimal number of assessments required, exploring the effects of employing non-medical assessors to assess senior trainees, establish reliability, validity and repeatability of assessments, and correlating WPBA scores with the achievement of various competencies. ^{Reference Searle9} It is tempting to speculate whether WPBA would have been met with a more cordial reception had these factors been addressed before its implementation. It would intuitively appear that many of the current problems might have been avoided had WPBA been introduced in a phased and structured manner, with preliminary address of issues around training of assessors and due consideration given to feedback from both trainees and trainers.

A standardised, transparent and reproducible system of appraisal carries obvious advantages in terms of affording robust assessment and certification of trainees’ competencies. Well-informed and competent assessors (which may include consultants, other senior doctors, psychologists, nurses and social workers) are essential for such a system to function effectively. Standardisation, however, mandates consistency, which in turn requires structured training and quality assurance. Training has been perceived as patchy at best, as evidenced by the poor scores allocated to assessors, particularly those from non-medical backgrounds, by an overwhelming majority of trainees, and yet there is no process for quality assurance. Perhaps worryingly, non-medical assessors were rated very poorly on the three most important parameters: understanding of WPBA (6%), the ability to assess accurately and impartially (62%), and the ability to provide constructive feedback (56%). This carries serious implications in that, for instance, an assessment by an inadequately trained assessor carries a higher than acceptable risk of inaccurately reflecting a trainee's ability or performance, with obvious potential to inappropriately boost or damage their confidence with attendant implications. It is interesting that even consultants, the group rated most highly by trainees, scored poorly on knowledge of WPBA and understanding of the online system. Structured and specific training of all assessors is therefore a fundamental requirement in order to minimise interpersonal variability (currently inherent to this system) and in doing so, to maximise the objectivity of assessments. Training of assessors is also relevant to equip them (if required) to deliver negative feedback to a trainee in a constructive and supportive manner, to ensure that assessment becomes a productive learning exercise. As a corollary, it is equally important that trainees too are educated about WPBA, so as to preclude the development of resentment and demoralisation in response to a less than flattering assessment. The confidence of all stakeholders would also be enhanced by the development of an effective quality assurance process.

Competency-based assessment is new to psychiatry and represents uncharted territory. Like any system of training, it carries both strengths and limitations. Needless to say, it is imperative that robust measures be employed to continuously build on its strengths, remedy its intrinsic limitations and tailor it to the idiosyncrasies of psychiatric training, while facilitating widespread understanding, cooperation and uptake, in order to facilitate the development of a new generation of comprehensively trained doctors.

Study strengths and limitations

Despite the intrinsic limitations of a cross-sectional survey, the high response rate indicates that the results are representative of the views of psychiatric trainees in Wales; indeed, this high response rate, unusual for this type of study, perhaps indicates the strength of feeling among trainees on this subject. This survey was conducted approximately a year after the introduction of WPBA and therefore involved respondents who had acquired first-hand practical experience with this system of assessment.

One possible limitation of a questionnaire-based survey involves the narrow scope and possible inadequacy of the spectrum of selectable responses for any given question with regard to accurately expressing the views of participants (a respondent might, for instance, select the ‘no opinion’ box for lack of one that best expresses their views). This survey sought to address this by affording responders the option to state, in free text, any further views they wished to express over and above the areas addressed specifically by the questionnaire. There again, this study design affords the advantage of a reduced level of ‘observer bias’ since the anonymity enjoyed by the respondents removes the constraints with regard to expressing their views freely.