Discussion

This study has evaluated the impact of teaching during a SABR contouring workshop for a relatively large number of participants and multiple target volume/OARs in the thorax and pelvis. The positive feedback provided by participants about the workshop suggests that it is feasible to deliver contouring teaching in a virtual capacity. This is important since, accelerated by the changes adopted during the COVID-19 pandemic, it is likely that medical training and teaching will increasingly be delivered using virtual methods. Another important message is that it is possible to continue to provide contouring training during the COVID-19 pandemic when in-person meetings are restricted. Virtual training also has the potential to reach a larger audience, without geographical restrictions, than could typically be achieved in person. We demonstrated that median DSC/LDE values for participants who completed pre- and post-workshop contours for most of the target volume/OARs were similar to the reference contour, with DSC > 0·7 for 75% of structures and LDE < 5 mm for 83% of structures. While statistically significant improvements post-workshop in DSC and LDE were observed for 50% and 58% of structures, respectively, the magnitude of improvement was small in most cases and the clinical significance of such modest improvements remains uncertain.

Although multiple studies on the effect of teaching on contouring variation have been reported, several factors make a direct comparison between these and our study challenging. ^{Reference Vinod, Jameson, Min and Holloway16} Heterogeneity exists between studies concerning the numbers of participants, types of teaching, the structures for which contouring variation is evaluated and the types of metrics used to evaluate this variation and the use of statistical tests. ^{Reference Jameson, Holloway, Vial, Vinod and Metcalfe1,Reference Vinod, Min, Jameson and Holloway2,Reference Vinod, Jameson, Min and Holloway16} However, systematic reviews of such studies have demonstrated that an improvement in contouring variation through teaching can be achieved. ^{Reference Vinod, Min, Jameson and Holloway2,Reference Cacicedo, Navarro-Martin, Gonzalez-Larragan, De Bari, Salem and Dahele20} We did not observe a large increase in DSC/reduction in LDE post-workshop, and a number of limitations of our work may explain this. While participants were asked to review their pre-workshop contours after teaching and produce a post-workshop submission, only approximately half of participants did so which reduced the number for which an analysis of teaching impact could be performed. Furthermore, even for those who did re-submit a second set of contours in some cases no changes were made. Possible reasons for this could include satisfaction with pre-workshop contours, insufficient time to re-contour every structure and a lack of hands-on time during the workshop to practise/fully compare contours with the reference contour. The latter point may be particularly relevant since it has been previously suggested that active participation is more likely to improve learning during contouring workshops. ^{Reference Eriksen, Salembier and Rivera21} Insufficient provision of practical experience was raised as a potential explanation for failure to observe improved contouring post-teaching in a previous study of a head and neck contouring programme, although there may be time/resource challenges to effectively deliver this especially for larger audiences and during the COVID-19 pandemic where face-to-face meetings are restricted. ^{Reference D’Souza, Jaswal and Chan22} Residual differences in knowledge/ability between participants despite teaching were also suggested as a possible reason why significant improvements in prostate/rectal contouring were not observed in a previous evaluation of the impact of teaching. ^{Reference Szumacher, Harnett and Warner23}

Low DSC/high LDE values for certain structures in our study could be related to interpretation of the case instructions, especially for BrachialPlex and Femur_Head_Left. The latter structure was also only to be delineated on a single axial CT slice at the very inferior aspect of the femoral head. Different methods for contouring BrachialPlex exist, and there remains variation in practice. ^{Reference Hall, Guiou and Lee24–26} Given the high dose per fraction used with SABR and variable reliance on MRI across different treatment centres, the UK SABR Consortium Guidance recommends contouring the subclavian/axillary vessels as a surrogate for BrachialPlex. ²⁶ National consensus is needed, and future iterations of the recently published OAR harmonisation guidance will support this. ^{Reference Mir, Kelly and Xiao25} For SacralPlex, some participants delineated the visible nerve using the MRI while others delineated a larger surrogate structure using the CT. Both of these may be legitimate approaches, although contouring as per the Yi et al guidance does not rely on expert MRI interpretation of nerve position and may therefore be simpler for those learning. ^{Reference Yi, Mak and Yang27} However, unfamiliarity with the contouring of certain OARs might have contributed to low DSC/high LDE results. Although not statistically significant, the median DSC/LDE for BrochusProx appeared slightly worse post-workshop. The reason for this was not clearly apparent, and the magnitude of difference was small, but it could possibly be related to delineation uncertainties regarding the distal extent of the lobar bronchi. A visual guide to delineation of BrachialPlex, BronchusProx and SacralPlex is illustrated in Figure 2 while recommended contouring guidance/atlases are collated in Table 3. ^{Reference Mir, Kelly and Xiao25–Reference Wright, Yom and Awan30}

Figure 2. Visual guide to delineation of SacralPlex, BrachialPlex and BronchusProx. In (a–c), SacralPlex; iliacus muscle (green), L5 vertebral body (dark blue), obturator internus muscle (orange), psoas muscle (light blue), SacralPlex (purple), vessels (yellow) are shown. SacralPlex is contoured using a 5 mm diameter roller ball. In 2A, superior border of SacralPlex is shown at L4/5 vertebral interspace; SacralPlex is shown bordered by (ilio)psoas muscle anteriorly and vertebral body posteriorly. In 2B, at the sacroiliac foramen, SacralPlex is shown bordered by vessels anteriorly, iliacus muscle laterally and sacral ala posteriorly. In 2C, inferior border of SacralPlex is shown at the level of the superior femoral neck bordered by obturator internus muscle anteriorly, gluteus maximus muscle posteriorly. In (d–f), BrachialPlex contoured as suggested by UK SABR Consortium Guidelines ²⁶ ; anterior scalene muscle (orange), BrachialPlex (light blue), common carotid artery (red), internal jugular vein (posterior scalene muscle (brown), subclavian artery (pink) and subclavian vein (dark blue) are shown. Intravenous contrast is helpful, and BrachialPlex is contoured using a 5 mm diameter roller ball. 2D shows a proximal slice: the superior border of BrachialPlex is at the bifurcation of the brachiocephalic trunk into the jugular/subclavian veins (or carotid/subclavian arteries). In 2E, a middle section of BrachialPlex is shown; the plexus sits between anterior and middle scalene muscles. In 2F, the neurovascular complex including the subclavian and axillary vessels is contoured as a surrogate for the brachial plexus, ending after the neurovascular structures cross the second rib. In (g–i), BronchusProx; BronchusProx (purple) is shown. In 2G, superior border of BronchusProx is the distal 2 cm of trachea including carina. In 2·2H, the mid-section of BronchusProx is shown and includes right/left upper lobe bronchi, bronchus intermedius, right middle lobe bronchus, lingular bronchus and right/left lower lobe bronchi. In 2I, contouring of lobar bronchi stops immediately at the site of a segmental bifurcation.

Table 3. A summary of resources to support target volume/organ at risk delineation

The metric thresholds that correlate to a minimum expected standard of contouring are uncertain but it has been previously suggested that DSC > 0·7 indicates a good level of agreement. ^{Reference Vinod, Min, Jameson and Holloway2} However, previous studies have demonstrated discrepancies between contours considered to be acceptable based on expert review and the results of overlap measure comparisons. ^{Reference Duke, Tan and Jensen31} In this study, 92% of the post-workshop contours were considered to be acceptable/within acceptable variation while 75% of structures had a DSC > 0·7. A range of comparison metrics exists, and each provides different information about the relationship between two contours and each has its limitations. ^{Reference Vinod, Jameson, Min and Holloway16} A summary of commonly used metrics for contour comparison is shown in Supplementary Table; it is unclear which is the optimum metric to use. ^{Reference Jameson, Holloway, Vial, Vinod and Metcalfe1,Reference Vinod, Min, Jameson and Holloway2,Reference Vinod, Jameson, Min and Holloway16,Reference Rivin del Campo, Rivera and Martínez-Paredes18,Reference Ackerly, Andrews, Ball, Guerrieri, Healy and Williams32–Reference Taha and Hanbury37} For this reason, it has previously been recommended that multiple metrics ideally be reported including measures of volume, overlap and distance. ^{Reference Jameson, Holloway, Vial, Vinod and Metcalfe1,Reference Vinod, Jameson, Min and Holloway16} In this study, we only reported DSC and LDE since we did not have volumetric contouring data. It should be emphasised that DSC may provide less reliable results when applied to very small contours and it may lack discrimination for very large volumes. ^{Reference Rivin del Campo, Rivera and Martínez-Paredes18} However, it does provide some insight into both the volumetric and spatial relationship between two contours and it is frequently reported in contouring studies. ^{Reference Jameson, Holloway, Vial, Vinod and Metcalfe1,Reference De Bari, Dahele, Palmu, Kaylor, Schiappacasse and Guckenberger11}

Quantitative concordance in target volume/OAR delineation does not necessarily equate to a clinically acceptable contour; incorrect delineation of even a small proportion of a target volume or an OAR could have profound clinical consequences, especially for SABR where tight margins, steep dose gradients and ablative doses are used. ^{Reference Vinod, Min, Jameson and Holloway2,Reference Murray, Franks and Hanna38,Reference Trakul, Koong and Chang39} This risk means that quantitative metrics should ideally be accompanied by visual review of contours and provision of qualitative feedback, analogous to the peer review process used in clinical practice and recommended by the RCR. ⁷ This approach is used in clinical trials for pre-trial approval for participation or on-trial individual case evaluation. Qualitative feedback can be provided detailing acceptable/unacceptable variation from the protocol, and a similar process was used in this study for feedback on post-workshop contours. ^{Reference Gwynne, Gilson, Dickson, McAleer and Radhakrishna3,Reference Cox, Cleves, Clementel, Miles, Staffurth and Gwynne40–Reference Gwynne, Jones and Maggs42} However, this approach may be time-consuming and an efficient/reliable method of assessment which can identify clinically relevant discrepancies is needed. ^{Reference Jameson, Holloway, Vial, Vinod and Metcalfe1,Reference Vinod, Jameson, Min and Holloway16,Reference Duke, Tan and Jensen31}

The practice of clinical oncology takes place against an increasingly complex backdrop of developments in imaging and novel methods of treatment delivery. Alongside ever-increasing pressures in healthcare services, considerable challenges exist for training and continuous professional development of trainees and consultants, respectively. ^{Reference Walls, Hanna and McAleer9} Formal training initiatives have been established to deliver the acquisition, and maintenance, of contouring competences in an attempt to improve target volume/OAR delineation beyond what could be achieved by a single workshop in isolation. The Fellowship in Anatomic deLineation and CONtouring (FALCON) programme is a European Society of Radiation Oncology (ESTRO) initiative that provides access to e-learning contouring resources in addition to its use within dedicated workshops. ^{Reference Rivin del Campo, Rivera and Martínez-Paredes18,Reference Eriksen, Salembier and Rivera21} The RCR ARENA and Clinical Oncology Planning Project (COPP) are some example of initiatives to increase access to expert/peer-led structured outlining training to promote consistency in target volume and OAR outlining, and facilitate robust assessment of outlining practice for all grades of Clinical Oncologists. ^{Reference Evans, Radhakrishna and Gilson43}

This study has a number of additional limitations. The workshop was limited in its time/level of interactivity because of restrictions imposed during the pandemic, and this could have impacted on the educational experience/DSC and LDE results that we observed (although participant feedback for the workshop remained positive). The same cases were used for both pre- and post-workshop contouring; while this enabled the analysis of paired data, it meant that post-workshop contour performance could have been influenced by familiarity with the case and thus extrapolation of similar levels of performance to other cases would not necessarily be guaranteed. We did not stratify by prior experience when undertaking our analysis; this was because this information was not available to the authors but it could have influenced the results that were obtained. The workshop was aimed at those without prior experience in SABR but experience with OAR delineation would have varied depending on disease site expertise. We also did not evaluate longer-term maintenance of contouring competences by provision of further cases for contouring as part of this workshop, although response rates for such interventions for a single workshop in isolation may be limited. ^{Reference Cacicedo, Navarro-Martin, Gonzalez-Larragan, De Bari, Salem and Dahele20} It might be expected that initial educational gains could progressively negate over time, meaning that ongoing evaluation of performance during training programmes/as part of consultant continuous professional development will be required. Finally, feedback on post-workshop contours was only available for approximately half of participants included in our analyses; this affected the conclusions that can be drawn regarding the qualitative feedback but does reflect the challenge of providing such information in a timely manner.

When planning a contouring workshop, the following considerations may be relevant based on prior recommendations/the authors’ experience ^{Reference Gwynne, Gilson, Dickson, McAleer and Radhakrishna3,Reference Vinod, Jameson, Min and Holloway16,Reference Cacicedo, Navarro-Martin, Gonzalez-Larragan, De Bari, Salem and Dahele20,Reference Duke, Tan and Jensen31,Reference Roques44} :

• Workshop format; incorporation of time to practise contouring/re-contouring is recommended in addition to didactic teaching (the duration of the workshop should be considered in relation to this)

• Clarity of instructions for cases to be contoured including detailed delineation guidance and specification of laterality, where relevant

• Timely access to relevant target volume/OAR guidance/atlases

• Provision of co-registered imaging

• Target audience; disease sites, numbers of target volume/OARs, number/complexity of cases

• Choice of assessment; quantitative metrics (such as volume, distance and overlap metrics) should ideally be used in conjunction with qualitative feedback. Be realistic about how much qualitative feedback can be provided in a timely manner to each participant

• Post-workshop, provision of expert contour (where available) for participant comparison

• Where a reference contour is used; discussion regarding variation that may occur between even ‘expert’ outliners. One approach could be to use three expert contours and demonstrate the union and overlap as the maximum and minimum acceptable contours

• Identification of common errors/sources of variation for particular target volume/OARs

• Highlight available e-learning resources for self-directed learning

• Design of workshop feedback to evaluate participant confidence in contouring before/after the workshop

• Audiovisual/technological considerations; including undertaking a ‘trial run’ of the equipment prior to the workshop, including a method of quality assurance for displayed imaging, provision for participants with disabilities and recommendations that participants ensure they have a stable Internet connection and adequate audiovisual equipment to fully participate