Technical Skill Assessment – Measuring Staff Capabilities and Optimizing Training Investments

Surgeons are increasingly being scrutinized for their performance and there is growing interest in objective assessment of technical skills. The purpose of this study was to review all evidence for these methods, in order to provide a guideline for use in clinical practice. A systematic search was performed using PubMed and Web of Science for studies addressing the validity and reliability of methods for objective skills assessment within surgery and gynaecology only. The studies were assessed according to the Oxford Centre for Evidence-based Medicine levels of evidence. In total 104 studies were included, of which 20 (19.2 per cent) had a level of evidence 1b or 2b. In 28 studies (26.9 per cent), the assessment method was used in the operating room. Virtual reality simulators and Objective Structured Assessment of Technical Skills (OSATS) have been studied most. Although OSATS is seen as the standard for skills assessment, only seven studies, with a low level of evidence, addressed its use in the operating room. Based on currently available evidence, most methods of skills assessment are valid for feedback or measuring progress of training, but few can be used for examination or credentialing. The purpose of the assessment determines the choice of method.

To assess where we currently stand in relation to simulator-based training within modern surgical training curricula. A systematic literature search was performed in PubMed database using keywords "simulation", "skills assessment" and "surgery". The studies retrieved were examined according to the inclusion and exclusion criteria. Time period reviewed was 2000 to 2013. The methodology of skills assessment was examined. Five hundred and fifteen articles focussed upon simulator based skills assessment. Fifty-two articles were identified that dealt with technical skills assessment in general surgery. Five articles assessed open skills, 37 assessed laparoscopic skills, 4 articles assessed both open and laparoscopic skills and 6 assessed endoscopic skills. Only 12 articles were found to be integrating simulators in the surgical training curricula. Observational assessment tools, in the form of Objective Structured Assessment of Technical Skills (OSATS) dominated the literature. Observational tools such as OSATS remain the top assessment instrument in surgical training especially in open technical skills. Unlike the aviation industry, simulation based assessment has only now begun to cross the threshold of incorporation into mainstream skills training. Over the next decade we expect the promise of simulator-based training to finally take flight and begin an exciting voyage of discovery for surgical trainees.

Objective assessment of intraoperative technical skill is necessary for technology to improve patient care through surgical training. Our objective in this study was to develop and validate deep learning techniques for technical skill assessment using videos of the surgical field. We used a data set of 99 videos of capsulorhexis, a critical step in cataract surgery. One expert surgeon annotated each video for technical skill using a standard structured rating scale, the International Council of Ophthalmology's Ophthalmology Surgical Competency Assessment Rubric:phacoemulsification (ICO-OSCAR:phaco). Using two capsulorhexis indices in this scale (commencement of flap and follow-through, formation and completion), we specified an expert performance when at least one of the indices was 5 and the other index was at least 4, and novice otherwise. In addition, we used scores for capsulorhexis commencement and capsulorhexis formation as separate ground truths (Likert scale of 2 to 5; analyzed as 2/3, 4 and 5). We crowdsourced annotations of instrument tips. We separately modeled instrument trajectories and optical flow using temporal convolutional neural networks to predict a skill class (expert/novice) and score on each item for capsulorhexis in ICO-OSCAR:phaco. We evaluated the algorithms in a five-fold cross-validation and computed accuracy and area under the receiver operating characteristics curve (AUC). The accuracy and AUC were 0.848 and 0.863 for instrument tip velocities, and 0.634 and 0.803 for optical flow fields, respectively. Deep neural networks effectively model surgical technical skill in capsulorhexis given structured representation of intraoperative data such as optical flow fields extracted from video or crowdsourced tool localization information.

Although virtual reality (VR) simulators can be used ability to objectively assess skills in endoscopy, the evaluation is solely quantitative. We have developed a novel method for the objective assessment of technical skills in lower gastrointestinal (GI) endoscopy that incorporates qualitative as well as quantitative measures. We developed a virtual endoscopy suite by deconstructing the VR simulator to enable a more realistic and ergonomic placement of the monitor. Trainee endoscopists with varying levels of experience performed the case 4 task of the simulator. Ten essential components of endoscopic performance were rated on a five-point Likert scale (global score) by three independent observers. These observers viewed two videos (one showing scope handling and the other showing the monitor image), which were synchronized and played simultaneously. The study population comprised six experts (group 1, > 200 procedures), seven subjects with intermediate experience (group 2, 20-80 procedures), and seven novices (group 3, 1-10 procedures). The global score was found to discriminate the level of skills across all three groups (p < 0.001). There were significant differences between groups 1 and 2 (p = 0.003) and groups 2 and 3 (p = 0.004). There was a significant correlation between the global score and the percentage of red-out (without vision) as recorded by the simulator (correlation coefficients = -0.61, p = 0.004). This novel method for the assessment of technical skills in lower GI endoscopy has construct validity and high interrater reliability.

EDITOR, - There is increasing pressure on the royal colleges to introduce some form of accreditation and assessment of practical skills in minimally invasive surgery. This is partly driven by attention in the media to cases that have had an unfavourable outcome. There has never, however, been a demand for such accreditation in general surgical skills. Theoretically, a doctor who has never performed surgery can obtain a fellowship of the Royal College of Surgeons or membership of the Royal College of Obstetricians and Gynaecologists. Yet there are no calls for assessment of the technical competence of general surgeons or gynaecologists. Perhaps it would be more appropriate if assessment of technical skill in surgical specialties generally, rather than just in innovations such as minimally invasive surgery, was addressed. C M S Royston and colleagues discuss different modes of training in laparoscopic surgery and suggest that the porcine model is useful.1 They do not make clear …

The clinical success of the percutaneous coronary intervention (PCI) is highly dependent on endovascular manipulation skills and dexterous manipulation strategies of interventionalists. However, the analysis of endovascular manipulations and related discussion for technical skill assessment are limited. In this study, a multilayer and multimodal-fusion architecture is proposed to recognize six typical endovascular manipulations. The synchronously acquired multimodal motion signals from ten subjects are used as the inputs of the architecture independently. Six classification-based and two rule-based fusion algorithms are evaluated for performance comparisons. The recognition metrics under the determined architecture are further used to assess technical skills. The experimental results indicate that the proposed architecture can achieve the overall accuracy of 96.41%, much higher than that of a single-layer recognition architecture (92.85%). In addition, the multimodal fusion brings significant performance improvement in comparison with single-modal schemes. Furthermore, the K -means-based skill assessment can obtain an accuracy of 95% to cluster the attempts made by different skill-level groups. These hopeful results indicate the great possibility of the architecture to facilitate clinical skill assessment and skill learning.

Aims Excellent surgical technical skills are of paramount importance to perform surgical procedures, safely and efficiently. Virtual reality surgical simulators can both simulate real operations while providing standardized, risk-free surgical hands-on experience. The integration of AI (artificial intelligence) and virtual reality simulators provides opportunities to carry out comprehensive continuous assessments of surgical performance. We developed and tested a deep learning algorithm which can continuously monitor and assess surgical bimanual performance on virtual reality surgical simulators. Methods Fifty participants from four expertise levels (14 experts/neurosurgeons, 14 senior residents, 10 junior residents, 12 medical students) performed a simulated subpial tumor resection 5 times and a complex simulated brain tumor operation once on the NeuroVR platform. Participants were asked to remove the tumors completely while minimizing bleeding and damage to surrounding tissues employing a simulated ultrasonic aspirator and bipolar forceps. A deep neural network continually tracked the surgical performance utilizing 16 performance metrics generated every 0.2-seconds. Results The deep neural network was successfully trained using neurosurgeons and medical students’ data, learning the composites of expertise comparing high and lower skill levels. The trained algorithm was able to score the technical skills of individuals continuously at 0.2-second intervals. Statistically significant differences in average scores were identified between the 4 groups. Conclusions AI-powered surgical simulators provide continuous assessment of bimanual technical skills during surgery which may further define the composites necessary to train surgical expertise. To our knowledge, this is the first attempt in surgery to continuously assess surgical technical skills using deep learning.

Background Simulation training has the potential to increase resident confidence in basic procedural tasks prior to operating room (OR) refinements, offsets a reduction in mentoring opportunities, avoids the use of patients for skills practice, and facilitates objective assessment of technical skills. Despite a multitude of benefits, adoption of simulation training and assessment in breast surgical oncology has not been forthcoming. Aims To develop and validate a high-fidelity surgical simulator for the practice, training and assessment of technical skills in wide local excision (WLE) of a palpable breast tumour. Methods Subjects (attendings and residents with an interest in breast surgery, and novice trainees) were invited to perform a WLE of 25mm palpable breast lesion located 30mm from the nipple areolar complex in the 3o clock position, on a synthetic breast simulator developed at Imperial College London as part of the London Deanery Skills Programme. Procedures were videotaped (blind) and were retrospectively reviewed and independently rated against procedure-specific ratings of performance (VAS 0-100) by two expert breast surgeons (&amp;gt;10 years experience). Specimen radiographs were performed (BioVision, Faxitron, USA) and margin of clearance (mm) were calculated from the edge of the “tumour” to the limits of surrounding breast tissue excised in 4 cardinal directions (i.e. N, S, E W). Specimen weights were recorded (g). Subjects completed a comprehensive questionnaire to determine simulator content and face validity. Results 21 subjects participated (5 attendings, 13 registrars, and 3 junior trainees). Data was analysed according to experience (high = &amp;gt;100 independent wide local procedures; low = no independent wide local procedures). Statistically significant (p ≤ 0.05) differences in performance were observed between high and low experience surgeons [for each category data are VAS score medians±IQR; ‘exposure’: low = 70.0±50.0, high = 80.0±20.0; ‘skin flap development’: low = 70.0±50.0, high = 80.0±40.0; ‘resection skills’: low = 60.0±55.0, high = 80.0±45.0; ‘glandular remodelling’: low = 70.0±25.0; high = 80.0±50.0, ‘skin closure’: low = 70.0±25.0, high = 90.0±20.0]. Resection margin width and specimen weight (MWU = 52.5, p = 1.0) did not discriminate high-low experience surgeons. The majority of participants believed that training on the model simulated the same steps as a real WLE (94.7%) and was useful for real practice (84.2%). Discussion Preliminary data on a WLE simulator suggests that the model is face and construct valid, and may be useful to supplement early stage practice prior to skills development in the OR. Video performance ratings whilst time consuming appear to distinguish high-low experience surgeons better than do specimen weight or radiographic margin width. Citation Information: Cancer Res 2013;73(24 Suppl): Abstract nr P2-18-18.

Technical Skill Assessment using Machine Learning and Artificial Intelligence Algorithm - written by Mohammad Suhaib Ahmed , Nikhunj Saraf , Rohan Atale published on 2019/12/14 download full article with reference data and citations

To implement the Colorectal Objective Structured Assessment of Technical skill (COSATS) into American Board of Colon and Rectal Surgery (ABCRS) certification and build evidence of validity for the interpretation of the scores of this high stakes assessment tool. Currently, technical skill assessment is not a formal component of board certification. With the technical demands of surgical specialties, documenting competence in technical skill at the time of certification with a valid tool is ideal. In September 2014, the COSATS was a mandatory component of ABCRS certification. Seventy candidates took the examination, with their performance evaluated by expert colorectal surgeons using a task-specific checklist, global rating scale, and overall performance scale. Passing scores were set and compared using 2 standard setting methodologies, using a compensatory and conjunctive model. Inter-rater reliability and the reliability of the pass/fail decision were calculated using Cronbach alpha and Subkoviak methodology, respectively. Overall COSATS scores and pass/fail status were compared with results on the ABCRS oral examination. The pass rate ranged from 85.7% to 90%. Inter-rater reliability (0.85) and reliability of the pass/fail decision (0.87 and 0.84) were high. A low positive correlation (r= 0.25) was seen between the COSATS and oral examination. All individuals who failed the COSATS passed the ABCRS oral examination. COSATS is the first technical skill examination used in national surgical board certification. This study suggests that the current certification process may be failing to identify individuals who have demonstrated technical deficiencies on this standardized assessment tool.

Road to automating robotic suturing skills assessment: Battling mislabeling of the ground truth

654 Procedural performance and assessment of technical skills in percutaneous renal access: using an assessment tool in virtual reality perc-mentor trainer

The assessment of technical skills in ultrasound-guided regional anesthesia is currently subjective and relies largely on observations of the trainer. The objective of this study was to develop a checklist to assess training progress and to detect training gaps in ultrasound-guided regional anesthesia using the Delphi method. A 30-item checklist was developed and then e-mailed to 18 reviewers for feedback. The checklist was modified on the basis of their feedback. This process of iteration was repeated until no further feedback was received, and a consensus was reached on the final composition of the checklist. A global rating scale (GRS) was introduced as a result of the feedback. Three rounds of feedback were required to reach consensus on the composition of the checklist and the GRS. The final checklist contains 22 items, and the GRS contains 9 categories. Using the Delphi method, a checklist and GRS were developed. These tools can serve as an objective means of assessing progress in ultrasound technical skills acquisition.

Introduction: Novel training modalities are being investigated to overcome the challenges associated with learning retrograde intrarenal surgery (RIRS). Consequently, a series of 3D printed models of the upper urinary tract and stones designed for ex vivo surgical simulation was introduced in 2021. This study aims to provide external validation of the training model and assess its role in the development of surgical skills. Materials and Methods: A mixed cohort of 20 urologists at different levels of expertise participated in a whole-day live simulation event to examine the model and perform a timed simulation of intrarenal navigation, stone relocation, and laser fragmentation. Operative times were recorded and two independent expert endourologists scored the simulations according to a modified "Objective Structured Assessment of Technical Skills" (OSATS) scale. Five novice urologists from the cohort performed three further simulations in a subsequent event to assess improvement in surgical skills. Results: Face validity was demonstrated with a median score of ≥4/5 in each of the 11 items investigated. Content validity was also effectively reached, with 100% positive impressions with regard to the usefulness for the acquisition of surgical skills. Significant differences were observed among operative times stratified per surgeon experience (all p < 0.0050), thus providing construct validity. Median total OSATS score for novices was 14 (range 8, 25) and was found to be significantly different from expected expert performance (p = 0.0010). Repeated simulations by novices led to a progressive reduction of operative times (p = 0.0313) and increase in median total OSATS (p = 0.0625). Conclusion: The 3D printed models of upper urinary tract and synthetic training stones for the high-fidelity simulation of each phase of RIRS were validated by this study. The results encourage the usage of the models in simulation courses and the evaluation of their potential role in standardized training curricula.

Simulation to become a better neurosurgeon. An international prospective controlled trial: The Passion study