Virtual Reality Simulation Has Weak Correlation with Overall Trainee Robot-Assisted Laparoscopic Hysterectomy Performance

The objective was to study the effect of immediate pre-operative warm-up using virtual reality simulation on intraoperative robot-assisted laparoscopic hysterectomy (RALH) performance by gynecology trainees (residents and fellows). We randomized the first, non-emergent RALH of the day that involved trainees warming up or not warming up. For cases assigned to warm-up, trainees performed a set of exercises on the da Vinci Skills Simulator immediately before the procedure. The supervising attending surgeon, who was not informed whether or not the trainee was assigned to warm-up, assessed the trainee's performance using the Objective Structured Assessment for Technical Skill (OSATS) and the Global Evaluative Assessment of Robotic Skills (GEARS) immediately after each surgery. We randomized 66 cases and analyzed 58 cases (30 warm-up, 28 no warm-up), which involved 21 trainees. Attending surgeons rated trainees similarly irrespective of warm-up randomization with mean (SD) OSATS composite scores of 22.6 (4.3; warm-up) vs 21.8 (3.4; no warm-up) and mean GEARS composite scores of 19.2 (3.8; warm-up) vs 18.8 (3.1; no warm-up). The difference in composite scores between warm-up and no warm-up was 0.34 (95% CI: -1.44, 2.13), and 0.34 (95% CI: -1.22, 1.90) for OSATS and GEARS respectively. Also, we did not observe any significant differences in each of the component/subscale scores within OSATS and GEARS between cases assigned to warm-up and no warm-up. Performing a brief virtual reality-based warm-up before RALH did not significantly improve the intraoperative performance of the trainees.

BackgroundThis study aimed to evaluate the use of binary metric-based (proficiency-based progression; PBP) performance assessments and global evaluative assessment of robotic skills (GEARS) of a robotic-assisted low anterior rectal resection (RA-LAR) procedure.MethodA prospective study of video analysis of RA-LAR procedures was carried out using the PBP metrics with binary parameters previously developed, and GEARS. Recordings were collected from five novice surgeons (≤30 RA-LAR previously performed) and seven experienced surgeons (>30 RA-LAR previously performed). Two consultant colorectal surgeons were trained to be assessors in the use of PBP binary parameters to evaluate the procedure phases, surgical steps, errors, and critical errors in male and female patients and GEARS scores. Novice and experienced surgeons were categorized and assessed using PBP metrics and GEARS; mean scores obtained were compared for statistical purpose. Also, the inter-rater reliability (IRR) of these assessment tools was evaluated.ResultsTwenty unedited recordings of RA-LAR procedures were blindly assessed. Overall, using PBP metric-based assessment, a subgroup of experienced surgeons made more errors (20 versus 16, P = 0.158) and critical errors (9.2 versus 7.8, P = 0.417) than the novice group, although not significantly. However, during the critical phase of RA-LAR, experienced surgeons made significantly fewer errors than the novice group (95% CI of the difference, Lower = 0.104 – Upper = 5.155, df = 11.9, t = 2.23, p = 0.042), and a similar pattern was observed for critical errors. The PBP metric and GEARS assessment tools distinguished between the objectively assessed performance of experienced and novice colorectal surgeons performing RA-LAR (total error scores with PBP metrics, P = 0.019–0.008; GEARS scores, P = 0.029–0.025). GEARS demonstrated poor IRR (mean IRR 0.49) and weaker discrimination between groups (15–41 per cent difference). PBP binary metrics demonstrated good IRR (mean 0.94) and robust discrimination particularly for total error scores (58–64 per cent).ConclusionsPBP binary metrics seem to be useful for metric-based training for surgeons learning RA-LAR procedures.

Crowdsourcing from the general population is an efficient, inexpensive method of surgical performance evaluation. In this study, we compared the discriminatory ability of experts and crowdsourced evaluators (the Crowd) to detect differences in robotic automated performance metrics (APMs). APMs (instrument motion tracking and events data directly from the robot system) of anterior vesico-urethral anastomoses (VUAs) of robotic radical prostatectomies were captured by the dVLogger (Intuitive Surgical). Crowdsourced evaluators and four expert surgeons evaluated video footage using the Global Evaluative Assessment of Robotic Skills (GEARS) (individual domains and total score). Cases were then stratified into performance groups (high versus low quality) for each evaluator based on GEARS. APMs from each group were compared using the Mann-Whitney U test. 25 VUAs performed by 11 surgeons were evaluated. The Crowd displayed moderate correlation with averaged expert scores for all GEARS domains (r > 0.58, p < 0.01). Bland-Altman analysis showed a narrower total GEARS score distribution by the Crowd compared to experts. APMs compared amongst performance groups for each evaluator showed that through GEARS scoring, the most common differentiated metric by evaluators was the velocity of the dominant instrument arm. The Crowd outperformed two out of four expert evaluators by discriminating differences in three APMs using total GEARS scores. The Crowd assigns a narrower range of GEARS scores compared to experts but maintains overall agreement with experts. The discriminatory ability of the Crowd at discerning differences in robotic movements (via APMs) through GEARS scoring is quite refined, rivaling that of expert evaluators.

Comparison of Robotic Surgery Video Quality Between YouTube and Curated Sources Using GEARS Criteria

To incorporate and validate clinically relevant performance metrics of simulation (CRPMS) into a hydrogel model for nerve-sparing robot-assisted radical prostatectomy (NS-RARP). Anatomically accurate models of the human pelvis, bladder, prostate, urethra, neurovascular bundle (NVB) and relevant adjacent structures were created from patient MRI by injecting polyvinyl alcohol (PVA) hydrogels into three-dimensionally printed injection molds. The following steps of NS-RARP were simulated: bladder neck dissection; seminal vesicle mobilization; NVB dissection; and urethrovesical anastomosis (UVA). Five experts (caseload >500) and nine novices (caseload <50) completed the simulation. Force applied to the NVB during the dissection was quantified by a novel tension wire sensor system fabricated into the NVB. Post-simulation margin status (assessed by induction of chemiluminescent reaction with fluorescent dye mixed into the prostate PVA) and UVA weathertightness (via a standard 180-mL leak test) were also assessed. Objective scoring, using Global Evaluative Assessment of Robotic Skills (GEARS) and Robotic Anastomosis Competency Evaluation (RACE), was performed by two blinded surgeons. GEARS scores were correlated with forces applied to the NVB, and RACE scores were correlated with UVA leak rates. The expert group achieved faster task-specific times for nerve-sparing (P = 0.007) and superior surgical margin results (P = 0.011). Nerve forces applied were significantly lower for the expert group with regard to maximum force (P = 0.011), average force (P = 0.011), peak frequency (P = 0.027) and total energy (P = 0.003). Higher force sensitivity (subcategory of GEARS score) and total GEARS score correlated with lower nerve forces (total energy in Joules) applied to NVB during the simulation with a correlation coefficient (r value) of -0.66 (P = 0.019) and -0.87 (P = 0.000), respectively. Both total and force sensitivity GEARS scores were significantly higher in the expert group compared to the novice group (P = 0.003). UVA leak rate highly correlated with total RACE score r value = -0.86 (P = 0.000). Mean RACE scores were also significantly different between novices and experts (P = 0.003). We present a realistic, feedback-driven, full-immersion simulation platform for the development and evaluation of surgical skills pertinent to NS-RARP. The correlation of validated objective metrics (GEARS and RACE) with our CRPMS suggests their application as a novel method for real-time assessment and feedback during robotic surgery training. Further work is required to assess the ability to predict live surgical outcomes.

Design and validation of a low-cost, high-fidelity model for robotic pyeloplasty simulation training

Transferability of Virtual Reality, Simulation-Based, Robotic Suturing Skills to a Live Porcine Model in Novice Surgeons: A Single-Blind Randomized Controlled Trial

Benchmarks for 3-D Systems (Simbionix) Bladder Flap Module for the Xi Robot: Differentiating Novice from Experienced and Expert Surgeons

Robotic surgery has seen increasing use in the field of pediatric surgery. Our clinical experience suggested instrument size can impact on the surgical ability. This study aimed to compare the performance of robot-assisted laparoscopic skills in confined spaces using either 5 or 8mm instruments. A preclinical randomized crossover study design was implemented. 24 assessors performed three different reproducible drill procedures (M1: peg transfer, M2: circle cutting, M3: intracorporeal suturing). To assess surgical proficiency in confined working spaces, these exercises were performed with 5 and 8mm instruments of the da Vinci® Surgical Systems Si in a cubic box with 60mm-sized edges. Each performance was recorded and evaluated by two reviewers using both objective structured assessment of technical skills (OSATS) and global evaluative assessment of robotic skills (GEARS) scores. Parietal iatrogenic impacts and instrument collisions were specifically analyzed using a dedicated scoring system. Regardless of their experience, trainees performed significantly better when using 8mm instruments in terms of OSATS scores (20.5 vs. 18.4; p<0.01) and GEARS scores (23.4 vs. 21.9; p<0.01) for most items, except for "depth perception" and "autonomy." The 8mm performances involved significantly less parietal box damage (4.1 vs. 3.4; p<0.01), and tool collisions (4.1 vs. 3.2; p<0.01). In light of the better performances with 8mm tools for specific tasks and parietal sparing constraints in restricted spaces, this study indicates that 5mm instruments can be deemed to be less effective for reconstructive procedures in small children.

In recent years, crowd-sourced assessments have been reported as a timesaving, cost-efficient, and practicable method of surgical skill evaluation. However, the differences in the assessment of surgical skills by the individual reviewers cannot be further examined in terms of characteristics of the reviewers because they are usually anonymously and randomly selected. This study aimed to reveal the effects of reviewers' occupations on their assessment of a surgeon's skill. In total, 42 urologists, 19 paramedics, 73 medical students, and 28 non-medical personnel used the Global Evaluative Assessment of Robotic Skills (GEARS) validated robotic surgery rating tool to assess the surgical skill of surgeons in nine edited video clips of complete urethrovesical anastomosis during a robot-assisted radical prostatectomy. The total GEARS scores of the four groups of reviewers were compared, and the similarities and the differences between the ratings of the urologists group and those of the other three groups were subsequently investigated. The rankings of video clips in the order of GEARS scores were very similar in each group, and a strong positive correlation (R2 values >0.8) was observed between the scores assigned by the urologists group and those assigned by the other three groups. Our findings indicate that the crude evaluation of robot-assisted urethrovesical anastomosis is not affected by the reviewers' occupations. Non-medical personnel may be able to provide a rudimentary screening evaluation of surgical skill.

This article examines the effectiveness of using virtual reality training environments for procedural training in fourth generation airliners. It is based on a study that assessed whether the training outcomes from a current recurrent training course for FAA certificated airframe and power plant technicians, which used a full flight simulator (FFS) to deliver and assess training, differed from the same training delivered using a virtual reality (VR) device. The study used an experimental design with three groups, and two within-group measures of training effectiveness. The control group followed the current training programme and was assessed in the FFS, while the second group was trained using a VR device and was subsequently assessed in the FFS. Training effectiveness was assessed using a modified Global Evaluative Assessment of Robotic Skills (GEARS) tool that measured both cognitive and psychomotor aspects of learning alongside the time to successful completion of the assessed task was also measured. The population sampled for the study were all Federal Aviation Administration (FAA) certificated airframe and power plant technicians who were engine-run qualified; a total sample of 100 was used to achieve a 95% confidence interval ( p &lt; 0.05). The hypothesis under test was that there is no difference in test performance between the three groups. A multivariate analysis of covariance (MANCOVA) analysis was performed using the GEARS scores and time to completion as variables, and the null hypothesis was retained. The VR system, as tested, was found to provide equivalent task performance to the traditional training method. Recommendations for future research and ongoing application of the specific experimental methodology were provided.

Introduction/BackgroundAdvanced surgery in oncological settings is often grouped for analysis based on morbidity and mortality data (MMD). However, MMD has not been able to reduce adverse clinical outcomes, because it...

Ureteropelvic junction obstruction (UPJO) is an uncommonly encountered pathology, posing a challenge for resident training. We describe the development and face validation of a robotic pyeloplasty simulation using a 3D-printed silicone-based model of UPJO for surgical training, in combination with crowdsourced scoring to objectively assess performance and learning outcomes. The organs were created using 3D modeling software and printed using a silicone-based material by Lazarus 3D, LLC. They were secured in a laparoscopic box trainer and the robotic system was docked. Eight residents and three faculty each performed two robotic-assisted right dismembered pyeloplasties on separate occaisions. Face validity was evaluated on a 5-point Likert scale. Crowd-Sourced Assessment of Technical Skills (C-SATS Inc.) scored surgical performance using the Global Evaluative Assessment of Robotic Skills (GEARS) criteria, based on video review of each simulation. All participants completed the simulation twice with fully patent anastomoses. Average time to complete the first and second trials was 44.4min and 43.2min, respectively. The average GEARS score was 17.1 and 17.6 for the first and second trials respectively. Participants improved on average in all 5 GEARS categories, with significant improvement in depth perception (p = 0.006). The model received mean scores (out of 5) of 4.36 for aesthetics, 4.18 for overall feel, 3.55 for realism, 4.72 for usability, and 4.72 for suturability. Residents had a significant increase in confidence between initial and final surveys on a 5-point Likert Scale: 1.63 vs. 2.38 (p = 0.03). Using 3D-printed silicone-based models, participants completed robotic-assisted dismembered pyeloplasties for training and skill acquisition. We demonstrated face validity of the simulation, which was also found to improve participant speed and significantly improve resident confidence. Crowdsourced assessment demonstrated significant improvement in depth perception.

A gold standard in surgical skill rating and evaluation is direct observation, which a group of experts rate trainees based on a likert scale, by observing their performance during a surgical task. This method is time and resource intensive. To alleviate this burden, many studies have focused on automatic surgical skill assessment; however, the metrics suggested by the literature for automatic evaluation do not capture the stylistic behavior of the user. In addition very few studies focus on automatic rating of surgical skills based on available likert scales. In a previous study we presented a stylistic behavior lexicon for surgical skill. In this study we evaluate the lexicon's ability to automatically rate robotic surgical skill, based on the 6 domains in the Global Evaluative Assessment of Robotic Skills (GEARS). 14 subjects of different skill levels performed two surgical tasks on da Vinci surgical simulator. Different measurements were acquired as subjects performed the tasks, including limb (hand and arm) kinematics and joint (shoulder, elbow, wrist) positions. Posture videos of the subjects performing the task, as well as videos of the task being performed were viewed and rated by faculty experts based on the 6 domains in GEARS. The paired videos were also rated via crowd-sourcing based on our stylistic behavior lexicon. Two separate regression learner models, one using the sensor measurements and the other using crowd ratings for our proposed lexicon, were trained for each domain in GEARS. The results indicate that the scores predicted from both prediction models are in agreement with the gold standard faculty ratings.

Transoral robotic surgery (TORS) has become a well-established surgical technique for the treatment of oropharyngeal cancer, but the significant learning curve and lack of standardized credentialing have resulted in wide variability in surgical outcomes. This study aims to define procedure-specific competence standards for TORS and test whether a hierarchical task analysis (HTA)-derived procedure-based assessment (PBA) distinguishes experience levels. We examined the ability of PBA and Global Evaluative Assessment of Robotic Skills (GEARS) scores to discriminate between novice and experienced surgeons and to assess their association with operative efficiency and margin quality. We built an HTA by deconstructing the TORS lateral oropharyngectomy into tasks and subtasks. Then, PBA metrics for mucosal incision and deep dissection were developed. Two independent raters scored 40 porcine tongue TORS videos (20 novice, 20 experienced) using PBA and GEARS and recorded global and phase times. Experienced surgeons scored higher on total PBA (39.98 vs 35.35, p = 0.0055) and GEARS (22.60 vs 19.63, p = 0.0009) and showed less score variability. The largest gaps were lateral tasks: lateral mucosal incision 4.65 vs 3.60 (p = 0.0015) and lateral deep dissection 4.58 vs 3.85 (p = 0.0115). Margin scores were higher in experienced surgeons (4.38 vs 3.80, p = 0.0149). Procedures were faster overall (298.47s vs 466.43s, p = 0.0003) with shorter mucosal incision and deep dissection times. An HTA-derived PBA reliably differentiates TORS expertise, aligns with speed and margin quality, and identifies lateral tasks as high-yield training targets. These metrics support standardized training, assessment, and integration into VR simulation for competency-based credentialing.