Robotic dismembered pyeloplasty surgical simulation using a 3D-printed silicone-based model: development, face validation and crowdsourced learning outcomes assessment.

Benchmarks for 3-D Systems (Simbionix) Bladder Flap Module for the Xi Robot: Differentiating Novice from Experienced and Expert Surgeons

BackgroundThis study aimed to evaluate the use of binary metric-based (proficiency-based progression; PBP) performance assessments and global evaluative assessment of robotic skills (GEARS) of a robotic-assisted low anterior rectal resection (RA-LAR) procedure.MethodA prospective study of video analysis of RA-LAR procedures was carried out using the PBP metrics with binary parameters previously developed, and GEARS. Recordings were collected from five novice surgeons (≤30 RA-LAR previously performed) and seven experienced surgeons (>30 RA-LAR previously performed). Two consultant colorectal surgeons were trained to be assessors in the use of PBP binary parameters to evaluate the procedure phases, surgical steps, errors, and critical errors in male and female patients and GEARS scores. Novice and experienced surgeons were categorized and assessed using PBP metrics and GEARS; mean scores obtained were compared for statistical purpose. Also, the inter-rater reliability (IRR) of these assessment tools was evaluated.ResultsTwenty unedited recordings of RA-LAR procedures were blindly assessed. Overall, using PBP metric-based assessment, a subgroup of experienced surgeons made more errors (20 versus 16, P = 0.158) and critical errors (9.2 versus 7.8, P = 0.417) than the novice group, although not significantly. However, during the critical phase of RA-LAR, experienced surgeons made significantly fewer errors than the novice group (95% CI of the difference, Lower = 0.104 – Upper = 5.155, df = 11.9, t = 2.23, p = 0.042), and a similar pattern was observed for critical errors. The PBP metric and GEARS assessment tools distinguished between the objectively assessed performance of experienced and novice colorectal surgeons performing RA-LAR (total error scores with PBP metrics, P = 0.019–0.008; GEARS scores, P = 0.029–0.025). GEARS demonstrated poor IRR (mean IRR 0.49) and weaker discrimination between groups (15–41 per cent difference). PBP binary metrics demonstrated good IRR (mean 0.94) and robust discrimination particularly for total error scores (58–64 per cent).ConclusionsPBP binary metrics seem to be useful for metric-based training for surgeons learning RA-LAR procedures.

Virtual Reality Simulation Has Weak Correlation with Overall Trainee Robot-Assisted Laparoscopic Hysterectomy Performance

Crowdsourcing from the general population is an efficient, inexpensive method of surgical performance evaluation. In this study, we compared the discriminatory ability of experts and crowdsourced evaluators (the Crowd) to detect differences in robotic automated performance metrics (APMs). APMs (instrument motion tracking and events data directly from the robot system) of anterior vesico-urethral anastomoses (VUAs) of robotic radical prostatectomies were captured by the dVLogger (Intuitive Surgical). Crowdsourced evaluators and four expert surgeons evaluated video footage using the Global Evaluative Assessment of Robotic Skills (GEARS) (individual domains and total score). Cases were then stratified into performance groups (high versus low quality) for each evaluator based on GEARS. APMs from each group were compared using the Mann-Whitney U test. 25 VUAs performed by 11 surgeons were evaluated. The Crowd displayed moderate correlation with averaged expert scores for all GEARS domains (r > 0.58, p < 0.01). Bland-Altman analysis showed a narrower total GEARS score distribution by the Crowd compared to experts. APMs compared amongst performance groups for each evaluator showed that through GEARS scoring, the most common differentiated metric by evaluators was the velocity of the dominant instrument arm. The Crowd outperformed two out of four expert evaluators by discriminating differences in three APMs using total GEARS scores. The Crowd assigns a narrower range of GEARS scores compared to experts but maintains overall agreement with experts. The discriminatory ability of the Crowd at discerning differences in robotic movements (via APMs) through GEARS scoring is quite refined, rivaling that of expert evaluators.

The objective was to study the effect of immediate pre-operative warm-up using virtual reality simulation on intraoperative robot-assisted laparoscopic hysterectomy (RALH) performance by gynecology trainees (residents and fellows). We randomized the first, non-emergent RALH of the day that involved trainees warming up or not warming up. For cases assigned to warm-up, trainees performed a set of exercises on the da Vinci Skills Simulator immediately before the procedure. The supervising attending surgeon, who was not informed whether or not the trainee was assigned to warm-up, assessed the trainee's performance using the Objective Structured Assessment for Technical Skill (OSATS) and the Global Evaluative Assessment of Robotic Skills (GEARS) immediately after each surgery. We randomized 66 cases and analyzed 58 cases (30 warm-up, 28 no warm-up), which involved 21 trainees. Attending surgeons rated trainees similarly irrespective of warm-up randomization with mean (SD) OSATS composite scores of 22.6 (4.3; warm-up) vs 21.8 (3.4; no warm-up) and mean GEARS composite scores of 19.2 (3.8; warm-up) vs 18.8 (3.1; no warm-up). The difference in composite scores between warm-up and no warm-up was 0.34 (95% CI: -1.44, 2.13), and 0.34 (95% CI: -1.22, 1.90) for OSATS and GEARS respectively. Also, we did not observe any significant differences in each of the component/subscale scores within OSATS and GEARS between cases assigned to warm-up and no warm-up. Performing a brief virtual reality-based warm-up before RALH did not significantly improve the intraoperative performance of the trainees.

Comparison of Robotic Surgery Video Quality Between YouTube and Curated Sources Using GEARS Criteria

A gold standard in surgical skill rating and evaluation is direct observation, which a group of experts rate trainees based on a likert scale, by observing their performance during a surgical task. This method is time and resource intensive. To alleviate this burden, many studies have focused on automatic surgical skill assessment; however, the metrics suggested by the literature for automatic evaluation do not capture the stylistic behavior of the user. In addition very few studies focus on automatic rating of surgical skills based on available likert scales. In a previous study we presented a stylistic behavior lexicon for surgical skill. In this study we evaluate the lexicon's ability to automatically rate robotic surgical skill, based on the 6 domains in the Global Evaluative Assessment of Robotic Skills (GEARS). 14 subjects of different skill levels performed two surgical tasks on da Vinci surgical simulator. Different measurements were acquired as subjects performed the tasks, including limb (hand and arm) kinematics and joint (shoulder, elbow, wrist) positions. Posture videos of the subjects performing the task, as well as videos of the task being performed were viewed and rated by faculty experts based on the 6 domains in GEARS. The paired videos were also rated via crowd-sourcing based on our stylistic behavior lexicon. Two separate regression learner models, one using the sensor measurements and the other using crowd ratings for our proposed lexicon, were trained for each domain in GEARS. The results indicate that the scores predicted from both prediction models are in agreement with the gold standard faculty ratings.

Transoral robotic surgery (TORS) has become a well-established surgical technique for the treatment of oropharyngeal cancer, but the significant learning curve and lack of standardized credentialing have resulted in wide variability in surgical outcomes. This study aims to define procedure-specific competence standards for TORS and test whether a hierarchical task analysis (HTA)-derived procedure-based assessment (PBA) distinguishes experience levels. We examined the ability of PBA and Global Evaluative Assessment of Robotic Skills (GEARS) scores to discriminate between novice and experienced surgeons and to assess their association with operative efficiency and margin quality. We built an HTA by deconstructing the TORS lateral oropharyngectomy into tasks and subtasks. Then, PBA metrics for mucosal incision and deep dissection were developed. Two independent raters scored 40 porcine tongue TORS videos (20 novice, 20 experienced) using PBA and GEARS and recorded global and phase times. Experienced surgeons scored higher on total PBA (39.98 vs 35.35, p = 0.0055) and GEARS (22.60 vs 19.63, p = 0.0009) and showed less score variability. The largest gaps were lateral tasks: lateral mucosal incision 4.65 vs 3.60 (p = 0.0015) and lateral deep dissection 4.58 vs 3.85 (p = 0.0115). Margin scores were higher in experienced surgeons (4.38 vs 3.80, p = 0.0149). Procedures were faster overall (298.47s vs 466.43s, p = 0.0003) with shorter mucosal incision and deep dissection times. An HTA-derived PBA reliably differentiates TORS expertise, aligns with speed and margin quality, and identifies lateral tasks as high-yield training targets. These metrics support standardized training, assessment, and integration into VR simulation for competency-based credentialing.

Robotic surgery has seen increasing use in the field of pediatric surgery. Our clinical experience suggested instrument size can impact on the surgical ability. This study aimed to compare the performance of robot-assisted laparoscopic skills in confined spaces using either 5 or 8mm instruments. A preclinical randomized crossover study design was implemented. 24 assessors performed three different reproducible drill procedures (M1: peg transfer, M2: circle cutting, M3: intracorporeal suturing). To assess surgical proficiency in confined working spaces, these exercises were performed with 5 and 8mm instruments of the da Vinci® Surgical Systems Si in a cubic box with 60mm-sized edges. Each performance was recorded and evaluated by two reviewers using both objective structured assessment of technical skills (OSATS) and global evaluative assessment of robotic skills (GEARS) scores. Parietal iatrogenic impacts and instrument collisions were specifically analyzed using a dedicated scoring system. Regardless of their experience, trainees performed significantly better when using 8mm instruments in terms of OSATS scores (20.5 vs. 18.4; p<0.01) and GEARS scores (23.4 vs. 21.9; p<0.01) for most items, except for "depth perception" and "autonomy." The 8mm performances involved significantly less parietal box damage (4.1 vs. 3.4; p<0.01), and tool collisions (4.1 vs. 3.2; p<0.01). In light of the better performances with 8mm tools for specific tasks and parietal sparing constraints in restricted spaces, this study indicates that 5mm instruments can be deemed to be less effective for reconstructive procedures in small children.

In recent years, crowd-sourced assessments have been reported as a timesaving, cost-efficient, and practicable method of surgical skill evaluation. However, the differences in the assessment of surgical skills by the individual reviewers cannot be further examined in terms of characteristics of the reviewers because they are usually anonymously and randomly selected. This study aimed to reveal the effects of reviewers' occupations on their assessment of a surgeon's skill. In total, 42 urologists, 19 paramedics, 73 medical students, and 28 non-medical personnel used the Global Evaluative Assessment of Robotic Skills (GEARS) validated robotic surgery rating tool to assess the surgical skill of surgeons in nine edited video clips of complete urethrovesical anastomosis during a robot-assisted radical prostatectomy. The total GEARS scores of the four groups of reviewers were compared, and the similarities and the differences between the ratings of the urologists group and those of the other three groups were subsequently investigated. The rankings of video clips in the order of GEARS scores were very similar in each group, and a strong positive correlation (R2 values >0.8) was observed between the scores assigned by the urologists group and those assigned by the other three groups. Our findings indicate that the crude evaluation of robot-assisted urethrovesical anastomosis is not affected by the reviewers' occupations. Non-medical personnel may be able to provide a rudimentary screening evaluation of surgical skill.

Introduction/BackgroundAdvanced surgery in oncological settings is often grouped for analysis based on morbidity and mortality data (MMD). However, MMD has not been able to reduce adverse clinical outcomes, because it...

Transferability of Virtual Reality, Simulation-Based, Robotic Suturing Skills to a Live Porcine Model in Novice Surgeons: A Single-Blind Randomized Controlled Trial

To incorporate and validate clinically relevant performance metrics of simulation (CRPMS) into a hydrogel model for nerve-sparing robot-assisted radical prostatectomy (NS-RARP). Anatomically accurate models of the human pelvis, bladder, prostate, urethra, neurovascular bundle (NVB) and relevant adjacent structures were created from patient MRI by injecting polyvinyl alcohol (PVA) hydrogels into three-dimensionally printed injection molds. The following steps of NS-RARP were simulated: bladder neck dissection; seminal vesicle mobilization; NVB dissection; and urethrovesical anastomosis (UVA). Five experts (caseload >500) and nine novices (caseload <50) completed the simulation. Force applied to the NVB during the dissection was quantified by a novel tension wire sensor system fabricated into the NVB. Post-simulation margin status (assessed by induction of chemiluminescent reaction with fluorescent dye mixed into the prostate PVA) and UVA weathertightness (via a standard 180-mL leak test) were also assessed. Objective scoring, using Global Evaluative Assessment of Robotic Skills (GEARS) and Robotic Anastomosis Competency Evaluation (RACE), was performed by two blinded surgeons. GEARS scores were correlated with forces applied to the NVB, and RACE scores were correlated with UVA leak rates. The expert group achieved faster task-specific times for nerve-sparing (P = 0.007) and superior surgical margin results (P = 0.011). Nerve forces applied were significantly lower for the expert group with regard to maximum force (P = 0.011), average force (P = 0.011), peak frequency (P = 0.027) and total energy (P = 0.003). Higher force sensitivity (subcategory of GEARS score) and total GEARS score correlated with lower nerve forces (total energy in Joules) applied to NVB during the simulation with a correlation coefficient (r value) of -0.66 (P = 0.019) and -0.87 (P = 0.000), respectively. Both total and force sensitivity GEARS scores were significantly higher in the expert group compared to the novice group (P = 0.003). UVA leak rate highly correlated with total RACE score r value = -0.86 (P = 0.000). Mean RACE scores were also significantly different between novices and experts (P = 0.003). We present a realistic, feedback-driven, full-immersion simulation platform for the development and evaluation of surgical skills pertinent to NS-RARP. The correlation of validated objective metrics (GEARS and RACE) with our CRPMS suggests their application as a novel method for real-time assessment and feedback during robotic surgery training. Further work is required to assess the ability to predict live surgical outcomes.

We evaluated feasibility and benefit of implementing structured learning in a robotics program. Furthermore, we assessed validity of a proficiency assessment tool for stepwise graduation. Teaching cases included robotic radical prostatectomy and partial nephrectomy. Procedure steps were categorized: basic, intermediate, and advanced. An assessment tool ["proficiency score" (PS)] was developed to evaluate ability to safely and autonomously complete a step. Graduation required a passing PS (PS≥3) on three consecutive attempts. PS and validated global evaluative assessment of robotic skills (GEARS) were evaluated for completed steps. Linear regression was utilized to determine postgraduate year/PS relationship (construct validity). Spearman's rank correlation coefficient measured correlation between PS and GEARS evaluations (concurrent validity). Intraclass correlation (ICC) evaluated PS agreement between evaluator classes. Twenty-one robotic trainees participated within the pilot program, completing a median of 14 (2-69) cases each. Twenty-three study evaluators scored 14 (1-60) cases. Over 4months, 229/294 (78%) cases were designated "teaching" cases. Residents completed 91% of possible evaluations; faculty completed 78%. Verbal and quantitative feedback received by trainees increased significantly (p=0.002, p<0.001, respectively). Average PS increased with PGY (post-graduate year) for basic and intermediate steps (regression slopes: 0.402 (p<0.0001), 0.323 (p<0.0001), respectively) (construct validation). Overall, PS correlated highly with GEARS (ρ=0.81, p<0.0001) (concurrent validity). ICC was 0.77 (95% CI 0.61-0.88) for resident evaluations. Structured learning can be implemented in an academic robotic program with high levels of trainee and evaluator participation, encouraging both quantitative and verbal feedback. A proficiency assessment tool developed for step-specific proficiency has construct and concurrent validity.

Education Time Out and Debrief: Structured Implementation of Perioperative Resident Technical Education Discussion