Model for simulated training of laparoscopic gastroenterostomy.

In order to construct and evaluate the validity argument for the Objective Structured Assessment of Technical Skills (OSATS), based on Kane's framework, we conducted a systematic review. We searched MEDLINE, EMBASE, CINAHL, PsycINFO, ERIC, Web of Science, Scopus, and selected reference lists through February 2013. Working in duplicate, we selected original research articles in any language evaluating the OSATS as an assessment tool for any health professional. We iteratively and collaboratively extracted validity evidence from included articles to construct and evaluate the validity argument for varied uses of the OSATS. Twenty-nine articles met the inclusion criteria, all focussed on surgical technical skills assessment. We identified three intended uses for the OSATS, namely formative feedback, high-stakes assessment and program evaluation. Following Kane's framework, four inferences in the validity argument were examined (scoring, generalization, extrapolation, decision). For formative feedback and high-stakes assessment, there was reasonable evidence for scoring and extrapolation. However, for high-stakes assessment there was a dearth of evidence for generalization aside from inter-rater reliability data and an absence of evidence linking multi-station OSATS scores to performance in real clinical settings. For program evaluation, the OSATS validity argument was supported by reasonable generalization and extrapolation evidence. There was a complete lack of evidence regarding implications and decisions based on OSATS scores. In general, validity evidence supported the use of the OSATS for formative feedback. Research to provide support for decisions based on OSATS scores is required if the OSATS is to be used for higher-stakes decisions and program evaluation.

To assess the learning curve of pancreaticojejunostomy during robotic pancreatoduodenectomy (RPD) and to predict the risk of postoperative pancreatic fistula (POPF) by using the objective structured assessment of technical skills (OSATS), taking the fistula risk into account. RPD is a challenging procedure that requires extensive training and confirmation of adequate surgical performance. Video grading, modified for RPD, of the pancreatic anastomosis could assess the learning curve of RPD and predict the risk of POPF. Post hoc assessment of patients prospectively included in 4 Dutch centers in a nationwide LAELAPS-3 training program for RPD. Video grading of the pancreaticojejunostomy was performed by 2 graders using OSATS (attainable score: 12-60). The main outcomes were the combined OSATS of the 2 graders and POPF (grade B/C). Cumulative sum analyzed a turning point in the learning curve for surgical skill. Logistic regression determined the cutoff for OSATS. Patients were categorized for POPF risk (ie, low, intermediate, and high) based on the updated alternative fistula risk scores. Videos from 153 pancreatic anastomoses were included. Median OSATS score was 48 (interquartile range: 41-52) points and with a turning point at 33 procedures. POPF occurred in 39 patients (25.5%). An OSATS score below 49, present in 77 patients (50.3%), was associated with an increased risk of POPF (odds ratio: 4.01, P =0.004). The POPF rate was 43.6% with OSATS < 49 versus 15.8% with OSATS ≥49. The updated alternative fistula risk scores category "soft pancreatic texture" was the second strongest prognostic factor of POPF (odds ratio: 3.37, P =0.040). Median cumulative surgical experience was 17 years (interquartile range: 8-21). Video grading of the pancreatic anastomosis in RPD using OSATS identified a learning curve and a reduced risk of POPF in case of better surgical performance. Video grading may provide a valid method to surgical training, quality control, and improvement.

Plaster cast removal simulation model for resident training.

Construct Validity of a Simple Laparoscopic Ovarian Cystectomy Model Using a Validated Objective Structured Assessment of Technical Skills

Measuring surgical competency is essential for surgical residents to ensure patient safety. Traditional assessment tools rely on subjective evaluation. This study evaluated whether artificial intelligence (AI)-based hand tracking can more objectively distinguish between levels of surgical competency and predict surgical years of experience versus traditional assessments. A total of 44 participants, including medical students, surgical residents, and surgical consultants, performed transcutaneous suturing, intracutaneous suturing, and surgical knot tying. Videos of intracutaneous suturing were scored using the objective structured assessment of technical skills (OSATS). Hand movements were analysed using AI tracking software to extract coordinates to measure velocity, pathlength, and jerk. Linear regression models predicted experience years using procedural time and OSATS in combination with hand tracking metrics. Hand tracking metrics varied mainly between medical students and more experienced groups. Traditional assessment tools (procedural time, OSATS) could predict experience years during training, with an adjusted coefficient of determination (R2) ranging from 0.537 to 0.638, dependent on procedure type. Hand tracking variables identified multiple significant predictors for years of experience, with an adjusted R2 of 0.540-0.712, which outperformed the traditional tools in each procedure. Combining all assessment tools (time, OSATS, and hand tracking) gave the best predictive value, with an adjusted R2 ranging from 0.540 to 0.809, with velocity, pathlength, jerk, and acceleration as significant predictors. AI-based hand tracking provides a new method for objective, reproducible measures of surgical skills. Incorporating hand tracking metrics enhances prediction of surgical experience, and supports standardized as well as objective evaluation of skills assessment in surgical training.

Background Current methods of surgeon performance assessment are unstructured, labour-intensive and subjective. Machine learning could provide rapid, automated and subjective assessment. We design a custom deep learning model to output performance metrics from videos of LSG and validate these against OSATS and clinical outcomes. Methods A novel dataset of 3210 images from videos of LSG was annotated to train an instrument segmentation framework based on Mask R-CNN, a state-of-the-art deep learning algorithm. Spatial outputs of detected instruments were used to determine workflow through a Markov chain and secondly to determine instrument metrics for assessment of surgical performance. Metrics including time spent in stage and instrument time, distance, smoothness, concentration, and efficiency were then validated against OSATS ratings from 2 independent scorers and clinical outcomes from 35 LSGs. Results Grasper and ligasure efficiency and smoothness were higher in videos rated within the top quartile by OSATS score when compared to those in the bottom quartile and when comparing consultants to trainees. Videos in the top quartile for grasper smoothness had higher total OSATS scores (17.31±2.27) compared to those in the lowest quartile (15.69±2.93) and across all 5 individual OSATS domains in addition to higher percentage of total weight loss at 3, 6, and 12 months. Conclusions This study demonstrates how vision based deep learning can be used for surgeon performance assessment. Future work will aim to validate this against a large number of videos, expand upon the range of metrics produced, and look towards the steps needed for clinical translation.

BackgroundTechnical proficiency of the operating surgeons is one of the most important factors in the safe performance of pancreaticoduodenectomy. The objective of the present study was to investigate whether surgical simulation of pancreatico-jejunostomy (PJ) using an inanimate biotissue model could improve the technical proficiency of hepato-biliary pancreatic (HBP) surgical fellows.MethodsThe biotissue drill consisted of sewing biotissues to simulate PJ. The drill was repeated a total of five times by each of the participant surgical fellows. The improvement of the surgical fellows’ technical proficiency was evaluated by the supervisor surgeons using the Objective Structured Assessment of Technical Skills (OSATS) scale.ResultsEight HBP surgical fellows completed all the 5 drills. Both the OSATS 25 score and OSATS summary score, assessed by the two supervisor surgeons, improved steadily with repeated execution of the PJ drill. The average OSATS score, as assessed by both the supervisor surgeons, improved significantly from the first to the final drill, with a P value of 0.003 and 0.014 for the assessment by the two surgeons, respectively. On the other hand, no chronological alteration was observed in time of procedure (P = 0.788).ConclusionRepeated execution of a biotissue PJ drill improved the HBP surgical fellows’ technical proficiency, as evaluated by OSATS. The present study lends support to the evidence that simulation training can contribute to shortening of the time required to negotiate the learning curve for the technique of PJ in the actual operating room.

Objective: Evaluate the acquired surgical suture skills in veterinary medicine students after one-day training using new designed interactive surgical suture training (ISST) model. Methodology: 150 students were subjected to one-day surgical suture training. An objective structured assessment of technical skills (OSATS) task-specific checklist of 20 tasks was performed by three faculty members. A questionnaire was completed by each student. Results: tasks like square knot using hand tie, appropriate tension of knotting, correct edge distance, and equal spacing between sutures were well performed by final year students significantly. For other tasks, the final year student got better scores without distinct differences (P > 0.05). More than 90% of students confirmed the easy use of the ISST model and it simulated the real training. They strongly recommended the ISST training model. Conclusions: one-day surgical suture training was very suitable for undergraduate students; also the ISST model was convenient and easy to use during training. The OSATS’s checklist and the questionnaire were a reasonable way of objective assessment of surgical suture training. We suggest training surgical sutures on the ISST model and using the OSATS checklist with a questionnaire for training evaluation to all medical students and trainees.

The success of Pavlik harness therapy is based, among other things, upon the correct reapplication by parents. This prospective study assesses the effectiveness of a validated Objective Structured Assessment of Technical Skill (OSATS) educational module for Pavlik harness application among Greek parents. Forty consecutive parents of newborns with developmental dysplasia of the hip (DDH), who were treated with Pavlik harness, were divided into two groups. Twenty parents (group 1) were educated with the Greek OSATS version, while the rest (group 2) served as the control. The OSATS score, quality of the final product (Global Rating Scale - GRS) and sonographic imaging of the hips were evaluated during follow-up. Post-intervention evaluation revealed significant higher means of OSATS score (p<0.01), GRS of performance (p<0.05) and GRS of final product (p<0.05) for group 1. At every retention time point, OSATS was signifi cantly higher for group 1, however there were no significant differences between either GRS score between groups. There was no significant difference in radiographic evaluation and successful outcome. This educative module increased the parents' skill level on application of Pavlik harness. However, this module revealed no significant effect in clinical and radiological evaluation of the hips, in this population.

Testing technical skill via an innovative “bench station” examination

Background: It is frequently suggested that minimally invasive surgery (MIS) is harder to acquire than conventional surgery. To test this hypothesis, residents’ learning curves of both surgical skills are compared. Methods: Residents had to be assessed using a general global rating scale of the OSATS (Objective Structured Assessment of Technical Skills) for every procedure they performed as primary surgeon during a 3-month clinical rotation in gynecological surgery. Results: Nine postgraduate-year-4 residents collected a total of 319 OSATS during the 2 years and 3 months investigation period. These assessments concerned 129 MIS (laparoscopic and hysteroscopic) and 190 conventional (open abdominal and vaginal) procedures. Learning curves (in this study defined as OSATS score plotted against procedure-specific caseload) for MIS and conventional surgery were compared using a linear mixed model. The MIS curve revealed to be steeper than the conventional curve (1.77 vs. 0.75 OSATS points per assessed procedure; 95% CI 1.19–2.35 vs. 0.15–1.35, p < 0.01). Conclusions: Basic MIS procedures do not seem harder to acquire during residency than conventional surgical procedures. This may have resulted from the incorporation of structured MIS training programs in residency. Hopefully, this will lead to a more successful implementation of the advanced MIS procedures.

To evaluate a previously validated low-cost sinus surgery task trainer as a means of acquiring basic endoscopic sinus surgery skills and as an objective structured assessment of technical skills (OSATS) to determine procedural competency. Prospective blinded study. Medical students (N = 52) with no sinus surgery experience learned to perform nasal endoscopy and five specific sinus surgery tasks using the validated task trainer. Training included regimented expert instruction, peer instruction/observation, and experienced-based learning. Pre- and post-training video recordings of nasal endoscopy and five sinus surgery skills were obtained. Two blinded expert otolaryngologists compared pre- and post-training performance using a checklist and global rating scale. Medical student post-training performance was significantly better than pre-training performance for each checklist item and global rating scale as calculated by paired t test (P < .001). Interrater reliability and internal consistency were confirmed by Kendall's coefficient of concordance and Cronbach's α calculations, respectively. The sinus surgery task trainer provides an effective means of teaching and evaluating nasal endoscopy and basic sinus surgery skills for novice surgeons. With repeated practice, there was significant improvement in performance. An OSATS using the sinus surgery task trainer suggests that we can effectively measure endoscopic sinus surgery ability with the potential to reliably determine competency outside the operating room.

IntroductionUpdating the method for evaluating suturing and scaling skills in dental education has attracted relatively little attention and there is no consensus to what should be assessed and how. The purpose of this study was to investigate the applicability of the Objective Structured Assessment of Technical Skill (OSATS) method for these two basic skills, the possible association between the scores and demographic factors, and the level of satisfaction of residents with this method.MethodsAll six periodontics and three oral medicine residents were recruited by census method and video-recorded while performing a simple interrupted suture, a figure eight suture and scaling on a model. Then, the videos were evaluated independently via a checklist and a global rating scale (GRS) by two expert raters. Agreement between raters and residents’ satisfaction were evaluated. Correlation between demographic factors of participants and scores was also assessed. T-test and linear regression analysis were used.ResultsThere was no significant difference between the scores based on the views of the two raters for each of the checklist (ICC = 0.99, CI = 0.96–0.99, P < 0.001) and GRS (ICC = 0.97, CI = 0.86–0.99, P < 0.001). Linear regression showed no correlation between gender and scores but periodontics major and higher year of education showed correlation with higher scores.ConclusionConsidering the excellent agreement between raters in using both the checklist and GRS components of OSATS, and satisfaction of 88% the residents with this method, it seems to be able to provide a reliable assessment.

The Effect of Stress on Task Acquisition in a Simulated Open Aortic Aneurysm Model

Background: Objective structured assessment of technical skills (OSATS) are crucial in assessing operative skills. The education of basic surgical skill (BSS) depends on precise assessment of their surgical skill levels and competency level. Hence, this study aimed to determine the effectiveness of BSS module on intern's competency and confidence. Materials and Methods: Fifty-four interns were included in the module. Three faculty members were included in the module. The module has been prepared on: (1) Asepsis – surgical hand scrubbing, gloving, gowning. (2) Simple interrupted suturing. Intervention: Small group teaching was given to each batch, and hands-on training in dry skill laboratory on simulation was given. Interns were assessed by OSATS, and feedbacks were taken at the end of the assessment. Results: The score of global rating scale for each trainee increased within the 15 days of posting. The knowledge domain of posttest score also improved. Conclusion: BSS module was found to be effective, aiding for the interns to improve their knowledge and skills.