Testing technical skill via an innovative “bench station” examination

In order to construct and evaluate the validity argument for the Objective Structured Assessment of Technical Skills (OSATS), based on Kane's framework, we conducted a systematic review. We searched MEDLINE, EMBASE, CINAHL, PsycINFO, ERIC, Web of Science, Scopus, and selected reference lists through February 2013. Working in duplicate, we selected original research articles in any language evaluating the OSATS as an assessment tool for any health professional. We iteratively and collaboratively extracted validity evidence from included articles to construct and evaluate the validity argument for varied uses of the OSATS. Twenty-nine articles met the inclusion criteria, all focussed on surgical technical skills assessment. We identified three intended uses for the OSATS, namely formative feedback, high-stakes assessment and program evaluation. Following Kane's framework, four inferences in the validity argument were examined (scoring, generalization, extrapolation, decision). For formative feedback and high-stakes assessment, there was reasonable evidence for scoring and extrapolation. However, for high-stakes assessment there was a dearth of evidence for generalization aside from inter-rater reliability data and an absence of evidence linking multi-station OSATS scores to performance in real clinical settings. For program evaluation, the OSATS validity argument was supported by reasonable generalization and extrapolation evidence. There was a complete lack of evidence regarding implications and decisions based on OSATS scores. In general, validity evidence supported the use of the OSATS for formative feedback. Research to provide support for decisions based on OSATS scores is required if the OSATS is to be used for higher-stakes decisions and program evaluation.

Construct Validity of a Simple Laparoscopic Ovarian Cystectomy Model Using a Validated Objective Structured Assessment of Technical Skills

The success of Pavlik harness therapy is based, among other things, upon the correct reapplication by parents. This prospective study assesses the effectiveness of a validated Objective Structured Assessment of Technical Skill (OSATS) educational module for Pavlik harness application among Greek parents. Forty consecutive parents of newborns with developmental dysplasia of the hip (DDH), who were treated with Pavlik harness, were divided into two groups. Twenty parents (group 1) were educated with the Greek OSATS version, while the rest (group 2) served as the control. The OSATS score, quality of the final product (Global Rating Scale - GRS) and sonographic imaging of the hips were evaluated during follow-up. Post-intervention evaluation revealed significant higher means of OSATS score (p<0.01), GRS of performance (p<0.05) and GRS of final product (p<0.05) for group 1. At every retention time point, OSATS was signifi cantly higher for group 1, however there were no significant differences between either GRS score between groups. There was no significant difference in radiographic evaluation and successful outcome. This educative module increased the parents' skill level on application of Pavlik harness. However, this module revealed no significant effect in clinical and radiological evaluation of the hips, in this population.

To assess the learning curve of pancreaticojejunostomy during robotic pancreatoduodenectomy (RPD) and to predict the risk of postoperative pancreatic fistula (POPF) by using the objective structured assessment of technical skills (OSATS), taking the fistula risk into account. RPD is a challenging procedure that requires extensive training and confirmation of adequate surgical performance. Video grading, modified for RPD, of the pancreatic anastomosis could assess the learning curve of RPD and predict the risk of POPF. Post hoc assessment of patients prospectively included in 4 Dutch centers in a nationwide LAELAPS-3 training program for RPD. Video grading of the pancreaticojejunostomy was performed by 2 graders using OSATS (attainable score: 12-60). The main outcomes were the combined OSATS of the 2 graders and POPF (grade B/C). Cumulative sum analyzed a turning point in the learning curve for surgical skill. Logistic regression determined the cutoff for OSATS. Patients were categorized for POPF risk (ie, low, intermediate, and high) based on the updated alternative fistula risk scores. Videos from 153 pancreatic anastomoses were included. Median OSATS score was 48 (interquartile range: 41-52) points and with a turning point at 33 procedures. POPF occurred in 39 patients (25.5%). An OSATS score below 49, present in 77 patients (50.3%), was associated with an increased risk of POPF (odds ratio: 4.01, P =0.004). The POPF rate was 43.6% with OSATS < 49 versus 15.8% with OSATS ≥49. The updated alternative fistula risk scores category "soft pancreatic texture" was the second strongest prognostic factor of POPF (odds ratio: 3.37, P =0.040). Median cumulative surgical experience was 17 years (interquartile range: 8-21). Video grading of the pancreatic anastomosis in RPD using OSATS identified a learning curve and a reduced risk of POPF in case of better surgical performance. Video grading may provide a valid method to surgical training, quality control, and improvement.

Validated Assessment Tools for Pediatric Airway Endoscopy Simulation

To develop an Objective Structured Assessment of Technical Skills (OSATS) tool for partial pulpotomy and to preliminary assess the validity of this tool. Secondary objectives were to check internal consistency as well as the interrater reliability of this specific checklist and to compare it with the previously validated global rating score (GRS). The study assessed three groups of dental participants (mix of practitioners or students) possessing differing levels of clinical experience (7 novices, 7 intermediates, 7 experts), during a partial pulpotomy procedure on a simulated tooth with deep caries, using a bespoke procedure-specific checklist (PSC) and the GRS. Two independent examiners received the study participants in groups of two, in order to be able to directly observe their actions and grade the different steps of the procedure from 1 to 5 on the PSC and GRS. The internal validity of the specific checklist was assessed using a Cronbach's alpha test. As the construct validity of such a tool can be determined when the tool differentiates performance based on the level of experience, the total score of the PSC of the three groups (e.g., novice group, intermediate group, experienced group) were compared using the Kruskal-Wallis and post-hoc Mann-Whitney pairwise U test. The total scores of the GRS were also analysed in the same manner. A correlation test (Correlation Matrices Test) was carried out for the entire sample between the specific checklist and the GRS (on total score). The intraclass correlation coefficient was used to measure the degree of agreement between the two evaluators. The statistical analysis was performed with XLSTAT® and Statistica® (significance p = .05). The partial pulpotomy specific checklist graded from one to five demonstrated good internal consistency (0.86) and a good interrater reliability (0.91), a correct construct validity (p generally < .05) and a good positive correlation with the validated GRS (r = .92). To address the need to develop education in vital pulp treatment, and more specifically the technical skills assessment of partial pulpotomy, an OSATS-specific checklist was developed and preliminary validated. The checklist will need to be validated in larger cohorts.

IntroductionUpdating the method for evaluating suturing and scaling skills in dental education has attracted relatively little attention and there is no consensus to what should be assessed and how. The purpose of this study was to investigate the applicability of the Objective Structured Assessment of Technical Skill (OSATS) method for these two basic skills, the possible association between the scores and demographic factors, and the level of satisfaction of residents with this method.MethodsAll six periodontics and three oral medicine residents were recruited by census method and video-recorded while performing a simple interrupted suture, a figure eight suture and scaling on a model. Then, the videos were evaluated independently via a checklist and a global rating scale (GRS) by two expert raters. Agreement between raters and residents’ satisfaction were evaluated. Correlation between demographic factors of participants and scores was also assessed. T-test and linear regression analysis were used.ResultsThere was no significant difference between the scores based on the views of the two raters for each of the checklist (ICC = 0.99, CI = 0.96–0.99, P < 0.001) and GRS (ICC = 0.97, CI = 0.86–0.99, P < 0.001). Linear regression showed no correlation between gender and scores but periodontics major and higher year of education showed correlation with higher scores.ConclusionConsidering the excellent agreement between raters in using both the checklist and GRS components of OSATS, and satisfaction of 88% the residents with this method, it seems to be able to provide a reliable assessment.

BackgroundTechnical proficiency of the operating surgeons is one of the most important factors in the safe performance of pancreaticoduodenectomy. The objective of the present study was to investigate whether surgical simulation of pancreatico-jejunostomy (PJ) using an inanimate biotissue model could improve the technical proficiency of hepato-biliary pancreatic (HBP) surgical fellows.MethodsThe biotissue drill consisted of sewing biotissues to simulate PJ. The drill was repeated a total of five times by each of the participant surgical fellows. The improvement of the surgical fellows’ technical proficiency was evaluated by the supervisor surgeons using the Objective Structured Assessment of Technical Skills (OSATS) scale.ResultsEight HBP surgical fellows completed all the 5 drills. Both the OSATS 25 score and OSATS summary score, assessed by the two supervisor surgeons, improved steadily with repeated execution of the PJ drill. The average OSATS score, as assessed by both the supervisor surgeons, improved significantly from the first to the final drill, with a P value of 0.003 and 0.014 for the assessment by the two surgeons, respectively. On the other hand, no chronological alteration was observed in time of procedure (P = 0.788).ConclusionRepeated execution of a biotissue PJ drill improved the HBP surgical fellows’ technical proficiency, as evaluated by OSATS. The present study lends support to the evidence that simulation training can contribute to shortening of the time required to negotiate the learning curve for the technique of PJ in the actual operating room.

To evaluate a previously validated low-cost sinus surgery task trainer as a means of acquiring basic endoscopic sinus surgery skills and as an objective structured assessment of technical skills (OSATS) to determine procedural competency. Prospective blinded study. Medical students (N = 52) with no sinus surgery experience learned to perform nasal endoscopy and five specific sinus surgery tasks using the validated task trainer. Training included regimented expert instruction, peer instruction/observation, and experienced-based learning. Pre- and post-training video recordings of nasal endoscopy and five sinus surgery skills were obtained. Two blinded expert otolaryngologists compared pre- and post-training performance using a checklist and global rating scale. Medical student post-training performance was significantly better than pre-training performance for each checklist item and global rating scale as calculated by paired t test (P < .001). Interrater reliability and internal consistency were confirmed by Kendall's coefficient of concordance and Cronbach's α calculations, respectively. The sinus surgery task trainer provides an effective means of teaching and evaluating nasal endoscopy and basic sinus surgery skills for novice surgeons. With repeated practice, there was significant improvement in performance. An OSATS using the sinus surgery task trainer suggests that we can effectively measure endoscopic sinus surgery ability with the potential to reliably determine competency outside the operating room.

The Effect of Stress on Task Acquisition in a Simulated Open Aortic Aneurysm Model

The purpose of the study was to assess the construct validity of an Objective Structured Assessment of Technical Skills (OSATS) developed for cystoscopic and ureteroscopic cognitive and psychomotor skills. An OSATS was designed based on a 14-point comprehensive curriculum prepared by two experts that targeted both cognitive and psychomotor cystoscopic and ureteroscopic skills. Ten urology residents from a single institution with different levels of training were assessed on a series of stations that targeted these skills. Evaluation of cognitive skills was done via a written examination, and psychomotor skills assessment was done by experts using both subjective and objective metrics. Twelve of 15 cognitive tasks and 5 of 5 psychomotor tasks demonstrated construct validity with correlation coefficient (r) more than .75. All three of the cognitive tasks that failed to initially demonstrate validity did so on editorial revision and restructuring of the questions. Our cystoscopic and ureteroscopic OSATS showed excellent construct validity for our population of residents, and we have incorporated it into our urologic skills curriculum.

The technical skill of surgical trainees is not well assessed. This study aimed (1) to compare the reliability of three scoring systems, (2) to compare live and bench formats and (3) to assess construct validity of a test of operative skill. Parallel examinations of operative skill, one using live animals and one using simulations, were developed. Performance was graded using operation-specific checklists, detailed global rating forms and pass/fail judgements. Twenty surgical residents each took both formats. Disattenuated correlations between live and bench scores were high (0.69-0.72). Mean interrater reliability across stations ranged from 0.64 to 0.72. Internal consistency was moderate to high (alpha: 0.61-0.74) for the live format using the checklist and for live and bench formats using global ratings. Global ratings discriminated between resident levels for both formats (bench: F(2,17) = 4.45, P < 0.05; live: F(2,17) = 3.55, P < 0.05), checklists did not. This preliminary study suggests that the Objective Structured Assessment of Technical Skill can reliably and validly assess surgical skills. Global ratings are a better method of assessment than task-specific checklists. Bench model simulation gives equivalent results to use of live animals for this test format.

Background: It is frequently suggested that minimally invasive surgery (MIS) is harder to acquire than conventional surgery. To test this hypothesis, residents’ learning curves of both surgical skills are compared. Methods: Residents had to be assessed using a general global rating scale of the OSATS (Objective Structured Assessment of Technical Skills) for every procedure they performed as primary surgeon during a 3-month clinical rotation in gynecological surgery. Results: Nine postgraduate-year-4 residents collected a total of 319 OSATS during the 2 years and 3 months investigation period. These assessments concerned 129 MIS (laparoscopic and hysteroscopic) and 190 conventional (open abdominal and vaginal) procedures. Learning curves (in this study defined as OSATS score plotted against procedure-specific caseload) for MIS and conventional surgery were compared using a linear mixed model. The MIS curve revealed to be steeper than the conventional curve (1.77 vs. 0.75 OSATS points per assessed procedure; 95% CI 1.19–2.35 vs. 0.15–1.35, p < 0.01). Conclusions: Basic MIS procedures do not seem harder to acquire during residency than conventional surgical procedures. This may have resulted from the incorporation of structured MIS training programs in residency. Hopefully, this will lead to a more successful implementation of the advanced MIS procedures.

Plaster cast removal simulation model for resident training.

Measuring surgical competency is essential for surgical residents to ensure patient safety. Traditional assessment tools rely on subjective evaluation. This study evaluated whether artificial intelligence (AI)-based hand tracking can more objectively distinguish between levels of surgical competency and predict surgical years of experience versus traditional assessments. A total of 44 participants, including medical students, surgical residents, and surgical consultants, performed transcutaneous suturing, intracutaneous suturing, and surgical knot tying. Videos of intracutaneous suturing were scored using the objective structured assessment of technical skills (OSATS). Hand movements were analysed using AI tracking software to extract coordinates to measure velocity, pathlength, and jerk. Linear regression models predicted experience years using procedural time and OSATS in combination with hand tracking metrics. Hand tracking metrics varied mainly between medical students and more experienced groups. Traditional assessment tools (procedural time, OSATS) could predict experience years during training, with an adjusted coefficient of determination (R2) ranging from 0.537 to 0.638, dependent on procedure type. Hand tracking variables identified multiple significant predictors for years of experience, with an adjusted R2 of 0.540-0.712, which outperformed the traditional tools in each procedure. Combining all assessment tools (time, OSATS, and hand tracking) gave the best predictive value, with an adjusted R2 ranging from 0.540 to 0.809, with velocity, pathlength, jerk, and acceleration as significant predictors. AI-based hand tracking provides a new method for objective, reproducible measures of surgical skills. Incorporating hand tracking metrics enhances prediction of surgical experience, and supports standardized as well as objective evaluation of skills assessment in surgical training.