Construct Validity of a Simple Laparoscopic Ovarian Cystectomy Model Using a Validated Objective Structured Assessment of Technical Skills

Measuring Surgical Trainee Perceptions to Assess the Operating Room Educational Environment

Validated Assessment Tools for Pediatric Airway Endoscopy Simulation

The goal of training in orthopedic residency is to produce surgeons who are proficient in all aspects of the practice of orthopedic surgery; however, most residents receive either inadequate or no training in medical coding. The purpose of this study was to determine how well orthopedic residents code when compared with practicing surgeons and to identify whether coding education improves accuracy in medical coding. A mock coding survey was developed using commonly encountered orthopedic clinical scenarios. The survey was distributed to orthopedic trainees post-graduate years (PGY) 1 to 6 at 2 training programs and to attending surgeons. Results were analyzed in 3 groups: junior residents (PGY 1-3), senior residents (PGY 4-6), and attending surgeons. Overall and subcategory scores of (1) type of visit, (2) modifiers, (3) Evaluation and Management (E/M), and (4) Current Procedural Terminology code identification were recorded. Participants were also asked if they had ever received various forms of coding education. Sixty-seven total participants were enrolled, including 28 junior residents, 24 senior residents, and 15 attendings. Practicing surgeons performed significantly better than both senior (P<.027) and junior (P<.001) residents in all categories, with a mean overall correct response rate of 72.8%, 51.0%, and 47.4%, respectively. Any form of coding education was associated with a significantly improved overall score for residents (P=.013) and a nonsignificant increase for attending surgeons (P=.390). This study demonstrates that residents performed poorly when identifying proper billing codes for common procedures and encounters in orthopedic surgery. Further, those participants who received coding education did better than those who did not. [Orthopedics. 2020;43(6):380-383.].

Testing technical skill via an innovative “bench station” examination

Exploring Entrustment: Housestaff Autonomy and Patient Readmission

In order to construct and evaluate the validity argument for the Objective Structured Assessment of Technical Skills (OSATS), based on Kane's framework, we conducted a systematic review. We searched MEDLINE, EMBASE, CINAHL, PsycINFO, ERIC, Web of Science, Scopus, and selected reference lists through February 2013. Working in duplicate, we selected original research articles in any language evaluating the OSATS as an assessment tool for any health professional. We iteratively and collaboratively extracted validity evidence from included articles to construct and evaluate the validity argument for varied uses of the OSATS. Twenty-nine articles met the inclusion criteria, all focussed on surgical technical skills assessment. We identified three intended uses for the OSATS, namely formative feedback, high-stakes assessment and program evaluation. Following Kane's framework, four inferences in the validity argument were examined (scoring, generalization, extrapolation, decision). For formative feedback and high-stakes assessment, there was reasonable evidence for scoring and extrapolation. However, for high-stakes assessment there was a dearth of evidence for generalization aside from inter-rater reliability data and an absence of evidence linking multi-station OSATS scores to performance in real clinical settings. For program evaluation, the OSATS validity argument was supported by reasonable generalization and extrapolation evidence. There was a complete lack of evidence regarding implications and decisions based on OSATS scores. In general, validity evidence supported the use of the OSATS for formative feedback. Research to provide support for decisions based on OSATS scores is required if the OSATS is to be used for higher-stakes decisions and program evaluation.

A Needs Assessment of Video-based Education Resources Among General Surgery Residents

To develop an Objective Structured Assessment of Technical Skills (OSATS) tool for partial pulpotomy and to preliminary assess the validity of this tool. Secondary objectives were to check internal consistency as well as the interrater reliability of this specific checklist and to compare it with the previously validated global rating score (GRS). The study assessed three groups of dental participants (mix of practitioners or students) possessing differing levels of clinical experience (7 novices, 7 intermediates, 7 experts), during a partial pulpotomy procedure on a simulated tooth with deep caries, using a bespoke procedure-specific checklist (PSC) and the GRS. Two independent examiners received the study participants in groups of two, in order to be able to directly observe their actions and grade the different steps of the procedure from 1 to 5 on the PSC and GRS. The internal validity of the specific checklist was assessed using a Cronbach's alpha test. As the construct validity of such a tool can be determined when the tool differentiates performance based on the level of experience, the total score of the PSC of the three groups (e.g., novice group, intermediate group, experienced group) were compared using the Kruskal-Wallis and post-hoc Mann-Whitney pairwise U test. The total scores of the GRS were also analysed in the same manner. A correlation test (Correlation Matrices Test) was carried out for the entire sample between the specific checklist and the GRS (on total score). The intraclass correlation coefficient was used to measure the degree of agreement between the two evaluators. The statistical analysis was performed with XLSTAT® and Statistica® (significance p = .05). The partial pulpotomy specific checklist graded from one to five demonstrated good internal consistency (0.86) and a good interrater reliability (0.91), a correct construct validity (p generally < .05) and a good positive correlation with the validated GRS (r = .92). To address the need to develop education in vital pulp treatment, and more specifically the technical skills assessment of partial pulpotomy, an OSATS-specific checklist was developed and preliminary validated. The checklist will need to be validated in larger cohorts.

The aim of this study was to assess the efficacy of intraperitoneal different combinations for postoperative pain relief in patients undergoing laparoscopic unilateral ovarian cystectomy. We conducted a double-blind randomized controlled trial that enrolled patients who were included to undergo laparoscopic unilateral ovarian cystectomy. Patients received one of the following combinations (30 patients each): group I: received bupivacaine plus magnesium sulfate, group II: received bupivacaine plus hydrocortisone, group III: received magnesium sulfate plus hydrocortisone, and group IV: received saline 0.9% only. The primary outcomes in the present study were the severity of postoperative abdominal and shoulder pain assessed by visual analog scale (VAS) every two hours till the end of the first day, and time for first postoperative analgesia requirement. Group I had statistically significant lower abdominal static and dynamic pain scores than the other groups till 18 hours postoperatively (P<0.001). In addition, group II had statistically significant lower abdominal static and dynamic pain scores than group III in the most of assessment points (P<0.05). The time for first required analgesics was significantly longer in group I (336.2±67.54 minutes) than other groups (P<0.001). The proportion of patients who required two or more doses of ketorolac was significantly lower in group I than other groups (P<0.001). The proportion of patients with nausea or vomiting was not significantly different across study's groups (P>0.05). Intraperitoneal bupivacaine-magnesium combination provides better analgesia and reduces postoperative morphine consumption than bupivacaine-hydrocortisone or magnesium-hydrocortisone combinations after laparoscopic ovarian cystectomy.

To evaluate a previously validated low-cost sinus surgery task trainer as a means of acquiring basic endoscopic sinus surgery skills and as an objective structured assessment of technical skills (OSATS) to determine procedural competency. Prospective blinded study. Medical students (N = 52) with no sinus surgery experience learned to perform nasal endoscopy and five specific sinus surgery tasks using the validated task trainer. Training included regimented expert instruction, peer instruction/observation, and experienced-based learning. Pre- and post-training video recordings of nasal endoscopy and five sinus surgery skills were obtained. Two blinded expert otolaryngologists compared pre- and post-training performance using a checklist and global rating scale. Medical student post-training performance was significantly better than pre-training performance for each checklist item and global rating scale as calculated by paired t test (P < .001). Interrater reliability and internal consistency were confirmed by Kendall's coefficient of concordance and Cronbach's α calculations, respectively. The sinus surgery task trainer provides an effective means of teaching and evaluating nasal endoscopy and basic sinus surgery skills for novice surgeons. With repeated practice, there was significant improvement in performance. An OSATS using the sinus surgery task trainer suggests that we can effectively measure endoscopic sinus surgery ability with the potential to reliably determine competency outside the operating room.

Wound construction is a critical step in phacoemulsification. Using anterior segment optical coherence tomography (AS-OCT), we compared the morphological features and complications of main incisions made by junior or senior residents during phacoemulsification. This cross-sectional comparative study included eyes with senile cataracts that underwent uneventful phacoemulsification with a clear corneal incision made by seven senior and eight junior ophthalmology residents. All eyes underwent postoperative image acquisition using AS-OCT on day one and at three months, examining for morphological features and potential complications of the main incision. We included 50 eyes of 50 patients with a male-to-female ratio of 22 (44%) to 28 (56%); 26 (52%) were operated on by junior residents and 24 (48%) by seniors. The mean geometric features of the main incisions and the frequency of early and late wound complications were comparable between the two groups (all P > 0.05). A significant correlation was found between the incision length and angle with the superior (r = + 0.80; P < 0.001 and r = - 0.63; P < 0.001, respectively) and inferior (r = + 0.84; P < 0.001 and r = - 0.68; P < 0.001, respectively) areas of the incision, as well as between the length and angle of incision (r = - 0.74; P < 0.001). The number of planes in the wound architecture was not significantly different according to senior or junior resident status (P > 0.05). Although the number of eyes with stromal hydration was significantly greater for junior residents than for seniors (P < 0.001), the corneal thickness at the entrance to the cornea or the anterior chamber, presence of endothelial wound gaping, and Descemet's membrane detachment were comparable between eyes with and without stromal hydration (all P > 0.05). At three months, 29 (58%) patients returned for examination, in whom seven (24%) had late wound complications. This study found no significant differences in the performances of junior and senior residents in terms of wound construction or its associated complications. However, considering the overall rate of some observed wound-related complications, we recommended revision of the resident educational curriculum concerning the structure and complications of the main incision.

The success of Pavlik harness therapy is based, among other things, upon the correct reapplication by parents. This prospective study assesses the effectiveness of a validated Objective Structured Assessment of Technical Skill (OSATS) educational module for Pavlik harness application among Greek parents. Forty consecutive parents of newborns with developmental dysplasia of the hip (DDH), who were treated with Pavlik harness, were divided into two groups. Twenty parents (group 1) were educated with the Greek OSATS version, while the rest (group 2) served as the control. The OSATS score, quality of the final product (Global Rating Scale - GRS) and sonographic imaging of the hips were evaluated during follow-up. Post-intervention evaluation revealed significant higher means of OSATS score (p<0.01), GRS of performance (p<0.05) and GRS of final product (p<0.05) for group 1. At every retention time point, OSATS was signifi cantly higher for group 1, however there were no significant differences between either GRS score between groups. There was no significant difference in radiographic evaluation and successful outcome. This educative module increased the parents' skill level on application of Pavlik harness. However, this module revealed no significant effect in clinical and radiological evaluation of the hips, in this population.

The Accreditation Council for Graduate Medical Education requires that residency programs have an effective plan for assessing their residents' competence. Current evaluation tools used for the assessment of operative competency are subjective, recall based, and do not allow providing formative feedback to the resident. The purpose of this study was to evaluate the feasibility and validity of video-based assessment of endoscopic sinus surgery (ESS) skills. Five junior (postgraduate year levels 1-3[PGY-1 to -3]) and three senior (PGY-4 and -5) residents were videotaped as they performed ESS on cadavers in the laboratory. Five faculty members watched the recorded videotapes and rated performance using global and checklist assessment tools for ESS. Evaluators were blinded to the identity of the performers. Rating was made feasible by allowing the raters to fast-forward through the tapes when appropriate. Construct validity was calculated by comparing total scores and score on each item of the instrument across the training levels. For all statistical purposes p < 0.05 was considered significant. Results showed construct validity with senior residents performing better than junior residents. It took an average of 20 minutes (range, 7- 39 minutes) to watch and evaluate a video. Interrater reliability, as measured by interclass correlation coefficient across evaluators, was 0.62. The use of videotapes combined with the objective assessment tool is a feasible and valid method for evaluating surgical skills. It has implications for residency programs, because it will increase faculty buy-in and reduce the potential for biased assessment.

To assess the ability of current 3D printing technology to generate a craniofacial bony and soft tissue anatomical model for use in simulating the performance of a fronto-orbital advancement (FOA) osteotomy and then to further assess the value of the model as an educational tool. Anatomic models were designed with a process of serial anatomic segmentation/design, 3D printing, dissection, and device refinement. A validation study was conducted with 5 junior and 5 senior plastic surgery residents. The validation study incorporated a multiple-choice Knowledge Assessment test (KA), an Objective Structured Assessment of Technical skills (OSATs), a Global Rating Scale (GRS) and a Michigan Standard Simulation Experience Scale (MiSSES). We compared the scores of both the junior and senior residents and compared junior resident scores, before and after viewing a lecture/demonstration. MiSSES showed high face validity with a score of 85.1/90, signifying high satisfaction with the simulator learning experience. Simulation and the lecture/demonstration improved the junior resident average KA score from 5.6/10 to 9.6/10 (P = .02), OSATs score from 32.4/66 to 64.4/66 (P < .001) and GRS score from 13.9/35 to 27.5/35 (P < .001). The senior residents OSATs score of 56.3/66 was higher than the pre-lecture juniors (32.4/66) (P < .001), but lower than the post-lecture juniors (64.4/66) (P < .001). We have successfully fabricated a 3D printed craniofacial simulator capable of being used as an educational tool alongside traditional surgical training. Next steps would be improving soft tissue realism, inclusion of patient and disease specific anatomy and creation of models for other surgical specialties.

The purpose of the study was to assess the construct validity of an Objective Structured Assessment of Technical Skills (OSATS) developed for cystoscopic and ureteroscopic cognitive and psychomotor skills. An OSATS was designed based on a 14-point comprehensive curriculum prepared by two experts that targeted both cognitive and psychomotor cystoscopic and ureteroscopic skills. Ten urology residents from a single institution with different levels of training were assessed on a series of stations that targeted these skills. Evaluation of cognitive skills was done via a written examination, and psychomotor skills assessment was done by experts using both subjective and objective metrics. Twelve of 15 cognitive tasks and 5 of 5 psychomotor tasks demonstrated construct validity with correlation coefficient (r) more than .75. All three of the cognitive tasks that failed to initially demonstrate validity did so on editorial revision and restructuring of the questions. Our cystoscopic and ureteroscopic OSATS showed excellent construct validity for our population of residents, and we have incorporated it into our urologic skills curriculum.