Development of a Standardized Kalamazoo Communication Skills Assessment Tool for Radiologists: Validation, Multisource Reliability, and Lessons Learned.

Aims This paper examines: the intra-rater, inter-rater and test-retest reliability of the Five-Times-Sit-To-Stand (FTSTS) test administered by experienced and inexperienced assessors; and the correlation of FTSTS test times with lower limb muscle strength and balance performance in healthy older and young people. Method The cross-sectional study investigated reliability using test batteries including: FTSTS times; hand-held dynamometer measurements of hip flexor, knee flexor and extensor, and ankle dorsiflexor and plantarflexor muscle strength; Berg Balance Scale (BBS) scores; and dynamic posturography measurements of the limits of stability (LOS). A convenience sample of 12 healthy older and 12 younger participants was recruited in the local university-based rehabilitation centre. Results The FTSTS test showed excellent intra-rater reliability (intraclass correlation coefficient (ICC) range: 0.914–0.933) and moderate intra-rater reliability (ICC range: 0.644–0.879) in healthy older and young subject groups respectively. Inter-rater reliability between assessors yielded an ICC of 0.990. Test-retest reliability (ICC range: 0.988–0.995) was excellent. FTSTS times were not significantly associated with the older subjects' lower limb muscle strength, BBS scores or LOS results. Conclusion The FTSTS has good intra-rater, inter-rater and test-retest reliability. There is no significant correlation of FTSTS times with lower limb muscle strength or balance performance in healthy older people. The FTSTS test is a reliable measurement tool whether used by experienced or inexperienced assessors. A future study with a larger sample size is required for establishing the validity of FTSTS test times in the older population.

The timed vibration test (TVT) is an easy-to-perform bedside sensory test; however, its reliability is not well established at sites commonly used for clinical testing. We evaluated intra- and interrater reliability of the TVT in a healthy control cohort of 25 adult volunteers and assessed the influence of neurologic training on TVT. Intrarater [intraclass correlation coefficient (ICC) range 0.79-0.92] and interrater (ICC range 0.0.82-0.95) reliability of TVT was high at different sites, which suggests that TVT is a reliable bedside examination when performed using a standardized protocol.

Psychometric study of the Required Care Levels for People with Severe Mental Disorder Assessment Scale (ENAR-TMG)

The American Board of Psychiatry and Neurology (ABPN) has recently replaced the traditional, centralized oral examination with the locally administered Neurology Clinical Skills Examination (NEX). The ABPN postulated the experience with the NEX would be similar to the Mini-Clinical Evaluation Exercise, a reliable and valid assessment tool. The reliability and validity of the NEX has not been established. NEX encounters were videotaped at 4 neurology programs. Local faculty and ABPN examiners graded the encounters using 2 different evaluation forms: an ABPN form and one with a contracted rating scale. Some NEX encounters were purposely failed by residents. Cohen's kappa and intraclass correlation coefficients (ICC) were calculated for local vs ABPN examiners. Ninety-eight videotaped NEX encounters of 32 residents were evaluated by 20 local faculty evaluators and 18 ABPN examiners. The interrater reliability for a determination of pass vs fail for each encounter was poor (kappa 0.32; 95% confidence interval [CI] = 0.11, 0.53). ICC between local faculty and ABPN examiners for each performance rating on the ABPN NEX form was poor to moderate (ICC range 0.14-0.44), and did not improve with the contracted rating form (ICC range 0.09-0.36). ABPN examiners were more likely than local examiners to fail residents. There is poor interrater reliability between local faculty and American Board of Psychiatry and Neurology examiners. A bias was detected for favorable assessment locally, which is concerning for the validity of the examination. Further study is needed to assess whether training can improve interrater reliability and offset bias.

Intrarater and Inter-rater Reliability of Maximal Voluntary Neck Muscle Strength Assessment Using a Handheld Dynamometer in Women With Headache and Healthy Women

Dear Editor, We read with great interest the article “Reliability of a Frailty Index Based on the Clinical Assessment of Health Deficits in Male C57BL/6J Mice” recently published online in the Journal of Gerontology Biological Sciences (1). Feridooni and colleagues investigated the reliability of a recently developed mouse clinical frailty index between two raters, in three separate groups of male C57BL/6 mice aged 0.9–1.2 years, with discussion and refinement of index descriptors between assessment of each group. They found high overall inter-rater reliability, and that measures of reliability increased sequentially over the three groups. Our assessment of the mouse clinical frailty index in one published (2) and two unpublished studies of male C57BL/6 mice shed further light on factors that impact on its inter-rater reliability. Our studies clarify that practice and experience with assessment of the mouse clinical frailty index, without discussion or refinement, does not improve inter-rater reliability. We have completed two projects assessing the frailty index of old (Project 1 n = 55, 26.1±0.5 months; Project 2 n = 17, 23.9±0.2 months) mice before and after 4–6 weeks of a dietary/pharmaceutical intervention. Each project used two raters for the frailty index, who did not discuss or compare their scoring until the completion of the projects. The inter-rater reliability between the two raters for both of these projects did not improve with practice or experience (Project 1 Pre ICC=0.522[CI 0.181–0.721], p = .00; r 2 = .361, p = .00; Post ICC = 0.488[CI 0.244–0.652], p = .01; r 2 = .328, p = .00; Project 2 Pre ICC = 0.705[CI 0.186–0.893], p = .01; r 2 = .562, p = .02; Post ICC = 0.561[CI 0.122–0.781], p = .01; r 2 = .391, p = .02). These results fall within the range of reliability statistics observed by Feridooni and colleagues (1), using different raters in different research groups and countries. Another factor that we have identified as affecting the inter-rater reliability when assessing the mouse clinical frailty index is the professional background and baseline animal training of the raters. In order to explore this, we had four raters assess the clinical frailty index of a cohort of old (n = 74, 19.0±1.0 months) mice, with no discussion or training period. Two raters were animal technicians with 5–10 years of experience and training in animal handling, and two raters were research scientists with minimal animal handling training, and 3–6 years of animal research experience. Interestingly, the inter-rater reliability between the technicians and researchers was poor (ICC = 0.201–0.489, p < .05; r 2 = −0.028 to 0.33, p < .05), despite moderate inter-rater reliability between the technicians (ICC = 0.605[CI 0.344–0.762], p = .00; r 2 = .454, p = .00) and excellent inter-rater reliability between the scientists (ICC=0.88[95%CI 0.80–0.92] p = 0.00; r 2 = .817, p = .00). Furthermore, it does appear to be the background, rather than the years of experience that are important, as the inter-rater reliabilities in our pre and post intervention studies (data above), which were all performed by scientists, were moderate to high despite the raters including PhD students and a senior postdoc/laboratory manager. This suggests that the selection of raters for the mouse clinical frailty index must be done carefully, and preference given to raters with the same training/background, but not necessarily experience level, if comparisons are to be made between and across mouse groups and studies.

The novel Next Step test is a reliable measure of anticipatory postural adjustments made by children with cerebral palsy prior to taking a step

BACKGROUND: The value assessment framework (VAF) is one approach to assessing the evidence and value of medications. VAFs are a way to measure and communicate the value of medications and other health care technologies for decision-making purposes. Given the increasing number of high-cost medications, challenging formulary inquiries, and critiques of currently available tools, health systems need to explore a standardized way to incorporate value assessment into formulary decision making. OBJECTIVES: To (a) evaluate existing VAFs by measuring inter-rater reliability among typical clinicians completing formulary reviews and (b) explore general implications of applying these tools to formulary decision making for all medications at a large academic health system. METHODS: This was a retrospective, observational study at a single health system. A list of medications added, denied, and removed from the system formulary from September 1, 2013, through August 31, 2018, was collected. Published VAFs, such as the American Society of Clinical Oncology (ASCO) Value Framework, European Society of Medical Oncology (ESMO) Magnitude of Clinical Benefit Scale, National Comprehensive Cancer Network (NCCN) Evidence Blocks, American College of Cardiology/American Heart Association Value Framework, and the incremental cost-effectiveness ratio (ICER) calculation were applied by 3 different reviewer groups. The primary outcome was inter-rater reliability among the 3 different reviewers for a given framework. Cohen's weighted kappa and the intraclass correlation coefficient (ICC) were used to assess inter-rater reliability. RESULTS: The frameworks were applied to 94 medications. The VAFs with the highest ICCs between all 3 raters were NCCN (0.635; 95% CI = 0.387-0.823) and ASCO (0.634; 95% CI = 0.370-0.832), both indicating moderate inter-rater reliability. The VAFs with the lowest ICCs were ESMO (0.368; 95% CI = 0.126-0.611) and ICER (0.159; 95% = CI -0.018-0.365), with ICCs corresponding to poor reliability. CONCLUSIONS: Because high-cost medications are a challenge to health systems, VAFs may be beneficial to target formulary decision making in this setting. Applying VAFs proactively may improve interrater reliability and usability in formulary decision making. DISCLOSURES: No outside funding supported this study. The authors have nothing to disclose.

Development and preliminary validation of the Lip Reanimation Outcomes Questionnaire

A bias in radiation risk estimates may be caused by diagnostic errors. In this paper radiation risk estimates are examined with the use of the contingency table 2х2 of irradiated cancer cases with account of sensitivity and specificity indicators of diagnostic methods accuracy were examined. Radiation risk is estimated by metrics of the odds ratio (OR) and relative risk ratio (RR). It is shown that the estimation of radiation risk in the RR metric did not depend on the diagnostic sensitivity indicator, and therefore is preferable, compared to the OR metric. When the specificity of the diagno-sis is reduced, the RR value increases, compared to the risk estimated with the use of observed number of cancer cases regardless of the specificity. As a numerical example, data on trachea, bronchus and lung cancers in a cohort of the Chernobyl cleanup workers of Russia formed by using data monitored within frames of the National Radiation Epidemiological Register (NRER) from 1992 to 2020. During this period, 2,222 cancer cases were reported in the cohort of 67,587 people. The cohort members were divided into two groups, those who were unexposed to radiation (radia-tion doses less than 0.1 Gy) and other, exposed to radiation. If the specificity indicator is taken to be 100%, then the RR=1.07 at 95% CI (1.02; 1.13). The estimates of radiation risks of malignant neoplasms obtained directly from the observed number of cancer cases in exposed and unexposed groups of people, excluding the specificity and sensitivity indicators of diagnosis accuracy, are the lower estimates of radiation risk. As the probability of diagnostic errors increases (as the specificity and sensitivity of diagnostic tests decrease), and when these errors are taken into account, the es-timates of radiation risks increase. If the specificity is reduced to 98.7% CI radiation risk estimates for trachea, bronchus and lung cancer among cleanup workers of the Chernobyl accident increase to RR=1.13, but it is within 95% CI of the RR estimate if specificity is 100%. The radiation risk as-sessment approach used in the study, based on the comparison of exposed and unexposed groups of cleanup workers in OR or RR metrics, imposes high requirements to the specificity indicator of the complex of diagnostic methods in trachea, bronchus and lung (Sp&gt;98.7%) in order to obtain unbiased estimates of radiation risks of cancer in the cleanup workers received dose below 1 Sv. These requirements are not supported by current screening methods, they can be fulfilled as a result of longterm medical monitoring within the system of the NRER.

The aim of this study was to evaluate the validity of the Veloflex infrared dynamic angle-meter (Veloflex-IDA) and the intra- and inter-rater reliability when measuring the ranges of motion (ROMs) of the upper limb joints. Thirty-five healthy and 20 symptomatic participants were evaluated. Twelve upper limb movements were measured in two sessions with the Veloflex-IDA, which is a device composed of a camera that tracks the trajectory of retro-reflective markers. In addition, a goniometer was used in the first session to evaluate concurrent validity. Validity and agreement were evaluated by Pearson correlation coefficient (r) and Bland–Altmann plots. Intra- and inter-rater reliability were evaluated using intra-class correlation (ICC), standard error of measurement (SEM), and minimal detectable change (MDC). Both instruments showed excellent correlation for all movements (r range from 0.992 to 0.999). The intra- and inter-rater reliability were excellent (ICC range from 0.95 to 0.99 and 0.90 to 0.98, respectively). Intra-rater reliability showed SEMs <1.38% and <5.19% and inter-rater reliability SEMs <2.26% and <5.22% for asymptomatic and symptomatic, respectively. Veloflex-IDA is a valid and reliable alternative to measure the upper limb joints’ ROM and it can be used in clinical practice and research after basic training.

Background and PurposeThe Expanded Cutting Alignment Scoring Tool (E-CAST) is a two-dimensional qualitative scoring system that has demonstrated moderate inter-rater and good intra-rater reliability for the assessment of trunk and lower extremity alignment during a 45-degree sidestep cut. The primary purpose of this study was to examine the reliability of the quantitative version of the E-CAST among physical therapists and to compare the reliability of the quantitative E-CAST to the original qualitative E-CAST. The hypothesis was that the quantitative version of the E-CAST would demonstrate greater inter-rater and intra-rater reliability compared to the qualitative E-CAST.Study DesignObservational cohort, repeated measures reliability studyMethodsTwenty-five healthy female athletes (age 13.8±1.4 years) performed three sidestep cuts with two-dimensional video capturing frontal and sagittal views. Two physical therapist raters independently scored a single trial using both views on two separate occasions. Based on the E-CAST criteria, select kinematic measurements were extracted using a motion analysis phone application. Intraclass correlation coefficients and 95% confident intervals were calculated for the total score, and kappa coefficients were calculated per kinematic variable. Correlations were converted to z-scores and compared to the six original criteria for significance (α<0.05).ResultsCumulative intra- and inter-rater reliability were both good (ICC=0.821, 95% CI 0.687-0.898 and ICC=0.752, 95% CI 0.565-0.859). Cumulative intra-rater kappa coefficients ranged from moderate to almost perfect, and cumulative inter-rater kappa coefficients ranged from slight to good. No significant differences were observed between the quantitative and qualitative criteria for either inter- or intra-rater reliability (Zobs(intra)= -0.38, p=0.352 and Zobs(inter)= -0.30, p=0.382).ConclusionThe quantitative E-CAST is a reliable tool to assess trunk and lower extremity alignment during a 45-degree sidestep cut. No significant differences were observed in reliability of the quantitative versus qualitative assessment.Level of evidence3b

Interobserver Agreement of Lung Ultrasound Findings of COVID-19

This study aimed to evaluate ChatGPT's performance on questions about periprosthetic joint infections (PJI) of the hip and knee. Twenty-seven questions from the 2018 International Consensus Meeting on Musculoskeletal Infection were selected for response generation. The free-text responses were evaluated by three orthopedic surgeons using a five-point Likert scale. Inter-rater reliability (IRR) was assessed via Fleiss' kappa (FK). Overall, near-perfect IRR was found for disagreement on the presence of factual errors (FK: 0.880, 95% CI [0.724, 1.035], p < 0.001) and agreement on information completeness (FK: 0.848, 95% CI [0.699, 0.996], p < 0.001). Substantial IRR was observed for disagreement on misleading information (FK: 0.743, 95% CI [0.601, 0.886], p < 0.001) and agreement on suitability for patients (FK: 0.627, 95% CI [0.478, 0.776], p < 0.001). Moderate IRR was observed for agreement on "up-to-dateness" (FK: 0.584, 95% CI [0.434, 0.734], p < 0.001) and suitability for orthopedic surgeons (FK: 0.505, 95% CI [0.383, 0.628], p < 0.001). Question- and subtopic-specific analysis revealed diverse IRR levels ranging from near-perfect to poor. ChatGPT's free-text responses to complex orthopedic questions were predominantly reliable and useful for orthopedic surgeons and patients. Given variations in performance by question and subtopic, consulting additional sources and exercising careful interpretation should be emphasized for reliable medical decision-making.

Full disclosure of medical errors to patients is now widely seen as an essential component of error management, although its update into daily clinical practice is variable. Laboratory diagnostic errors are discovered in retrospective reviews of previous surgical and cytopathology cases. This quality assurance practice is a valuable tool of practice audit and change for both cytotechnologists and pathologists. Presently, these diagnostic errors are only reported to the clinician and patient if the new finding affects current patient management. Mandatory full disclosure of all diagnostic errors discovered in the retrospective review process would have a significant adverse impact on cytotechnologists, pathologists, the laboratory, the clinic, the institution, and insurers. Retrospective pathology review would become so burdensome that its survival would be in jeopardy-unless measures are undertaken to ameliorate the anticipated adverse consequences.