Measuring Patient-Reported Outcomes Adaptively: Multidimensionality Matters!

In computerized adaptive testing (CAT), a variable-length stopping rule refers to ending item administration after a pre-specified measurement precision standard has been satisfied. The goal is to provide equal measurement precision for all examinees regardless of their true latent trait level. Several stopping rules have been proposed in unidimensional CAT, such as the minimum information rule or the maximum standard error rule. These rules have also been extended to multidimensional CAT and cognitive diagnostic CAT, and they all share the same idea of monitoring measurement error. Recently, Babcock and Weiss (J Comput Adapt Test2012. https://doi.org/10.7333/1212-0101001) proposed an "absolute change in theta" (CT) rule, which is useful when an item bank is exhaustive of good items for one or more ranges of the trait continuum. Choi, Grady and Dodd (Educ Psychol Meas 70:1-17,2010) also argued that a CAT should stop when the standard error does not change, implying that the item bank is likely exhausted. Although these stopping rules have been evaluated and compared in different simulation studies, the relationships among the various rules remain unclear, and therefore there lacks a clear guideline regarding when to use which rule. This paper presents analytic results to show the connections among various stopping rules within both unidimensional and multidimensional CAT. In particular, it is argued that the CT-rule alone can be unstable and it can end the test prematurely. However, the CT-rule can be a useful secondary rule to monitor the point of diminished returns. To further provide empirical evidence, three simulation studies are reported using both the 2PL model and the multidimensional graded response model.

Most computerized adaptive testing (CAT) programs do not allow item review due to a decrease in estimation precision and aberrant manipulation strategies. In this article, a block item pocket (BIP) method that combines the item pocket method with the successive block method to realize reviewable CAT was proposed. A worst-case but still reasonable answering strategy and the Wainer-like manipulation strategy were simulated to evaluate the estimation precision of reviewable unidimensional computerized adaptive testing (UCAT) and multidimensional computerized adaptive testing (MCAT) under a series of BIP settings. For both UCAT and MCAT, it was found that the estimation precision of the BIP method improved as the number of blocks increased or the item pocket size decreased under the reasonable strategy. The BIP method was more effective in handling the Wainer-like strategy. With the help of block design, the BIP method can still maintain acceptable estimation precision under slightly large total IP size conditions. These results suggested that the BIP method was a reliable solution for both reviewable UCAT and MCAT.

Most item response theory models assume a uni-dimensional latent space. This study extended previous work on the effects of dimensionality on parameter estimation for dichotomous models to the polytomous graded response model. A multidimensional graded response model was developed to generate data in one, two, and three dimensions. The two- and three-dimension conditions contained datasets that varied in their interdimensional association. The effects of test length and the ratio of sample size to the number of item parameters estimated also were investigated. For the unidimensional data, a sample size ratio of 5:1 provided reasonably accurate estimation; increasing test length from 15 to 30 items did not have a significant impact on the accuracy of item parameter estimation. Regardless of the dimensionality of the data, the difficulty parameters were well estimated. For the multidimensional data, the correlations between the estimated item discrimination and the average (as well as the sum of the) dimensional discrimination were greater than the correlations between the estimated item discrimination and the individual dimensional discriminations. Fidelity coefficients between the mean trait level and the trait level estimate (θ) were greater than those between θ and the latent traits. The impact of equating on accuracy indexes in a multidimensional context is discussed. Index terms: dimensionality, graded response model, IRT, marginal maximum likelihood estimation, polytomous models.

PurposeThe animated activity questionnaire (AAQ) is a computer-based measure of activity limitations. To answer a question, patients choose the animation of a person performing an activity that matches their own level of limitation. The AAQ has not yet been tested for suitability to be applied as computer-adaptive test (CAT). Thus, the objective of this study was to develop and evaluate an AAQ-based CAT to facilitate the application of the AAQ in daily clinical care.MethodsPatients (n = 1408) with hip/knee osteoarthritis from Brazil, Denmark, France, The Netherlands, Norway, Spain, and the UK responded to all 17 AAQ items. Assumptions of item-response theory (IRT) modelling were investigated. To establish item parameters for the CAT, a graded response model was estimated. To evaluate the performance of post-hoc simulated AAQ-based CATs, precision, test length, and construct validity (correlations with well-established measures of activity limitations) were evaluated.ResultsUnidimensionality (CFI = 0.95), measurement invariance (R2-change < 2%), and IRT item fit (S-X2p > .003) of the AAQ were supported. Performing simulated CATs, the mean test length was more than halved (≤ 8 items), while the range of precise measurement (standard error ≤ 0.3) was comparable to the full AAQ. The correlations between original AAQ scores and three AAQ-CAT versions were ≥ 0.95. Correlations of AAQ-CAT scores with patient-reported and performance measures of activity limitations were ≥ 0.60.ConclusionThe almost non-verbal AAQ-CAT is an innovative and efficient tool in patients with hip/knee osteoarthritis from various countries, measuring activity limitations with lower respondent burden, but similar precision and construct validity compared to the full AAQ.

Background: Demoralization, anxiety, irritability, and depression are common among hospital patients and are associated with poorer outcomes and greater healthcare burden. Early identification is essential, but simultaneous screening across multiple domains is often impractical with questionnaires. Computerized Adaptive Testing (CAT) offers a solution by tailoring item administration, reducing test length while preserving measurement precision. The aim of this study was to develop and validate FRIDA (Four-factor Rapid Interactive Diagnostic Assessment), a freely accessible, web-based CAT for rapid multidimensional screening of psychopathology in hospital patients. Methods: We analysed data from 472 medically ill in-patients at a University Hospital. Item calibration was performed using a four-factor graded response model (demoralization, anxiety, irritability, depression) in the mirt package. CAT simulations were run with 1000 virtual respondents to optimize item selection, exposure control, and stopping rules. The best configuration was applied to the real dataset. Criterion validity for demoralization was evaluated against the Diagnostic Criteria for Psychosomatic Research (DCPR). Results: The four-factor model showed good fit (CFI = 0.947, RMSEA = 0.080). Factor correlations were moderate to high, with the strongest overlap between demoralization and depression (r = 0.93). In simulations, the CAT required, on average, 7.8 items and recovered trait estimates with high accuracy (r = 0.94-0.97). In real patients, mean test length was 11 items, with accuracy of r = 0.95 across domains. FRIDA demonstrated good criterion validity for demoralization (AUC = 0.816; sensitivity 80%, specificity 67.5%). Conclusions: FRIDA is the first freely available, multidimensional CAT for rapid screening of psychopathology in hospital patients. It offers a scalable, efficient, and precise tool for integrating mental health assessment into routine hospital care.

This monte carlo study assessed the accuracy of simultaneous estimation of item and person parameters in item response theory. Item responses were simulated using the two- and three-parameter logistic models. Samples of 200, 500, 1,000, and 2,000 simulated examinees and tests of 15, 30, and 60 items were generated. Item and person parameters were then estimated using the appropriate model. The root mean squared error between recovered and actual item characteristic curves served as the principal measure of estimation accuracy for items. The accuracy of estimates of ability was assessed by both correlation and root mean squared error. The results indicate that minimum sample sizes and tests lengths depend upon the response model and the purposes of an investigation. With item responses generated by the two-parameter model, tests of 30 items and samples of 500 appear adequate for some purposes. Estimates of ability and item parameters were less accurate in small sample sizes when item responses were generated by the three-parameter logistic model. Here samples of 1,000 examinees with tests of 60 items seem to be required for highly accurate estimation. Tradeoffs between sample size and test length are apparent, however.

The aim of this study is twofold. The first one is to investigate the effect of sample size and test length on the estimation of item parameters and their standard errors for the two parameter item response theory (IRT). Another is to provide information about the performance of Mplus, BILOG-MG and R (ltm) programs in terms of parameter estimation under the conditions which were mentioned above. The simulated data were used in this study. The examinee responses were generated by using the open-source program R. After obtaining the data sets, the parameters were estimated in BILOG-MG, Mplus and R (ltm). The accuracy of the item parameters and ability estimates were evaluated under six conditions that differed in the numbers of items and examinees. After looking at the resulting bias and root mean square error (RMSE) values, it can be concluded that Mplus is an unbiased program when compared to BILOG-MG and R (ltm). BILOG-MG can estimate parameters and standard errors close to the true values, when compared to Mplus and R (ltm).

Despite tremendous efforts to increase the reliability of pain measures and other self-report instruments, improving or even evaluating the reliability of change scores has been largely neglected. In this study, we investigate the ability of 2 instruments from the Patient-Reported Outcomes Measurement Information System, pain interference (6 items) and pain behavior (7 items), to reliably detect individual changes in pain during the postsurgical period of a hernia repair in 98 patients who answered daily diaries over almost 3 weeks after surgery. To identify the most efficient strategy for obtaining sufficiently reliable estimates of change (reliability >0.9), the number of measurement occasions over the study period (sampling density), the number of items (test length), and the mode of administration (ie, static short form vs Computer adaptive testing) were manipulated in post-hoc simulations. Reliabilities for different strategies were estimated by comparing the observed change with the best approximation of "real" (ie, latent) change. We found (1) that near perfect reliability can be achieved if measures from all days over the whole study period, obtained with all pain interference or pain behavior items, were used to estimate the observed change, (2) that various combinations of the number of items and the number of measurement occasions could achieve acceptable reliability, and (3) that computer adaptive testings were superior to short forms in achieving sufficient reliability. We conclude that the specific strategy for assessing individual postoperative change in pain experience must be selected carefully.

Computerized adaptive testing (CAT) is a modern alternative to classical paper and pencil testing. CAT is based on an automated selection of optimal item corresponding to current estimate of test-taker’s ability, which is in contrast to fixed predefined items assigned in linear test. Advantages of CAT include lowered test anxiety and shortened test length, increased precision of estimates of test-takers’ abilities, and lowered level of item exposure thus better security. Challenges are high technical demands on the whole test work-flow and need of large item banks. In this study, we analyze feasibility and advantages of computerized adaptive testing using a Monte-Carlo simulation and post-hoc analysis based on a real linear admission test administrated at a medical college. We compare various settings of the adaptive test in terms of precision of ability estimates and test length. We find out that with adaptive item selection, the test length can be reduced to 40 out of 100 items while keeping the precision of ability estimates within the prescribed range and obtaining ability estimates highly correlated to estimates based on complete linear test (Pearson’s $\rho$=0.96). We also demonstrate positive effect of content balancing and item exposure rate control on item composition.

To evaluate and compare the measurement precision and sensitivity to change of the Health Assessment Questionnaire disability index (HAQ DI), the Short Form 36 physical functioning scale (PF-10), and simulated Patient-Reported Outcomes Measurement Information System (PROMIS) physical function computer adaptive tests (CATs) with 5, 10, and 15 items, using item response theory-based simulation studies. The measurement precision of the various physical function instruments was evaluated by calculating root mean square errors (RMSEs) between true physical function levels (latent physical function score) and estimated physical function levels. Measurement precision was evaluated at 9 levels of physical function, with 5,000 simulated response patterns per level. Sensitivity to change was evaluated by the ability of a simple statistical test to detect simulated change scores of small to moderate magnitude (standardized effect sizes 0.20, 0.35, and 0.50). RMSEs were smaller for the PROMIS physical function 15-item CAT (CAT-15) and CAT-10 than for the HAQ DI and PF-10 across all levels of the latent physical function scale. Only marginal improvement in performance was observed for the CAT-15 compared with the CAT-10, and the CAT-5 performed quite similarly to the HAQ DI and PF-10 across most levels of the latent physical function scale. Substantially improved sensitivity to change was observed for the CAT-10 compared with the HAQ DI and PF-10, particularly in detecting moderate effect sizes. Clearly higher measurement precision was observed for the PROMIS CAT compared with the HAQ DI and PF-10. Higher reliability also translated into lower sample size requirements for detecting changes in clinical status.

Purpose: We developed the Arm Function in Multiple Sclerosis Questionnaire (AMSQ) to measure arm and hand function in MS, based on existing scales. We aimed at developing a unidimensional scale containing enough items to be used as an itembank. In this study, we investigated reliability and differential item functioning of the Dutch version. Method: Patients were recruited from two MS Centers and a Dutch website for MS patients. We performed item factor analysis on the polychoric correlation matrix, using multiple fit-indices to investigate model fit. The graded response model, an item response theory model, was used to investigate item goodness-of-fit, reliability of the estimated trait levels (θ), differential item functioning, and total information. Differential item functioning was investigated for type of MS, gender, administration version, and test length. Results: Factor analysis results suggested one factor. All items showed p-values of the item goodness-of-fit statistic above 0.0016. The reliability was 0.95, and no items showed differential item functioning on any of the investigated variables. Conclusion: AMSQ is a unidimensional 31-item questionnaire for measuring arm function in MS. Because of a well fit in a graded response model, it is suitable for further development as a computer adaptive test.Implications for RehabilitationA new questionnaire for arm and hand function recommended in people with multiple sclerosis (AMSQ).Scale characteristics make the questionnaire suitable for use in clinical practice and research.Good reliability.Further development as a computer adaptive test to reduce burden of (repetitive) testing in patients is feasible.

The aim of the present research is to calculate the Relative Accuracy (R.A) and Root Mean Square Error (RMSE) for the location of Technical Institute, Mosul in Nineveh Governorate, Iraq. at the northern part of Mosul city by using surveying Instruments (GPS 1230 , Total Station 06 & Level NAK2), to calculate the coordinates and elevations of (16) Ground Control Points which covered the study area. The field data, remote sensing techniques and GIS software (Arc Map GIS10.3) are used to check the digital map of study area that helped the administrator to take the correct decision, determining positions of the future expansionist for buildings, roads, measuring lengths, areas and there coordinates, directions as well as the total costs estimation needed for any development from the final map. The gradient percentage between the highest and lowest points is calculated to be (9.954%). Relative accuracy for the based measured points (R.A) = 1:60000), and Root Mean Square error in Easting coordinates (RMSE) = ± 0.102m, in Northing coordinates (RMSN) = ± 0.096m, the Root Mean Square for the resultant (RMSR) = ± 0.099m, and Root Mean Square for Elevations ( RMSRL)= ± 0.005m. We Matched and correlated (10) measured points from study area with their graphic data positions on the aerial photo, determining the offsets for position and elevations as follows: (RMSE = ± 0.076m), (RMSN = ± 0.125m), (RMSR = ± 0.088m), and (Root Mean Square for Elevations = RMSRL= ± 0.045m). This research enable us after checking the results to produce new maps from Aerial photograph or Digital Image to use it for land uses and future extensions in whole study area.

BackgroundTo drive improvement in clinical services, an important innovation will be to regularly assess patients’ psychotic experiences in order to guide, monitor and, when needed, alter treatment provision. The great heterogeneity in presentations of psychosis means that a comprehensive assessment battery is impractical. A plausible solution is computerised adaptive testing (CAT), which uses real-time computation to present the most informative questions to an individual. Fewer questions are needed to reach similar precision as a full questionnaire.ObjectiveWe tested the potential of a CAT for paranoia to halve the number of items that need to be presented.MethodsWe used the established item response theory psychometric properties of the 10-item Revised Green et al Paranoid Thoughts Scale (Persecution) to run CAT simulations in four datasets in which participants had completed the full scale: a representative survey of 10 382 UK adults; a clinical trial with 319 patients with psychosis; a cohort study of 836 National Health Service (NHS) male patients with psychosis; and a clinical trial with 89 patients with persecutory delusions. The CAT algorithm used the graded response model and the test was concluded when the SE of estimation dropped below 0.3 or five items had been answered.FindingsOn average, the CAT administered 4.2, 4.0, 4.2 and 4.0 items to each person in the four datasets. The correlations between the CAT score and the full-scale paranoia score were 0.95, 0.94, 0.94 and 0.87. Minimal systematic error in paranoia estimation occurred (mean bias scores=−0.01, –0.06, −0.07 to –0.10). Estimation was the least precise for people at the boundary of normal and elevated levels of paranoia.ConclusionsIn datasets with people across the whole paranoia continuum, accurate estimates of paranoia can be provided by a CAT with fewer than half the items of the full scale. Tailored testing may work well with people with psychosis.Clinical implicationsCAT may be a way to implement informative measurement-based care in psychosis services.

BackgroundThe multileaf collimator (MLC) is a critical component to accurate intensity-modulated radiotherapy (IMRT) delivery. This study examined MLC positional accuracy via MLC logs from Varian machines from six institutions and three delivery techniques to evaluate typical positional accuracy and treatment and mechanical parameters that affect accuracy. Typical accuracy achieved was compared against TG-142 recommendations for MLC performance; more appropriate recommendations are suggested.MethodsOver 85,000 Varian MLC treatment logs were collected from six institutions and analyzed with FractionCHECK. Data were binned according to institution and treatment type to determine overall root mean square (RMS) and 95th percentile error values, and then to look for correlations between those errors and with mechanical and treatment parameters including mean and maximum leaf speed, gantry angle, beam-on time, mean leaf error, and number of segments.ResultsResults of treatment logs found that leaf RMS error and 95th percentile leaf error were consistent between institutions, but varied by treatment type. The step and shoot technique had very small errors: the mean RMS leaf error was 0.008 mm. For dynamic treatments the mean RMS leaf error was 0.32 mm, while volumetric-modulated arc treatment (VMAT) showed an RMS leaf error of 0.46 mm. Most MLC leaf errors were found to be well below TG-142 recommended tolerances. For the dynamic and VMAT techniques, the mean and maximum leaf speeds were significantly linked to the leaf RMS error. Additionally, for dynamic delivery, the mean leaf error was correlated with RMS error, whereas for VMAT the average gantry speed was correlated. For all treatments, the RMS error and the 95th percentile leaf error were correlated.ConclusionsRestricting the maximum leaf speed can help improve MLC performance for dynamic and VMAT deliveries. Furthermore, the tolerances of leaf RMS and error counts for all treatment types should be tightened from the TG-142 values to make them more appropriate for clinical performance. Values of 1 mm for the 95th percentile of leaf RMS error and 1.5 mm for the 95th percentile leaf error are suggested as action levels for all treatment types.

OPINION article Front. Physiol., 22 January 2014Sec. Integrative Physiology Volume 4 - 2013 | https://doi.org/10.3389/fphys.2013.00403