History repeating: guidelines to address common problems in psychedelic science

The aim of this paper is to acquaint pain researchers and practitioners with recent developments in the single-case experimental approach and their potential to allow for tailoring the treatment and its evaluation to the specific complaints, aptitudes, or profile of the individual patient, without violating the canons of good science and practice. After contrasting the single-case experimental approach and the case-study approach, we show the possibilities of customization in design, measurement, and test statistics. This is done by distinguishing 2 types of single-case designs--alternation designs and phase designs--and 2 types of replication strategies--simultaneous replications and sequential replications. In addition, tailor-made randomization tests are proposed for alternation, phase, and simultaneous replication designs and the combining of P values to perform a meta-analysis on designs that are sequentially replicated. With our emphasis on: 1) randomization in the design; 2) the possibilities for a statistical test (together with the determination of power and the calculation of effect sizes); 3) the importance of reliable and valid measurement; and 4) the role of replication, we demonstrate how internal validity, statistical-conclusion validity, construct validity, and external validity concerns can be dealt with within a single-case experimental approach framework. Finally, the many research examples and references to clinical work illustrate the usefulness of the approach.

Cross-cultural adaptation and psychometric analysis. To develop the Turkish version of the Fremantle Back Awareness Questionnaire (FreBAQ-T) and to evaluate its psychometric properties. There is a growing interest in the role that the disturbance of body perception may lead to long-lasting pain problems such as chronic low back pain (CLBP). The Fremantle Back Awareness Questionnaire (FreBAQ) is a simple and low-cost way of evaluating disturbed back awareness in people with CLBP. The FreBAQ was translated from English into Turkish using the forward-backward-forward method. One hundred four participants with low back pain completed the FreBAQ-T. The FreBAQ-T was repeated in 15 participants after 1 week to establish test-retest reliability. Although internal and external construct validity was investigated using Rasch analysis and Spearman correlation coefficient, respectively; reliability was evaluated in terms of internal consistency by Cronbach alpha and Person Separation Index. All items of the FreBAQ-T were found to fit the Rasch Model (chi-square 6.17 [df = 9], P = 0.723). The internal construct validity was good, overall mean item fit residual was -0.305 (standard deviation: 0.369) and mean person fit residual was -0.290 (standard deviation: 1.349). The reliability was good with Cronbach alpha of 0.87 and Person Separation Index of 0.82. When the test-retest was examined via differential item functioning by time, none of the items showed differential item functioning. The FreBAQ-T is a valid, reliable, and unidimensional scale for patients with CLBP. This scale will allow assessing back-specific perception in the Turkish population with CLBP. 3.

The aim of this descriptive study was to evaluate the construct validity and reliability of the Behavioural Inattention Test (BIT) in patients with acquired brain injury in Turkey. One hundred and eighteen acquired brain injury patients undergoing rehabilitation were assessed by the BIT. Internal construct validity was tested by Rasch analysis; reliability by internal consistency and the Person Separation Index; and external construct validity by associations with physical and cognitive disability. Analysis of the data revealed that some subtests deviated from Rasch model expectation and the conventional subscale of the BIT had an unsatisfactory reliability for individual use. Consequently, a common 10-item scale (BIT-10) was derived from both the behavioural and conventional subscales of the BIT. Reliability of .87 met expectation for individual use. The BIT-10 correlated at .52 with cognitive disability upon admission. As a conclusion the original BIT adapted for use in Turkey was shown to lack reliability and internal construct validity. A revised 10-item new version, BIT-10, gave a valid unidimensional summed score, with high sensitivity and specificity to the original cut points. Reliability of the BIT-10 was high and external construct validity was as expected.

This article discusses the tendency for some researchers to privilege positivistic and post-positivist epistemologies and certain methodologies (e.g., experimental and quantitative methods) over others in the name of scientific rigor. We argue that the uncritical privileging of certain research designs over others contributes to scientific bias and potentially scientific racism. There has been a longstanding epistemological debate about which research methodologies produce rigorous research. Those who argue that only research that examines causal links is considered real science neglect the other stages of the scientific method which also consider description, observation, and prediction, in addition to explanation. The focus on causal questions as the epitome of “real” science dismisses earlier stages of science that must first describe, observe, and predict before it can explain phenomena. We discuss less emphasized forms of validity (e.g., statistical conclusion validity, external validity, and construct validity), which are equally important to internal validity in addressing scientific rigor. We also discuss how the tendency to conflate statistical analysis procedures with research design works to dismiss rigorous research not focused on causal questions. This is especially problematic for research that examines underrepresented groups and marginalized populations who are typically also hard-to-reach research populations. Finally, we review previous epistemological debates that privileged certain quantitative methods over other quantitative methods and qualitative approaches and offer recommendations for how to combat scientific racism in psychological science.

Purpose. Assessment of cognitive impairment with a valid cognitive screening tool is essential in neurorehabilitation. The aim of this study was to test the reliability and validity of the Turkish-adapted version of the Middlesex Elderly Assessment of Mental State (MEAMS) among acquired brain injury patients in Turkey.Methods. Some 155 patients with acquired brain injury admitted for rehabilitation were assessed by the adapted version of MEAMS at admission and discharge. Reliability was tested by internal consistency, intra-class correlation coefficient (ICC) and person separation index; internal construct validity by Rasch analysis; external construct validity by associations with physical and cognitive disability (FIM); and responsiveness by Effect Size.Results. Reliability was found to be good with Cronbach's α of 0.82 at both admission and discharge; and likewise an ICC of 0.80. Person separation index was 0.813. Internal construct validity was good by fit of the data to the Rasch model (mean item fit −0.178; SD 1.019). Items were substantially free of differential item functioning. External construct validity was confirmed by expected associations with physical and cognitive disability. Effect size was 0.42 compared with 0.22 for cognitive FIM.Conclusion. The reliability and validity of the Turkish version of MEAMS as a cognitive impairment screening tool in acquired brain injury has been demonstrated.

To assess the internal consistency, construct validity and sensitivity to change of a pelvic organ prolapse symptom score (POP-SS). Analysis of data from three prolapse studies, including symptomatic and asymptomatic women who completed the POP-SS. (1) A community setting in New Zealand, (2) two gynaecology outpatient departments in Scotland and (3) a gynaecological surgery department in Scotland. (1) Participants from a survey of postnatal women at 12-year follow up, invited to complete a prolapse questionnaire and have prolapse assessment, (2) new gynaecology outpatients presenting with prolapse symptoms, randomised to pelvic floor muscle training (PFMT) or control and (3) women having anterior and/or posterior prolapse surgery, randomised to mesh insert or no mesh. Data were analysed to assess internal consistency, construct validity and sensitivity to change of the POP-SS. Cronbach's alpha, significance of differences in POP-SS scores between studies and significance of difference in POP-SS scores pre- to post-intervention. For internal consistency, Cronbach's alpha ranged from 0.723 to 0.828. Women having surgery had higher POP-SS scores than those having conservative management (mean difference 5.0, 95% CI 3.1-6.9), who in turn had higher scores than the asymptomatic women (mean difference 5.9, 95% CI 4.4-7.4). Significant differences in POP-SS score were detected after surgery and PFMT. The improvement due to surgery was significantly greater than that associated with PFMT (z =-3.006, P = 0.003). The POP-SS has good internal consistency and construct validity and is sensitive to change.

Our aim was to validate a culturally adapted, Tunisian-language version of the Roland-Morris Disability Questionnaire (RMDQ), which is a reliable evaluation instrument for low-back-pain disability. A total of 62 patients with low back pain were assessed by the questionnaire. Reliability for the 1-week test/re-test was assessed by a construction of a Bland Altman plot. Internal construct validity was assessed by Cronbach's alphatest. External construct validity was assessed by association with pain, the Schober test and the General Function Score. Sensitivity to change was determined using a t-test for paired data to compare RMDQ scores at inclusion and at completion of the therapeutic sequence of local corticosteroid injections. We also compared the questionnaire score with the General Function Score, both taken after completion of the therapeutic sequence. The constructed Bland Altman plot showed good reliability. Internal consistency of the RMDQ was found to be very good and the Cronbach's alpha test was 0.94, indicating a good internal construct validity. The questionnaire is correlated with the pain visual analogue scale (r=33; p=0.0001), with the Schober test (r=0.27; p=0.0001) and the General Function Score (r=56; p=0.0001) indicating an adequate external construct validity. The RMDQ administered after the therapeutic sequence is sensitive to change (r=0.83; p=0.000). Comparison of the questionnaire score to the General Function Score, after completion of the therapeutic sequence, was satisfactory (r=0.75; p=0.000). We conclude that the Tunisian version of the Roland-Morris questionnaire has good reliability and internal consistency. Furthermore, it has a good internal- and external construct validity and high sensitivity to change. It is an adequate and useful tool for assessing low-back-pain disability.

IntroductionIdiopathic multicentric Castleman disease (iMCD) is a rare, chronic, debilitating lymphoproliferative disorder where the mainstay of treatment is symptom management. Our recent international patient survey showed that patients with iMCD have a high symptom burden that has a significant negative patient-reported impact on several aspects of daily life. As part of our ongoing work towards the development of an iMCD symptom burden scale, assessing the survey’s psychometric properties is a critical step in understanding its adequacy, relevance, and usefulness. As iMCD is a rare disease, there are challenges to conducting such psychometric analyses which we describe.MethodsAs part of the exploratory psychometric analysis, three a priori hypothesis sets (HS) were generated by interviewing an iMCD-experienced clinician, a patient, and a caregiver to explore the iMCD patient survey’s internal construct validity, given no gold standard iMCD measure exists for external construct validation. HS-1 hypothesized that a convergent or discriminant relationship exists with the patients’ self-assessment of symptom effect on daily life between two potentially related or unrelated symptoms, respectively. HS-2 hypothesized that having a greater number of symptoms has a positive convergent relationship with the patients’ assessment of symptoms’ effect on daily life. Finally, HS-3 hypothesized that patients receiving treatment versus no treatment was associated with patients reporting less effect of symptom burden on their daily life. Spearman’s rank absolute correlation strength (ACS) was used for HS-1 and HS-2 (convergent relationship, ACS ≥ 0.3 and p value < 0.05; divergent relationship, ACS < 0.3), and Cohen’s d to quantify standardized absolute effect sizes (AES) for HS-3 (AES ≥ 0.5 and p value < 0.05).ResultsOur analyses partially supported HS-1. None of the three positive convergent relationships were supported. Of the six discriminant relationships, only dizziness with impaired cognitive function and tiredness with dizziness were supported. HS-2 analyses showed there was convergent validity between the number of symptoms and their effect on aspects of daily life. HS-3 analyses did not provide evidence to support the hypothesis.ConclusionThese internal psychometric construct analyses provide initial support for the bespoke iMCD patient survey and will guide additional work towards the development of the first iMCD-specific symptom burden scale.Supplementary InformationThe online version contains supplementary material available at 10.1007/s40487-024-00293-4.

Guidelines have been established for cross-cultural adaptation of outcome measures. However, invariance across cultures must also be demonstrated through analysis of Differential Item Functioning (DIF). This is tested in the context of a Turkish adaptation of the Health Assessment Questionnaire (HAQ). Internal construct validity of the adapted HAQ is assessed by Rasch analysis; reliability, by internal consistency and the intraclass correlation coefficient; external construct validity, by association with impairments and American College of Rheumatology functional stages. Cross-cultural validity is tested through DIF by comparison with data from the UK version of the HAQ. The adapted version of the HAQ demonstrated good internal construct validity through fit of the data to the Rasch model (mean item fit 0.205; SD 0.998). Reliability was excellent (alpha = 0.97) and external construct validity was confirmed by expected associations. DIF for culture was found in only 1 item. Cross-cultural validity was found to be sufficient for use in international studies between the UK and Turkey. Future adaptation of instruments should include analysis of DIF at the field testing stage in the adaptation process.

This article examines the methodological choices of researchers studying the HR practices–outcome relationship via a content analysis of 281 studies published across the past 20 years. The prevalence and trajectory of change over time are reported for a wide range of methodological choices relevant to internal, external, construct, and statistical conclusion validity. While the results indicate a high incidence of potentially problematic cross‐sectional, single‐informant, and single‐level designs, they also reveal significant improvements over time across many validity relevant methodological choices. This broad‐based improvement in the methodological underpinnings of HR research suggests that researchers and practitioners can view the findings reported in the HR literature with increasing confidence. Directions for future research are provided. © 2016 Wiley Periodicals, Inc.

Objective: To describe the development of the Adelaide Driving Self-Efficacy Scale (ADSES) and to report on its reliability and validity. Methods: A set of 12 driving behaviours, developed through literature review, clinical experience and expert review, were rated for self-efficacy using a Likert scale. Internal consistency was investigated using a Cronbach's alpha coefficient and construct validity by comparing ADSES scores of stroke and non-stroke drivers. Criterion-related validity was examined by comparing ADSES scores with the result on a standardized on-road assessment. Setting: A rehabilitation hospital in Adelaide, South Australia. Participants: Staff from the hospital and stroke patients from the rehabilitation unit. Data from a non-stroke sample (n -/ 79) and stroke patients (n -/ 81) were used to test internal consistency and construct validity. A separate group of 45 people recommended for a driving assessment, of whom 34 were stroke patients, were used to test criterion validity. Results: Cronbach's alpha coefficient was 0.98, indicating high internal consistency. The non-stroke and stroke groups showed significant differences in ADSES scores (t(158)-/ 5.5, P B < 0.05), demonstrating construct validity. Differences in ADSES scores for those participants who passed or failed the on-road assessment were significant for both the entire driving assessment group (t(43)-/ 3.2, P B < 0.05) and the stroke subgroup (t(43)-/ 3.2, P B < 0.05), indicating criterion validity. Conclusion: The ADSES has demonstrated internal consistency and construct validity with the stroke and non-stroke population. The scale demonstrated criterion validity in its relationship with outcome of an on-road driving assessment. It appears to be a reliable and valid measure of driving self-efficacy.

Special education research is an interdisciplinary field requiring a broad range of methodologies. This paper focuses on group research, which can be distinguished from single-subject design in several important ways: (1) group research involves the extensive and limited observations of many subjects rather than the intensive observations of a small number of subjects; (2) group research emphasizes the average performance of groups of subjects rather than the process of change itself; (3) results are evaluated through statistical analyses rather than visual interpretation of individual subject data; and (4) results are more broadly generalizable across subjects, settings, and occasions. The distinction between experimental and nonexperimental strategies is addressed within the context of group strategies. Four classes (Cook &amp; Campbell, 1979) of threats to valid inference which may be controlled by experimental designs are identified: internal validity, statistical conclusion validity, external validity, and construct validity of causes and effects.

This chapter provides a comprehensive, in-depth discussion of quantitative research methods in ISLA. It starts by informing the reader of the basic elements and principles of quantitative ISLA research and the major research questions examined. It then identifies three categories of ISLA research: experimental, correlational, and observational, followed by a discussion of the methods for evaluating the quality of quantitative research from the perspectives of construct validity, internal validity, external validity, and statistical conclusion validity. The chapter proceeds to offer advice for future researchers on how to carry out original and robust quantitative research and propose strategies to address common methodological issues. A conclusion is offered at the end.

To investigate the psychometric properties of the Rivermead Motor Assessment by Rasch analysis and conventional statistics to improve its clinical utility. A total of 107 patients after stroke were evaluated using the Rivermead Motor Assessment and Functional Independence Measure (FIM). Scaling properties were assessed using Mokken scaling, internal construct validity using Rasch analysis, reliability using Cronbach's alpha and intra-class correlation coefficients, external construct validity through convergent validity with FIM, and responsiveness using the effect size and standardized response mean. Cronbach's alpha and intra-class correlation coefficients for 3 sections of the Rivermead Motor Assessment were between 0.88 and 0.95. Mokken scaling showed appropriate Guttman patterns, but the hierarchical ordering of the items differed from that of the original. After removing 4 items of gross function, 1 of leg-trunk, and 4 of arm, all sections met Rasch model expectations. External construct validity was confirmed. Mean values of effect size and standardized response were 0.38-0.51 and 0.60-0.89, respectively. The Rivermead Motor Assessment has been shown to be reliable and responsive. Guttman scaling is apparent, but not as originally defined. After removing some items, the scale satisfies the most stringent Rasch measurement criteria and can produce interval scaling for the assessment of motor function in stroke.

level and is expected to have several distinct subdimensions, it is important to think about not only the relationships between the measures and these first-order subdimensions but also about the relationships between the first-order subdimensions and the second-order construct they measure. My colleagues and I (Jarvis et al. forthcoming) provide several examples of marketing constructs conceptualized at this level of abstraction (e.g., market orientation, trust, helping behavior, perceived risk, etc.) and call attention to the fact that the measurement model relating the first-order subdimensions to the measures need not be the same as the measurement model relating the second-order construct to its first-order subdimensions. Although one could reasonably argue that all constructs should be unidimensional, our review suggests that such a view is often inconsistent with the way constructs are defined in the marketing literature. So as a practical matter, this is something authors should think about carefully. Defend the Construct Domain and Insist on the Conceptually Appropriate Measurement Model Do not sacrifice construct validity at the altar of internal consistency reliability. Although this is good advice regardless of whether the measures are reflective or formative, it is particularly important to remember when your construct has formative measures, because formative indicator measurement models do not imply high levels of internal consistency reliability (Bollen and Lennox 1991). Therefore, when your measures are formative, it is important to resist the temptation to delete items as a means of improving Cronbach’s alpha (internal consistency reliability). Following this advice may be difficult if the temptation comes from a reviewer in the form of a recommendation. However, you must be vigilant, because the likelihood of inappropriately restricting the domain of the construct and threatening construct validity tends to be greater when formative indicators are eliminated than when reflective indicators (of a unidimensional construct) are eliminated (cf. Bollen and Lennox 1991). The best way to avoid this unpleasant situation is to head it off by carefully discussing the construct domain and the hypothesized relations between the construct and its measures, and explicitly noting the implications of your measurement model for how it should be evaluated. If internal consistency reliability is irrelevant, consider providing other evidence of reliability (e.g., item test-retest reliability) and/or empirically examining the sensitivity of your findings to different assumptions of reliability. In other words, try to gently educate the reviewers of your manuscript as a means of avoiding problems in the review process, always remembering that your arguments will be more convincing if they come in advance rather than in response to a reviewer’s criticisms. In conclusion, the problems of poor construct validity and statistical conclusion validity that plague many manuscripts can be minimized if you carefully define the focal constructs in your research, make sure that your measures fully represent them, correctly specify the relations between the measures and constructs in your measurement model, and stick to it. I believe that following this advice will greatly improve your chances of publication success— perhaps more than anything else I might recommend.