Evaluating the Construct Validity of Instructional Manipulation Checks as Measures of Careless Responding to Surveys.

Careless response and attrition as sources of bias in online survey assessments of personality traits and performance

With the increasing publication of self-report online studies, concerns are growing about the quality of the data collected through these methods. This study focused on response bias, a major threat to data quality, by analyzing data from a real-world study on adult temperament conducted in two different countries. The sample included 1,497 participants aged 18-80 years from the United States (n = 598) and Lithuania (n = 899). The primary objectives were to determine the prevalence of response bias and to evaluate its impact on psychometric outcomes. Indicators of biased responding included patterns suggestive of potentially careless responding (e.g., invariant and random response patterns) and those flagged by internal validity checks or clinical controls (e.g., social desirability and ratings-perception discrepancies). Results indicated that the inclusion of data reflecting potentially careless responding reduced internal consistency and distorted factor structure, whereas its exclusion improved these psychometric indicators. In contrast, with regard to clinical controls, removing flagged data resulted in a decline in psychometric quality. Additionally, higher rates of careless responding were observed in the sample subjected to forced answering. These findings highlight the importance of mitigating response bias in online self-report research and raise broader questions about the integrity of data in existing survey-based datasets. By jointly evaluating careless responding and clinical threats in real-world, cross-national samples, this study extends prior work and demonstrates the applied value of post-hoc screening for improving psychometric quality.

Careless responses by survey participants threaten data quality and lead to misleading substantive conclusions that result in theory and practice derailments. Prior research developed valuable precautionary and post-hoc approaches to detect certain types of careless responding. However, existing approaches fail to detect certain repeated response patterns, such as diagonal-lining and alternating responses. Moreover, some existing approaches risk falsely flagging careful response patterns. To address these challenges, we developed a methodological advancement based on first-order Markov chains called Lazy Respondents (Laz.R) that relies on predicting careless responses based on prior responses. We analyzed two large datasets and conducted an experimental study to compare careless responding indices to Laz.R and provide evidence that its use improves validity. To facilitate the use of Laz.R, we describe a procedure for establishing sample-specific cutoff values for careless respondents using the “kneedle algorithm” and make an R Shiny application available to produce all calculations. We expect that using Laz.R in combination with other approaches will help mitigate the threat of careless responses and improve the accuracy of substantive conclusions in future research.

Affective organizational commitment is a construct important to both practitioners and researchers; thus, construct-valid measurement is needed. Allen and Meyer’s (J Occup Psychol 63:1–18, 1990) Affective Commitment Scale is a popular measure of affective organizational commitment. Although conceptualized as unidimensional, a two-factor solution sometimes emerges. Whether the two factors are substantive, reflecting AC-love and AC-joy constructs, or methodological, reflecting positively and negatively worded items, is unclear. This issue is examined in five studies. In a new approach, conditions designed to produce cognitive fatigue are manipulated. Support for the method factor interpretation would be provided if the two-factor solution emerges only when participants are fatigued and negatively worded items are present. Cross-sample analyses are also conducted. Analyses indicated that the two-factor solution fit the data well only when (a) participants were cognitively fatigued and (b) negatively worded items were present. This finding extended to both students and employees, and it held regardless of which items were negatively worded or whether negatively worded items were emphasized. Results suggest that the second factor is likely methodological and seems to result from careless responding or fatigue on negatively worded items. It is suggested that users modify the items to be positively worded or administer the scale when respondents have sufficient cognitive resources to respond. This study was the first to use experimental methodology to test whether the two-factor solution is methodological or substantive. Results provide guidance for improving the scale’s construct validity in research and in practical applications.

In this paper, we examine rates of careless responding and reactions towards detection methods (i.e., attention check item and instruction) in an experimental setting based on two different samples. First, we use a quota sample (with monetary incentive), a central data source for internet-based surveys in sociological and political research. Second, we include a voluntary opt-in panel (without monetary incentive) well suited to conduct survey experiments (e.g., factorial surveys). Respondents’ reactions towards the detection items are analyzed by objective, non-reactive indicators (i.e., break-off, item nonresponse, and measurement quality), and two self-report scales. Our reaction analyses reveal that the detection methods that we applied are not only well suited to identify careless respondents but also exert a motivational rather than demotivating influence on respondents’ answer behavior and, hence, contribute to data quality.

The Spanish version of the five-item Francis scale of attitude toward Christianity is a refinement of the short version of the Francis scale of attitude toward Christianity. The scale is a good measurement for intrinsic religiosity. It has been applied previously among Colombian adolescent students. The internal consistency and construct and nomological validity were tested among a sample of 1349 Colombian college students from Santa Marta, Colombia. Participants were aged between 18- and 30-years-old; 51% were men; and 68% were social science students. Exploratory analysis factor, Cronbach alpha and omega McDonald were computed. The scale showed high internal consistency (both alpha and omega = .96) and excellent construct and nomological validity (one factor which explained for 88% of variance; women scored higher than men; and Protestants scored higher than Catholics and students with other religion filiations). The five-item Francis scale can be used further researches in Hispano-America among college students.

Major changes in thinking about validity have occurred during the past century, shifting the focus in thinking from the validity of the test to the validity of test score interpretations. These changes have resulted from the 'new' thinking about validity in which construct validity has emerged as the central or unifying idea of validity today. Construct validity was introduced by Cronbach and Meehl in the mid-1950s in an attempt to address the validity of those many psychological concepts that have no clear referent in reality. To do this, construct validity theory required a nomological network--an elaborate theoretical network of constructs and observations connected by scientific laws--to validate the constructs. However, nomological networks are hard to come by and none that would do the job required by construct validity has been forthcoming to date. Thus, the current construct validity approach has retreated to one of simply 'interpretation and argument', but this seems to be too general to tie down the constructs in the way a nomological network would do to give credibility to the validity of the construct. As a result, the concept of validity seems to have been watered down and the credibility of validity claims weakened. The purpose of this paper is to encourage a discussion of the use of construct validity in medical education, and to suggest that test developers and users reconsider the use of abstract theoretical constructs that have no referent apart from theory. We present a critical review of these concerns about construct validity and provide for contrast a brief overview of a recently proposed view of measurement based on scientific realism and causality analysis.

The present research addresses three key questions about social, emotional, and behavioral (SEB) skills. First, how do SEB skills relate with the Big Five traits and Collaborative for Academic, Social, and Emotional Learning (CASEL) core competencies? Second, how do SEB skills relate with consequential outcomes in adolescence? Third, do SEB skills provide incremental validity beyond personality traits? Results from a diverse sample of high school students (N = 897) indicate that SEB skills converge with the Big Five traits and CASEL competencies in expected and conceptually meaningful ways. Analyses of self-reported and school-reported outcomes extend SEB skills’ nomological network by showing that they predict academic achievement and engagement, occupational interests, social relationships, civic engagement, and well-being. Finally, tests of incremental validity indicate that SEB skills provide unique information beyond personality traits and that this information matters for predicting outcomes during adolescence. These findings advance our understanding of the nature, correlates, and consequences of SEB skills.

IntroductionWhen datasets are collected mainly for administrative rather than research purposes, data quality checks are necessary to ensure robust findings and to avoid biased results due to incomplete or inaccurate data.
 When done manually, data quality checks are time-consuming. We introduced automation to speed up the process and save effort.
 Objectives and ApproachWe have devised a set of automated generic quality checks and reporting, which can be run on any dataset in a relational database without any dataset-specific knowledge or configuration.
 The code is written in Python. Checks include: linkage quality, agreement with a population data source, comparison with previous data version, duplication checks, null count, value distribution and range, etc.
 Where dataset metadata is available, checks for validity against lookup tables are included, and the output report includes documentation on data contents. An HTML report with dynamic datatables and interactive graphs, allowing easy exploration of the results, is produced using RMarkdown.
 ResultsThe automation of the generic data quality check provides an easy and quick tool to report on data issues with minimal effort. It allows comparison with reference tables, lookups and previous versions of the same table to highlight differences. Moreover, this tool can be provided for researchers as a means to get more detailed understanding about their data.
 While other research data quality tools exist, this tool is distinguished by its features specific to linked data research, as well as implementation in a relational database environment. It has been successfully tested on datasets of over two billion rows.
 The tool was designed for use within the SAIL Databank, but could easily be adapted and used in other settings.
 Conclusion/ImplicationsThe effort spent on automating generic testing and reporting on data quality of research datasets is more than compensated by its outputs. Benefits include quick detection and scrutiny of many sources of invalid and incomplete data. This process can easily be expanded to accommodate more standard tests.

Large-scale multiparameter screening has become increasingly feasible and straightforward to perform thanks to developments in technologies such as high-content microscopy and high-throughput flow cytometry. The automated toolkits for analyzing similarities and differences between large numbers of tested conditions have not kept pace with these technological developments. Thus, effective analysis of multiparameter screening datasets becomes a bottleneck and a limiting factor in unbiased interpretation of results. Here we introduce compaRe, a toolkit for large-scale multiparameter data analysis, which integrates quality control, data bias correction, and data visualization methods with a mass-aware gridding algorithm-based similarity analysis providing a much faster and more robust analyses than existing methods. Using mass and flow cytometry data from acute myeloid leukemia and myelodysplastic syndrome patients, we show that compaRe can reveal interpatient heterogeneity and recognizable phenotypic profiles. By applying compaRe to high-throughput flow cytometry drug response data in AML models, we robustly identified multiple types of both deep and subtle phenotypic response patterns, highlighting how this analysis could be used for therapeutic discoveries. In conclusion, compaRe is a toolkit that uniquely allows for automated, rapid, and precise comparisons of large-scale multiparameter datasets, including high-throughput screens.

The H & H Lactation (H & H) scale is a useful instrument for understanding mothers' perceptions of insufficient milk supply, which is the common reason given for premature cessation of breastfeeding. This instrument may be used to intervene in postbirth dropoff in breastfeeding among mothers. To examine construct and nomological validity and related psychometric evidence for the translated H & H scale using a repeated measurement design in a sample of Thai mothers. A repeated measurement design with 196 mother-infant dyads. Data were collected at two time periods. At Time 1, data on the H & H and Index of Breastfeeding Status (IBS) scales were obtained between 36 and 48 hr postbirth. For Time 2, the same mothers completed the same instruments at 1 month postbirth. Psychometricproperties of the H & H scale were examined using convergent, discriminant, and nomological validity principles with instructural equation modeling. Based on judgment/classification task, exploratory/confirmatory factor analysis, and cross-validation using multisample analysis, only nine items of the H & H scale measured at Time 1 and Time 2 were found to be psychometrically sound for measuring insufficient milk supply for Thai mothers. An analysis of the influence of the nine-item of the H & H scale measured at Time 1 and Time 2 on the IBS measured at Time 2 showed that (a) the direct effects of two subscales of the H & H scale measured at Time 2 on the IBS were significant and positive, providing support for the nomological validity of the H & H scale; (b) each H& H factor at Time 1 had a significant influence on its corresponding factor at Time 2, supporting convergent validity; and (c) the crossover effects for each H & H factor for Time 1 and Time 2 data were nonsignificant, indicating discriminant validity. Overall, the 20-item H & H scale appears to be unreliable and invalid for measuring the concept of insufficient milk supply in cross-cultural studies. However, the short form of nine items is reliable and valid for Thai mothers. Further testing and development of the H & H scale and the proposed short form in other cultures and contexts is warranted.

Purpose– This paper aims to develop and validate a scale measurement of trust in long-term relations, specifically in the maintenance phase, between buyers and sellers. Relying on a cognitive conception, existing scales do not measure the affective trust occurring in the maintenance stage of the relationship.Design/methodology/approach– Three surveys were conducted with purchasing managers of enterprises in various business-to-business (B2B) sectors to build a bi-dimensional measurement scale of affective trust as a sentiment of security and affective attachment. For measurement scale development, established construction procedures were followed, including qualitative and quantitative surveys. The process can be summarized as: domain specification, generation of questionnaire items, empirical survey and iterative process of scale purification based on reliability assessment and validity checks.Findings– The results indicate satisfactory psychometric properties of the new Affective Trust Scale (ATS). Furthermore, they demonstrate the scale’s measurement invariance across business sectors. The research confirms the importance of affective aspects of trust and supports the reliability and validity of the measure. Nomological validity assessment of the scale shows that sentiment of security-based trust impacts investment in business relationships.Research limitations/implications– The ATS developed and tested in the B2B French context needs to be evaluated taking into account several limitations. First, the specific context was a sample of buyers in France experiencing lasting relationships, suggesting that an extension of the study to other countries would be desirable. Also, the ATS needs to be further validated and confirmed in other contexts, for instance, within buyer–supplier relationship intensity.Practical implications– The ATS can help firms to identify key parameters in buyer–seller relationships. It is important for the seller to collect information to determine the stage of the relationship before spending money on targeting customers, as they may not be ready to broaden the scope of their contract. The ATS can be very useful for companies to assess the state of the relationship and the strength of the bond in a timely manner and, therefore, anticipate the relational orientation. Segmentation based on relational phases requires tailoring to each form of trust strategies and hence the accurate identification of the relationship phase could help to better categorize and subcategorize customers with respect to the sentiment of security and affective attachment degrees. Furthermore, an understanding of the two dimensions is useful for key account managers to adjust relationship management toward specific actions (e.g. sentiment of security and/or affective attachment). The ATS could be very useful to guide managers in taking the right decision, by focusing both on sentiment of security and affective attachment dimensions of affective trust.Originality/value– Affective trust is important to B2B managers, who frequently struggle to build and maintain close relationships with customers and suppliers. This paper adds insights into the complex but important construct of trust. The scale could be used for empirical studies of affective trust in B2B relationships. It may also help marketing managers develop better relationships with partners.

The Internet addiction components model and personality: Establishing construct validity via a nomological network

The DSM-5 Section III model of personality disorders remains largely unexplored in older adults. More specifically, there is a need for further research on the generalizability of the five trait domains in old age. The development of a short operationalization to screen for maladaptive trait domains, the Personality Inventory for DSM-5 Brief Form (PID-5-BF), can stimulate the use of the alternative DSM-5 model in older adults by addressing the need for short instruments. The primary goal of this study was to examine the construct validity of the PID-5-BF by comparing its structural model and nomological network with the original PID-5 in terms of relations with domains of personality functioning and a gero-specific personality disorder indicator. A five-factor model was supported, but the domain Disinhibition was not replicated in the original PID-5, and some PID-5-BF items showed weak loadings. Nevertheless, the nomological network was similar and showed meaningful relations, supporting the use of the Brief PID-5 in older clinical samples.

Vein-To-Vein Databases: Uses and Considerations in Transfusion Research.