Split-Sample Design with Parallel Protocols to Reduce Cost and Nonresponse Bias in Surveys

ObjectiveThe objective is to estimate the relative contributions of nonresponse, coverage, and measurement biases in survey estimates of voting.MethodsWe survey 3,000 Boston‐area households sampled from an address‐based frame matched, when possible, to telephone numbers. A two‐phase sampling design was used to follow up nonrespondents from phone interviews with personal interviews. All cases were then linked to voting records.ResultsNonresponse, coverage, and measurement‐biased survey estimates at varying stages of the study design. Coverage error linked to missing telephone numbers biased estimates that excluded nonphone households. Overall estimates including nonphone households and nonrespondent interviews include 25 percent relative bias equally attributable to measurement and nonresponse.ConclusionBias in voting measures is not limited to measurement bias. Researchers should also assess the potential for nonresponse and coverage biases.

Reduction in nonresponse bias has been a key focus in responsive and adaptive survey designs, through multiple phases of data collection, each defined by a different protocol, and targeting interventions to a subset of sample elements. Key in this approach is the identification of nonrespondents who, if interviewed, can reduce nonresponse bias in survey estimates. From a design perspective, we need to identify an appropriate model to select targeted cases, in addition to an effective intervention (change in protocol). From an evaluation perspective, we need to compare estimates to a control condition that is often omitted from study designs, in addition to the need for benchmark estimates for key survey measures to provide estimates of nonresponse bias. We introduced a bias propensity approach for the selection of sample members to reduce nonresponse bias. Unlike a response propensity approach in which the objective is to maximize the prediction of nonresponse, this new approach deliberately excludes strong predictors of nonresponse that are uncorrelated with survey measures and uses covariates that are of substantive interest to the study. We also devised an analytic approach to simulate which sample members would have responded in a control condition. This study also provided a rare opportunity to estimate nonresponse bias, using rich sampling frame information, prior round survey data, and data from extensive nonresponse follow-up. The bias propensity model yielded reasonable fit despite the exclusion of the strongest predictors of nonresponse. The intervention was found to be effective in increasing participation among identified sample members. On average, the responsive and adaptive survey design reduced nonresponse bias by more than one-quarter—almost one percentage point—regardless of the choice of benchmark estimates. Effort under the control condition did not reduce nonresponse bias. While results are strongly encouraging, we argue for replication with varied populations and methods.

Non-response problem is an issue of great concern to all the researchers because it pervades almost all survey research. This ultimately creates potential for bias. The higher the non-response rate the greater the bias if the characteristic under study in respondents differs markedly from non-respondents. This paper reviews and explores further the nature and properties of non-response weighting adjustment approach as applied to sample surveys as a means of handling non-coverage and unit non-response bias in survey estimates. Consequently, a non-response weighting scheme is herein developed. Not only the attitude of the interviewees, but also sample and survey design, fieldwork approach and characteristics of organization influence response results. This study encompasses the application of stratified random sampling because it is well known that stratification method minimizes the variance of the estimates through the process of dividing the population into groups (strata) which are more or less homogeneous. Thus, stratification increases the precision of the estimate. The emphasis of this paper is practical rather than theoretical. Hence the discussion is illustrated with real examples from surveys (in particular 2003 KDHS) conducted by Central Bureau of Statistics (CBS) - Kenya. Some suggestions are made for improving the quality of non-response weighting. Keywords: Survey non-response; non-response adjustment factors; weighting; sampling frame; stratified sampling > East African Journal of Statistics Vol. 1 (2) 2006: pp. 143-162

PurposeThe aim of this paper is to suggest and demonstrate how anchoring vignettes, as a survey instrument, can be applied to study quality management (QM) across cultures. Cultural differences may create challenges in QM. Quantitative surveys are commonly used to study QM practices but do not consider the cultural bias in the survey results. An important question is how to study QM so that the results are comparable across cultures. Herein, the use of anchoring vignettes is suggested to reduce the problem of cross-cultural incomparability.Design/methodology/approachThis paper focuses on developing and testing vignettes for studying QM. Based on previous survey studies, two vignettes for each QM dimension are developed. The vignettes are then tested with two pilot tests on the web-based survey platform.FindingsThe paper provides a concrete example of how a survey using anchoring vignettes could be designed and used to deal with cultural bias in QM survey Based on hypothetical examples from Swedish and Indian cultures, the results from pilot studies evaluating vignettes are promising and show the applicability of the proposed method. Anchoring vignettes may help to provide more accurate survey results and thereby contribute to understanding of cross-cultural differences in QM practices. One of the challenges is the design of vignettes, which requires high precision and several tests to make the method work.Originality/valueThe paper discusses the potential of using anchoring vignettes to study QM practices across cultures. This may contribute to better understanding of QM practices in an international context, and thereby help improve QM practices in multinational organisations.

One objective of adaptive data collection is to secure a better balanced survey response. Methods exist for this purpose, including balancing with respect to selected auxiliary variables. Such variables are also used at the estimation stage for (calibrated) nonresponse weighting adjustment. Earlier research has shown that the use of auxiliary information at the estimation stage can reduce bias, perhaps considerably, but without eliminating it. The question is: would it have contributed further to bias reduction if, prior to estimation, that information had also been used in data collection, to secure a more balanced set of respondents? If the answer is yes, there is clear incentive, from the point of view of better accuracy in the estimates, to practice adaptive survey design, otherwise perhaps not. A key question is how the regression relationship between the survey variable and the auxiliary vector presents itself in the sample as opposed to the response. Strength in the relationship is helpful but is not the only consideration. The dilemma with nonresponse is one of inconsistent regression: a regression model appropriate for the sample often fails for the responding subset, because nonresponse is selective, non-random. In this article, we examine how nonresponse bias in survey estimates depends on regression inconsistency, both seen as functions of response imbalance. As a measure of bias we use the deviation of the calibration adjusted estimator from the unbiased estimate under full response. We study how the deviation and the regression inconsistency depend on the imbalance. We observe in empirical work that both can be reduced, to a degree, by efforts to reduce imbalance by an adaptive data collection.

Health survey statistics are essential for setting national targets, monitoring population-level trends, and tracking progress in adults’ utilization of preventive health services. However, survey estimates can be subject to nonresponse bias. This paper primarily assesses the potential for nonresponse bias in six utilization estimates from the Clinical Preventive Services Self-Administered Questionnaire (PSAQ) survey. The article highlights several findings. First, the eligible PSAQ sample person’s selective nonresponse associated with influenza vaccinations and blood pressure check-ups reported in the Medical Expenditure Panel Survey provides indirect evidence of overestimating the utilization of those services in the PSAQ survey. Second, the study found a few weighting class variables correlated with PSAQ response propensity and survey variables. Third, surprisingly, base-weighted estimates of preventive service utilization changed little despite multiple post-survey adjustments, suggesting no substantial nonresponse bias. Additionally, the study finds moderate to near-perfect concordance in responses to selected preventive service questions across interview modes.

Oral health surveys are undertaken to provide estimates of the dental health and behaviors of populations or population subgroups. However, the integrity of the data from sample surveys may be compromised by one or more sources of sampling and nonsampling error. An important source of nonsampling error is the failure to collect data from some of the individuals comprising the sample. Consequently, the response to a sample survey, and the direction and magnitude of bias induced by nonresponse, need to be taken into account when using estimates derived from sample surveys. Although the response rate to a survey is usually used as an indicator of the quality of the data it provides, nonresponse error is a function of nonresponse and the extent of differences in the characteristics of responders and nonresponders. Nonresponse may be managed in two ways. The first is to reduce nonresponse to a minimum using response-enhancement strategies. The second is the post-survey adjustment of data using weighting or imputation techniques to produce estimates that correct for nonresponse. This paper discusses issues concerning response and nonresponse bias in oral health surveys and provides guidelines on the management and reporting of nonresponse. It describes response-enhancement strategies to reduce noncontacts and refusals, sources of data to facilitate the comparison of responders and nonresponders, methods of assessing the degree of bias induced by nonresponse, techniques for producing adjusted survey estimates, and the assumptions on which these procedures and processes are based.

Social isolation, survey nonresponse, and nonresponse bias: An empirical evaluation using social network data within an organization

Respondent biases in agricultural household surveys

Survey designs for producing population prevalence estimates, such as abortion rates, need to consider multiple sources of error. Abortion prevalence has been found to suffer from underreporting. Abortion rates can also be underestimated due to unit nonresponse. However, it is when these two phenomena are linked that it becomes a particularly critical problem for both researchers interested in studying abortion rates and survey research practitioners. For substantive researchers, results may depend on the mix of these survey errors, in unexpected ways. For survey methodologists, decreasing one source of error may lead to greater bias in estimates by increasing the influence of another source. Identifying common causes can inform when to expect differences in rates due to the mix of nonresponse and measurement error, while helping practitioners design surveys that reduce both sources of error. This article addresses both nonresponse and measurement error in abortion estimates. We found that those with a lower likelihood to participate in the survey were also more likely to underreport such experiences. We interpret both nonresponse and measurement error as stemming from a common cause: the likely social stigma that reporting of these experiences poses to individuals. Although these results show that naive increasing of response rates may lead to greater bias in survey estimates, we also find limited evidence that some survey design changes can reduce the link between nonresponse and measurement error. This finding is worthy of replication for practitioners in order to break the link between these sources of error.

To analyse the costs and evaluate the equity, efficiency and feasibility of four strategies to identify poor households for premium exemptions in Ghana's National Health Insurance Scheme (NHIS): means testing (MT), proxy means testing (PMT), participatory wealth ranking (PWR) and geographic targeting (GT) in urban, rural and semi-urban settings in Ghana. We conducted the study in 145-147 households per setting with MT as our gold standard strategy. We estimated total costs that included costs of household surveys and cost of premiums paid to the poor, efficiency (cost per poor person identified), equity (number of true poor excluded) and the administrative feasibility of implementation. The cost of exempting one poor individual ranged from US$15.87 to US$95.44; exclusion of the poor ranged between 0% and 73%. MT was most efficient and equitable in rural and urban settings with low-poverty incidence; GT was efficient and equitable in the semi-urban setting with high-poverty incidence. PMT and PWR were less equitable and inefficient although feasible in some settings. We recommend MT as optimal strategy in low-poverty urban and rural settings and GT as optimal strategy in high-poverty semi-urban setting. The study is relevant to other social and developmental programmes that require identification and exemptions of the poor in low-income countries.

How confident can we be that the comparatively low response rates associated with mail surveys of groups of political activists, such as participants of a demonstration, does not hide a substantial nonresponse bias? The paper compares the results of a face-to-face survey of 2003 anti-Iraq war demonstrators in Glasgow, achieving a near perfect response rate, with the data derived from a mail survey handed out to demonstrators eliciting valid responses from 37% of marchers. The comparison shows that better educated, older, female demonstrators were more likely to return the mail questionnaire. Also demonstrators having born a higher ‘cost’ of travelling to the demonstration are more likely to respond. There was no evidence that political interest or political orientation played an important role. However, those who had taken part in demonstrations very frequently in recent years were less likely to return the mail questionnaire. While these results provide some reassurance that even with response rates below 40%, no substantive political bias is present, researchers undertaking surveys of activists should be alerted to the need to address possible nonresponse biases in a systematic way.

Earlier studies on turnout bias in postelection surveys have focused on vote overreporting (measurement bias) and have disregarded nonresponse bias associated with voter overrepresentation. Based on a unique data set of validated votes collected for two different ballots, we offer an in-depth analysis of the size and sources of nonresponse bias. On the aggregate level, we find that voter overrepresentation accounts for a larger share of total turnout bias than misreporting. On the individual level, past voting record, ballot vote, and age appear as strong predictors of survey participation. Breaking down further the analysis, we show that these factors contribute to nonresponse bias through both contact and cooperation. By contrast, the effects of some other factors work at cross-purpose in the contact and cooperation stages.

To investigate nonresponse bias in a two-phase epidemiologic study of eating-disordered behavior. Self-report questionnaires were delivered to a community sample of women aged 18-45 drawn from the electoral roll. Follow-up interviews were completed with a subgroup of respondents. Eating disorder psychopathology, general physical and mental health, and sociodemographic characteristics were compared among early (n = 259) and late (n = 71) respondents at the first phase of the study and among individuals with whom interviews were completed (n = 208) and individuals declining to be interviewed (n = 63) at the second phase. With respect to levels of eating disorder psychopathology, and on all other measures, individuals who responded at the first phase of the study only after repeated reminders did not differ from those who responded to the initial mailout, and individuals who declined to be interviewed did not differ from individuals with whom interviews were completed. Nonresponse bias among individuals declining to be interviewed is unlikely to pose a problem in two-phase epidemiologic studies of eating disorders. Further research is needed to examine the characteristics of nonrespondents at the first phase of such studies.

Examining non-response bias in substance use research—Are late respondents proxies for non-respondents?