Sensitivity Testing of Net Impact Estimates of Workforce Development Programs Using Administrative Data

Net Impact Estimates of the Workforce Development System in Washington State

The quantity of administrative data created, stored and processed in the world has grown exponentially over recent years, but the statistical theory to support its use in official statistics has not. That’s not to say that within the statistical community we are not using or investigating a wide variety of statistical sources, but the methods used are not based on well-established theory like we have for surveys. As a National Statistical Institute (NSI), we are transforming to put administrative and alternative data sources at the core of our statistics. Combining new sources with surveys will allow us to meet the ever-increasing user demand for improved and more detailed statistics. However, using administrative data involves addressing a range of statistical challenges as identified in “Statistical challenges of administrative and transaction data” (Hand, 2018).
 The Administrative Data Methods Research Programme was set up to address these challenges by developing a statistical framework for using and integrating administrative and transactional data to produce official statistics and analysis. The key themes of the programme include quality, linkage, privacy, estimation, statistical design, analysis and innovative techniques.
 The programme will ensure the rigor and assurance of methods developed, allowing where possible wide-reaching application, and to create and share best practice and guidance. A key part of this programme will be engaging and working collaboratively with the statistical community in academia, government and private sector to help prioritise the challenges, shape the work programme and drive the work forward.

This research measures earnings progression among participants in federally funded Workforce Investment Act ( WIA ) programs in the state of Washington during the period 2001 through 2008, using state administrative data and propensity score‐weighted regressions. Unlike previous evaluations that have emphasized earnings levels, this study addresses both earnings progression and levels to assess whether workers are on a path to reaching economic self‐sufficiency within a short time after participation. The analysis finds that participants in WIA Adult services had similar earnings progression as people receiving only less‐intensive Labor Exchange services.

The author examined cost-effectiveness of employment transportation (ET) services funded by the Job Access and Reverse Commute program of FTA in 2002. These services transport low-wage workers to jobs. The author presented multiple sources of primary and administrative data on users and programs in 23 ET sites. ET cost per ride measures were compared with cost per ride measures of transit services that were operating in the same area (called peer transit services). A cost per user per year was estimated to determine the level of transportation investments on the mobility of low-wage workers. On a per ride basis, ET services cost more to deliver than did peer transit services, but a wide variation existed in the level of investments and subsidies among different types of users, areas, and types of service. The median estimated cost per ET service user per year was similar to the annual investment per recipient by several social and human services and workforce development programs that supported low-income individuals to achieve improved economic outcomes. ET annual expenditures were the greatest for low-wage individuals who were traveling to work and less so for low-income individuals who were traveling to job training, to look for jobs, or to shop and make other discretionary trips. Implications are drawn for human services transportation coordination and for data and reporting requirements.

Unemployment can pose very serious public safety, public health, and mental health challenges to most, if not all, local jurisdictions throughout the United States. These problems clearly become heightened when a major recession or economic downturn plagues the country. The challenges associated with unemployment have been deemed so complex that the United Sates Department of Labor (DOL), through its Workforce Investment Act (WIA) program, has prescribed a highly integrated service delivery system which is designed to mitigate the challenges encountered by those individuals who become unemployed through no fault of their own. With their so-called One-Stop Centers, the DOL has theoretically positioned the public, private, and nonprofit sectors to work together in order to more effectively serve the unemployed. In this sense, the DOL’s One-Stop Center service delivery model specifically features the colocation of various workforce development and social services organizations at a single service site. The primary question addressed by this paper’s research concerns the extent to which the WIA program, through its One-Stop Service delivery system, has had an impact upon the plight of those who are unemployed. Accordingly, the study’s research questions are as follows: How does the WIA employment counseling intervention impact the successful implementation of the WIA program within the state of New Hampshire?

Under more and more fiscal scrutiny because of shrinking state and local budgets, workforce development programs are being asked to estimate their return on investment (ROI). This paper introduces basic concepts of ROI in workforce development programs. It distinguishes ROIs estimated for workforce programs from those that are estimated for financial investments or capital projects. The paper furthermore exposits the basic ingredients of an ROI study—identification of the treatment and time periods of analysis, identification of the net impacts of the program, and identification of net costs. Finally, the paper presents results from the estimation of the ROI for postsecondary career and technical education in the State of Washington.

SummaryWe compared measures of hospital performance by using both administrative and clinical data sources. Hospital-specific mortality outcomes on 10086 patients who had been admitted to 102 hospitals with a diagnosis of acute myocardial infarction in Ontario, Canada, were used as a test-case. Four and six hospitals were identified as having mortality that was statistically significantly higher than expected by using administrative and clinical data respectively, when model-based indirect standardization was used. When using random-effects models, zero and two hospitals were identified as having significantly higher mortality by using administrative and clinical data respectively. Approximately one in four hospitals changed at least two decile rankings when clinical data were used compared with when administrative data were used.

Integrated Data System Person Identification: Accuracy Requirements and Methods

Expert groups have recommended ongoing monitoring of the public health workforce to determine its ability to execute designated objectives. Resource- and time-intensive surveys have been a primary data source to monitor the workforce. We evaluated an administrative data source containing US Department of Health and Human Services (HHS) aggregate federal civil service workforce-related data to determine its potential as a workforce surveillance system for this component of the workforce. We accessed FedScope, a publicly available online database containing federal administrative civilian HHS personnel data. Using established guidelines for evaluating surveillance systems and identified workforce characteristics, we evaluated FedScope attributes for workforce surveillance purposes. We determined FedScope to be a simple, highly accepted, flexible, stable, and timely system to support analyses of federal civil service workforce-related data. Data can be easily accessed, analyzed, and monitored for changes across years and draw conclusions about the workforce. FedScope data can be used to calculate demographics (eg, sex, race or ethnicity, age group, and education level), employment characteristics (ie, supervisory status, work schedule, and appointment type), retirement projections, and characterize the federal workforce into standard occupational categories. This study indicates that an administrative data source containing HHS personnel data can function as a workforce surveillance system valuable to researchers, public health leaders, and decision makers interested in the federal civil service public health workforce. Using administrative data for workforce development is a model that can be applicable to federal and nonfederal public health agencies and ultimately support improvements in public health.

Measures of the gap in living standards, life expectancy, education, health and employment between Indigenous and non‐Indigenous Australians are primarily derived from administrative data sources. However, Indigenous identification in these data sources is affected by administrative practices, missing data, inconsistency, and error. As these factors have changed over time, assessing whether the gap between Indigenous and non‐Indigenous Australians has changed over time, based on data unadjusted for these sources of error can potentially lead to misguided conclusions. Combining administrative data on the same individuals collected from different sources provides a method by which a more consistent derived Indigenous status can be applied across all records for an individual within a linked data environment. We used the Western Australian Data Linkage system to produce derived Indigenous statuses for individuals using a range of algorithms. We found that these algorithms reduced the amount of missing data and improved within‐individual consistency. Based on these findings, we recommend our Multi‐Stage Median algorithm be used as the standard indicator of Indigenous status for any reporting based on administrative datasets when multiple datasets are available for linkage, and that algorithmic approaches also be considered for improving the quality of other demographic variables from administrative data sources.

This article describes two administrative data sources - UCAS applicant data and Higher Education Statistical Agency (HESA) data - and demonstrates how they can be utilised to monitor the progression of students from vocational and educational training (VET) programmes in to higher education (HE) in the UK. First the article provides a general discussion of the administrative data sets, their coverage and resulting limitations. Second, it illustrates how a combined HESA-UCAS data set can be utilised to provide a descriptive statistical analysis that demonstrates that including VET students within HE programmes widens participation for under-represented groups in the UK but shows that VET students are at a disadvantage compared with those progressing through academic pathways in terms of their probability of gaining access, which types of higher education institution (HEI) they gain access to, and an increased risk of dropping out during their first year of study. Third, multilevel modelling indicates that drop-out risks differ across HEIs and suggests that HEIs can take actions that will reduce the risk of VET students dropping out. Finally some conclusions about the utility of administrative data to answer substantive research questions are presented, with suggestions for improving the quality and access to such data for researchers.

Synthesizes what we know empirically about the WIA program and discusses lessons for WIA reauthorization.

Across the UK, the National Referral Mechanism (NRM) is the framework by which those affected, or those at risk of, modern slavery, are referred by ‘first-responders’ (such as the police) to other support services. For children, a key support mechanism is the Independent Child Trafficking Guardianship (ICTG) service, run by Barnardo’s for the Home Office. Administrative data from the NRM and ICTG were utilised in a novel mixed-method study to evaluate child-centred outcomes for children receiving ICTG support. We utilised publicly available NRM data to look at the nature of reported modern slavery between 2017 and 2022. We also negotiated access to Home Office held ICTG service data for the same period. Whilst our findings reveal patterns in the extent of reported exploitation and the nature of service support, examination of the two data sets revealed a number of issues related to categorisation of children, exploitation experiences, and geographical representation that hinder a comprehensive understanding of outcomes for child exploitation and trafficking. Qualitative ICTG data, however, revealed a ‘pyramid’ of support areas that were crucial to supporting positive outcomes, whilst participatory research methods revealed significant positive outcomes for children. The research identified a failure of administrative data to directly collect information relevant to positive child-centred outcomes. The recommendations, detailed within the final report, suggest that there is further work that could be done to improve administrative data collection, and highlight the potential of linked administrative data sources to understand related outcomes, such as health, education and crime/justice.

Background with rationaleChildren born to opioid-dependent mothers are at a developmental disadvantage from pre-birth. They are additionally affected by the mother’s compromised ability to recognise and respond to the infant’s cues. Development is often compounded by environmental factors. Research to date has primarily focused on early infancy and small, clinical samples. This group is difficult to follow-up using traditional methods due to chaotic home environments, housing instability and parent-child separation. The use of administrative data circumnavigates such difficulties, allowing follow-up of children over longer periods, even when removed from the birth parent.
 Main AimThis paper will describe the complex creation of a cohort of children born to opioid-dependent women, using administrative data. It will also describe early results about pregnancy and neonatal outcomes.
 Methods/ApproachData were pooled from women who gave birth between 2007 and 2017 using five datasets (c.5,000 women): women who were recorded as using heroin, street methadone or opioid substitution therapy (OST) on the Drugs Misuse Database, or on OST prescription records; women admitted to hospital, or psychiatric care, for an opioid related reason; and/or women whose children were recorded as having Neonatal Abstinence Syndrome (NAS). Data on children’s neonatal outcomes will be described, including birth weight and gestation, congenital abnormalities, neonatal death and NAS treatment. Models will be fitted to investigate the associations between possible teratogenicity of prenatal opioids and developmental outcomes.
 ResultsThe development of this cohort using administrative data sources has been complex, requiring five different datasets to ensure all women of interest are captured. Descriptive results on outcomes will be available in the Autumn.
 ConclusionThis administrative data study demonstrates the value of using linked data sources to enhance our knowledge of the trajectories of this vulnerable group of children, and the additional support that they, and their carers, may require.

Given the growing popularity of administrative data for health research, information on the differences and similarities between administrative data and customary data sources (e.g. surveillance) will help to inform the use of administrative data in the field of sexually transmissible infections (STIs). The objective of this study was to compare the incidence rates of three nonviral STIs from a large health insurance administrative database (MarketScan) with surveillance data. We computed and compared STI rates for 2005-10 from MarketScan and national surveillance data for three major nonviral STIs (i.e. chlamydia (Chlamydia trachomatis), gonorrhoea (Neisseria gonorrhoeae) and syphilis (Treponema pallidum)). For administrative data, we assessed the sensitivity of the rates to enrollee inclusion criteria: continuous (≥320 member-days) versus all enrollees. Relative rates were computed for 5-year age groups and by gender. The administrative database rates were significantly lower (P<0.01) than those in the national surveillance data, except for syphilis in females. Gonorrhoea and syphilis rates based on administrative data were significantly lower (P<0.01) for all enrollees versus continuous enrollees only. The relative STI rates by age group from the administrative data were similar to those in the surveillance data. Although absolute STI rates in administrative data were lower than in the surveillance data, relative STI rates from administrative data were consistent with national surveillance data. For gonorrhoea and syphilis, the estimated rates from administrative data were sensitive to the enrollee inclusion criteria. Future studies should examine the potential for administrative data to complement surveillance data.