• Home
  • Search
  • Auditing the Audits: Lessons for Algorithmic Accountability from Local Law 144's Bias Audits
  • Cite Icon2
  • https://doi.org/10.1145/3715275.3732004Copy DOI Icon

Auditing the Audits: Lessons for Algorithmic Accountability from Local Law 144's Bias Audits

  • Jun 23, 2025
  • Marissa Kumar Gerchick +5 more
Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

In this work, we "audit the audits," analyzing the documents produced pursuant to one of the United States' first enacted laws regulating the use of artificial intelligence in employment: New York City's Local Law 144.This law requires employers and employment agencies using certain types of automated tools to publish "bias audits" with statistics about how different sex and racial groups fare in the hiring process when the tools are used.We collect and conduct a comprehensive analysis of all Local Law 144 bias audits (N=116) made publicly available to our knowledge from the law taking effect in July 2023 until early November 2024, and describe the extensive challenges we faced in identifying, archiving, extracting information from, and ultimately analyzing these bias audits.We identify several ways that bias audits produced in accordance with Local Law 144 are incomplete evaluations of algorithmic bias, despite news coverage and characterizations by employers and vendors suggesting otherwise.We show that Local Law 144 bias audits are significantly hampered by several issues, including missing demographic data, opaque data aggregation, problematic uses of "test data, " and reliance on metrics that do not represent how automated hiring tools are used in practice.We analyze the reported results in Local Law 144 bias audits alongside the four-fifths rule often used as a measure for assessing adverse impact in employment contexts.Most audits do not report results that would not suggest violations of the four-fifths rule.Crucially, however, we show that these tools could often be in violation of the four-fifths rule when considering potential impacts of missing demographic data.We offer ten practical recommendations to strengthen future legislative efforts that mandate algorithm auditing in hiring and other areas, and contribute an open dataset and codebase for extracting and combining bias audit results to support future auditing efforts.

Similar Papers
  • PDF
  • Research Article
  • Citations10

Investigation of the Disparity Between New York City and National Prevalence of Nonspecific Psychological Distress Among Hispanics

  • Feb 09, 2012
  • Preventing Chronic Disease
  • Katharine H Mcveigh +1
  • Book Chapter

Manhattan is not New York: The Divergent Import of the Great Recession and Natural Disasters on New York City’s Five Boroughs

  • Mar 01, 2016
  • Carolyn E Predmore +1
  • Research Article

Foodscape gastropolis New York City

  • Aug 12, 2016
  • Arnold Van Der Valk
  • Research Article
  • Citations2

Immigration to New York City in the Post–1965 Era: Sex Selectivity and the Economic Role of Women

  • Jan 01, 1994
  • Center for Migration Studies special issues
  • Joseph J Salvo
  • Research Article
  • Citations2

A “Machine That Makes the Land Pay”: The West Street Building in New York

  • Apr 01, 1989
  • Technology and Culture
  • Sharon Irish
  • Research Article

Program Design, Incentives, and Response: Evidence from Educational Interventions

  • Jan 01, 2010
  • SSRN Electronic Journal
  • Rajashri Chakrabarti
  • Research Article
  • Citations6

Project 2011 and the Preparation of Black and Latino Students for Admission to Specialized High Schools in New York City

  • Jan 01, 2012
  • The Journal of Negro Education
  • Mercedes E Ebanks +3
  • Research Article
  • Citations18

The 2015 New York City Legionnaires' Disease Outbreak: A Case Study on a History-Making Outbreak.

  • May 25, 2017
  • Journal of public health management and practice : JPHMP
  • Allison T Chamberlain +2
  • Research Article

Abstract PO-054: Hepatitis B and C screening and knowledge of liver cancer in underserved Chinese American adults in New York City: A community-based assessment

  • Nov 30, 2020
  • Cancer Epidemiology, Biomarkers & Prevention
  • Wenyue Lu +11
  • Research Article
  • Citations2

Review: The Greatest Grid: The Master Plan of Manhattan 1811–2011

  • Mar 01, 2013
  • Journal of the Society of Architectural Historians
  • Zeynep Turan Hoffman
  • Research Article
  • Citations2

Motion Pictures: A Problem to Be Co-operatively Solved

  • Jan 01, 2017
  • Film History
  • Martin L Johnson
  • Research Article
  • Citations2

Poverty in Eighteenth-Century New York City: A Jewish Pauper Demands Kosher Meals

  • Jan 01, 2014
  • New York History
  • F M Loewenberg
  • Research Article

Unpacking Police and Policing in New York City Through the Lens of Caribbean Immigrants

  • Nov 29, 2024
  • American Behavioral Scientist
  • Wendell C Wallace
  • Research Article
  • Citations6

Contraceptive Use and Sterilization Among Puerto Rican Women

  • Sep 01, 1992
  • Family Planning Perspectives
  • Joseph J Salvo +2
  • Front Matter
  • Citations7

Tuberculosis: A Global Problem at Our Doorstep

  • Dec 01, 2004
  • Chest
  • E Neil Schachter
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.