Machine bias

"Machine bias" is the headline phrase from a ProPublica investigation published 23 May 2016 exposing racial disparities in COMPAS, a proprietary algorithmic risk-assessment tool used by U.S. criminal courts to predict defendants' likelihood of future criminality. Within months of publication, the phrase entered civil-society, legal, and academic discourse as the dominant shorthand for racial discrimination encoded in algorithmic decision-making — a framing that directly contested the neutrality claims used to justify the deployment of predictive scoring tools at bail hearings, sentencing, and parole decisions. The argument at the framing's center: that a tool's technical accuracy is distinct from its racial fairness, and that the choice between competing definitions of fairness is a normative judgment about whose harms matter most, not a technical optimization.

Origin

The ProPublica team — Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner — filed FOIA requests in Broward County, Florida, a large jurisdiction using COMPAS in pretrial decisions whose open-records laws made the underlying data accessible. They obtained COMPAS scores for more than 7,000 people arrested in 2013 and 2014 and matched those scores against two-year follow-up arrest records. COMPAS — Correctional Offender Management Profiling for Alternative Sanctions, developed by Northpointe (later Equivant) — generates a 1–10 recidivism risk score from a 137-item questionnaire covering criminal history, social environment, and self-reported attitudes; judges use the score as a supplementary input at bail, sentencing, and parole hearings.

The findings reported on 23 May 2016: Black defendants who did not reoffend were nearly twice as likely to be falsely labeled high-risk as white defendants (45 percent vs. 23 percent); white defendants who did reoffend were nearly twice as likely to be incorrectly labeled low-risk (48 percent vs. 25 percent). The algorithm's overall prediction for violent recidivism was wrong 80 percent of the time. Northpointe responded formally, defending COMPAS as "equally accurate" across racial groups in a calibration sense — its scores had equal predictive validity within each group. That defense was technically correct, and the dispute between the two claims named a structural tension the article had surfaced but not resolved.

The core argument

The framing's lasting contribution was to show that the gap between technical accuracy and racial fairness was not an oversight but an inherent property of the system — and that the choice between competing fairness definitions could not be avoided. Researchers Alexandra Chouldechova and Jon Kleinberg, Sendhil Mullainathan, and Astrid Luca Smola independently proved, within months of publication, that error-rate equality (the civil rights standard ProPublica applied) and calibration equality (Northpointe's defense) cannot simultaneously hold across groups with differing base rates except in degenerate cases. This impossibility result formalized what the article had surfaced empirically: deploying any risk-assessment tool in a context with racially unequal base rates requires a normative choice about whose error rate to equalize. In the criminal justice context, base rates are themselves a product of racially discriminatory policing and prosecution history — so a model that faithfully captures historical patterns faithfully encodes their racial disparities into predictions that courts then treat as objective.

The framing is distinct from adjacent algorithmic-bias framings. It is not about technical accuracy alone (the model may be accurate), not about data quality alone (the data may reflect real patterns), and not about design intent (no deliberate bias is required). The argument is that structural racial disparities in the criminal justice system are the input to COMPAS, so a faithful model encodes those disparities and reproduces them as algorithmic outputs that carry the authority of scientific prediction. "Machine bias" names that feedback loop: the machine mirrors the historical bias it was trained on.

Propagation

The article drew rapid academic response. Within months of publication, the impossibility proofs by Chouldechova and by Kleinberg, Mullainathan, and Raghavan had emerged directly from the COMPAS debate, spawning a body of work on algorithmic fairness that continues to grow. The phrase "machine bias" appears as a term of art in titles and abstracts across computer science, law, sociology, and public policy. The publication of the dataset and analytical code set a new template for open, replicable data journalism on algorithmic systems — subsequent audits, including the Algorithmic Justice League's Gender Shades study, drew on the approach.

In the courts, the Wisconsin case of State v. Loomis became the first significant legal test of algorithmic sentencing. Eric Loomis challenged his six-year sentence on due-process grounds, arguing that COMPAS's proprietary status denied him the ability to contest the scientific validity of the score. The Wisconsin Supreme Court upheld COMPAS use at sentencing in July 2016 but ruled that courts could not use the score as a determinative sentencing factor, and mandated that presentencing reports carrying COMPAS scores include a five-part warning about the algorithm's limited utility. The U.S. Supreme Court declined to hear the appeal in June 2017, leaving the Wisconsin ruling as the leading legal precedent on algorithmic sentencing.

In civil-society advocacy, the Electronic Frontier Foundation adopted the machine-bias framing in sustained work against pretrial risk assessment tools, finding that such tools do not create fair pretrial systems. Privacy International catalogued the ProPublica findings as a canonical international reference case for algorithmic bias in criminal justice. The ProPublica "Machine Bias" series expanded beyond COMPAS to cover Amazon's pricing algorithm, Facebook's advertising targeting, and other domains — extending the framing from criminal justice to algorithmic decision-making across commercial platforms. The project was a 2017 Pulitzer Prize finalist for Explanatory Reporting.

Why it has carried

Three features explain the framing's persistence. First, the phrase works across every level of the discourse. "Machine bias" fits a newspaper headline, a congressional hearing testimony, a placard, a law review article title, and an algorithm-auditing methodology description without requiring translation between registers. Adjacent framings — the coded gaze, data is a civil rights issue — name the same underlying phenomenon but in domain-specific registers (optical/surveillance; civil rights law); "machine bias" is register-neutral and extensible, as the ProPublica series' expansion beyond criminal justice demonstrated.

Second, the open methodology made the findings durable. Publishing the dataset and code meant independent verification was possible, which removed the standard industry defense of proprietary opacity. It also enabled the impossibility proofs to emerge from the same data — researchers could engage with the argument concretely rather than abstractly. The open approach did not prevent challenge (Northpointe published a formal rejoinder; academic papers disputed aspects of the analysis) but it ensured the challenge was substantive.

Third, the impossibility result amplified rather than neutralized the framing. Proving that the choice of fairness definition could not be avoided — that deploying any risk-assessment tool required a normative judgment about whose harms matter more — relocated the question from the domain of technical expertise to the domain of democratic accountability. The framing's political argument (that algorithmic tools used to decide people's freedom encode value choices that deserve democratic scrutiny) was strengthened, not weakened, by the mathematical proof that the choice was genuine. The impossibility result became itself a movement resource: it gave advocates a formal basis for arguing that "better algorithms" alone cannot resolve the civil rights stakes, and that algorithmic sentencing tools require democratic governance, not merely technical refinement.