Guilt by Algorithm

I. The question of the week

For three years the academic integrity office had a machine that could read a student’s paragraph and return a number — a percentage, a probability, a verdict dressed as a measurement. The number was called an AI-detection score, and for a while it was treated the way a breathalyzer reading is treated: as the kind of evidence that ends an argument. This week’s premise is that the machine is being switched off. Universities are disabling the detectors, quietly retiring the dashboards, instructing faculty to stop pasting suspect essays into tools that were never validated for the job. What was sold as enforcement is being abandoned as enforcement.

The temptation is to read this as a story about the classroom, a discrete drama of Turnitin and honor codes and anxious sophomores. It is not. The classroom is the last room to learn a lesson that was taught, expensively and cruelly, in welfare offices and police precincts years earlier: that an automated system trained to detect wrongdoing will confuse a statistical signature with a confession, and that institutions will act on the confusion until enough innocent people are punished to force a reckoning.

So the arc this column traces runs in two registers at once. There is what we have been saying about detection-as-enforcement — a rhetoric that began in warning, swung hard toward confidence in early 2025, and then cracked. And there is what has actually been happening, which never swung at all, because the failure mode was documented before the confidence arrived. The interesting distance is between the two. The people writing most optimistically about detection in 2025 were describing a machine whose defining catastrophe was already a matter of public record. This is the story of how long it took for the record to catch up with the talk.

II. What we’ve been saying

The dominant way of talking about automated detection has been the language of the catch — the confident fantasy that a system can perceive the guilt a human observer would miss, and can do it at scale, cheaply, without fatigue or favoritism. But the confidence is recent. As late as the winter of 2024 the loudest voices in the conversation were sounding alarms. When the American Civil Liberties Union released a six-page white paper that December, the coverage framed the technology as a hazard first and a tool second: Rising use of generative AI by police is a threat to Americans’ civil liberties, ACLU warns treated automated report-writing as something to be contained, and the organization’s own analysis, AI Generated Police Reports Raise Concerns Around Transparency, Bias, argued that the tools threatened “to exacerbate existing problems in law enforcement, and to create new ones.” The register was defensive. Detection was a thing happening to people.

Then, over the first quarter of 2025, the register flipped. The warnings did not disappear so much as get outvoted. Detection was reframed as rescue and, crucially, as a solvable engineering problem. Can deep learning rescue victims through recognition and prevention? recast pattern-recognition as a humanitarian instrument, one that could “promote more timely interventions” against traffickers. The bias problem, which had anchored the earlier skepticism, was absorbed into the optimistic frame as a to-do item: AI Discrimination and the 10-Step Bias Elimination Audit proposed that the discriminatory outputs of automated systems could be audited away in ten legible steps, a checklist standing in for a cure. Alongside it, Revolutionizing Cybersecurity: AI and Machine Learning in Network Security narrated detection as pure upside, “reshaping how organizations tackle modern cyber threats.”

The tell, across this optimistic turn, was grammatical. The more confident the writing became, the more it treated the machine’s output as evidence rather than as suggestion. Even a document whose title screamed the opposite — INHUMAN REASON: Predictive Policing Algorithms and the Fourth Amendment — was coded, in the trade-journal register of the moment, as a neutral-to-favorable examination of a maturing tool rather than an indictment. The question had shifted from should the machine accuse to how do we tune the machine’s accusations.

By the middle of 2025 the language of solvability hardened into the language of inevitability. AI in Mass Surveillance: Big Data is Watching You assessed that automated surveillance “is rapidly expanding across the globe” and that “regulators still have to match this trend” — expansion presented as weather, and oversight as the thing running late. AI-Powered Criminal Identification in India: Evaluating Human Rights Concerns in Automated Identification Systems opened by calling automated identification “a transformative shift in law enforcement, promising enhanced efficiency, accuracy, and predictive capabilities” before it reached its caveats. The frenzy had its own momentum; AI Personas May Be Undercover Police, Engaging With Suspects Online Through Social Media And Text reported the deployment of synthetic detectives without ever quite pausing on the question of what such personas prove. And the numbers underwrote the mood: The AI Revolution–Navigating and Safeguarding Justice in the Digital Age cited a McKinsey finding that ninety-two percent of companies planned to increase AI investment over three years — a figure that functions in these pieces less as data than as permission.

We had said as much in our own pages. An earlier essay in our AI-literacy series, the AI News Social-Weekly Critical Analysis-AI Literacy-EN-20250217 briefing of February 2025, noted how consistently the literature framed automated tools as instruments for enhancing critical thinking, as though the software and the scrutiny pointed the same direction. In the enforcement literature they rarely did. The scrutiny was what the software was meant to replace.

Then, over the third quarter, the confidence cracked. It cracked from the inside, in the vocabulary of the proponents themselves. The role and place of artificial intelligence in modern society: from terminological uncertainty to the legal regulation of its use in law enforcement activities is, on its face, a case for adoption — but it spends its length cataloguing “key limitations and risks arising from the use of algorithmic systems.” ImpACT International | AI and Surveillance Are Reshaping Global Human Rights Protections Rapidly let the human-rights frame back into the center of a sentence that a year earlier would have led with efficiency. This is the inversion point the arc registers: not a reversal into opposition, but the return of doubt to the mouths of the enthusiasts. Regret entered the standard vocabulary. What had been the ACLU’s warning in December 2024 was, by August 2025, the tone of the field.

III. What’s been happening

While the rhetoric swung, the record sat still, because the definitive failure of detection-as-enforcement had already been written down, in detail, before the optimistic turn ever began. Kate Crawford’s The Atlas of AI (2021) records it plainly: the Michigan Integrated Data Automated System, “a system built to ‘roboadjudicate’ and punish those it determined to be defrauding the state’s unemployment insurance.” The system, she writes, “was designed to treat almost any data discrepancies or inconsistencies in an individual’s record as potential evidence of illegal conduct,” and it “inaccurately identified more than forty thousand Michigan residents of suspected fraud.” Forty thousand. This is not a cautionary parable. It is a completed experiment, and it returned an unambiguous result: an automated adjudicator that treats an anomaly as an accusation will manufacture guilt at scale.

The reason is structural, not incidental, and Crawford names it in the same book. AI’s achievements, she writes, have depended on “boiling things down to a terse set of formalisms based on proxies: identifying and naming some features while ignoring or obscuring countless others.” A detection score is a proxy. The AI-writing detector does not read a student’s mind; it measures the statistical smoothness of prose against a model of what a language machine tends to produce, and flags the essays that look too clean. The predictive-policing model does not observe a crime; it observes a correlation. In every case the machine names the features it can see and discards the ones it cannot, and then the institution mistakes the visible proxy for the invisible truth. A well-edited paragraph and a plagiarized one can carry the same signature.

The failure has a second structural source, which is that the systems are fragile in ways their operators do not anticipate. As You Look Like a Thing and I Love You (2019) observes, adversarial manipulation is not exotic — “Even humans are susceptible to the Wile E. Coyote style of adversarial attack: putting up a fake stop sign, for example, or drawing a fake tunnel on a solid rock wall. It’s just that machine learning algorithms can be fooled by adversarial attacks that humans would never even register.” A detector that can be defeated by a determined cheater running the text through a paraphraser, while simultaneously flagging a non-native speaker who writes in careful, formulaic English, is not an instrument of justice. It is a machine that catches the honest and misses the guilty, which is the precise inversion of what enforcement is for.

The third failure is opacity, and it is the one that turns a bad tool into a due-process crisis. The ACLU’s objection to AI Generated Police Reports Raise Concerns Around Transparency, Bias was never merely that the machine erred; it was that a defendant cannot cross-examine a model. When the evidence against you is a probability generated by a proprietary system whose workings are a trade secret, the accusation cannot be tested, and an accusation that cannot be tested is not evidence at all. This is what INHUMAN REASON: Predictive Policing Algorithms and the Fourth Amendment was circling, and what AI-Powered Criminal Identification in India: Evaluating Human Rights Concerns in Automated Identification Systems reached when it moved past the promise of accuracy to the human-rights cost of getting it wrong.

And beneath all three sits the fact that should have stopped the enterprise before it started: the tools were deployed without proof that they worked. Stanford’s HAI_AI-Index-Report-2024 documents, across the industry, “the lack of standardized responsible AI benchmark reporting” — meaning there was no agreed, external, validated measure of how accurate these systems were. Detection was sold on the vendor’s word. An academic-integrity office that suspended a student on a detector’s say-so was relying on a number for which no benchmark existed, in exactly the way MiDAS relied on discrepancies that turned out to mean nothing. The same report catalogues how cheap manufactured plausibility has become: a complete, “authentic-appearing misinformation system,” it notes, could be assembled “for around $400.” If a few hundred dollars buys convincing fake engagement, the premise that a detector can reliably sort the machine-made from the human-made was already collapsing on the generation side even as institutions leaned harder on it for enforcement.

So the reality across these quarters is not a story of a promising technology hitting unexpected snags. It is a story of a known failure being re-run in domain after domain — welfare, then policing, then identification, then the classroom — by institutions that had access to the Michigan result the entire time. The regulatory catch-up that Forging Clarity: A Framework for Navigating AI Regulation described as a “perplexing regulatory patchwork” is what it looks like when the law arrives after the harm and tries to build the guardrail on top of the wreck.

IV. Where they meet, where they miss

They miss on the question of novelty. The rhetoric treats each domain as a frontier — the classroom’s AI-detection debate proceeds as though it were the first institution ever to point an automated adjudicator at a human being and act on the output. It is among the last. The reality is a single continuous pattern with a single recurring failure: proxies mistaken for proof, opacity foreclosing appeal, and bias — the discriminatory misfire that AI Discrimination and the 10-Step Bias Elimination Audit promised to checklist away — turning out to be a property of the method rather than a bug in the settings. The classroom detector is MiDAS in miniature, pointed at a paragraph instead of an unemployment claim, and the false positive it produces costs a student a transcript notation instead of a benefits clawback, which is a difference of scale and not of kind.

They meet, belatedly, on the recognition that the collapse of detection-as-enforcement is not a retreat from artificial intelligence. This is the crucial thing for a reader to carry, and it is where this column takes a side. The universities switching off their detectors are not Luddites and they are not surrendering to cheaters. They are rediscovering due process — the old, unglamorous principle that punishment requires evidence a person can confront, and that a secret score generated by an unvalidated proprietary model is not that. The same rediscovery is what the ACLU was demanding of police departments in December 2024, and what the human-rights literature was demanding of India’s identification systems by the middle of 2025. Detection is collapsing as enforcement precisely because enforcement is the one use that requires the thing detection cannot supply: proof.

And here the skepticism should point at the vendors, not the institutions. Someone sold these tools. Someone attached a confidence percentage to an output that HAI_AI-Index-Report-2024 confirms had no standardized benchmark behind it, and someone in a faculty-development office ran the workshop that taught professors to trust the number. The mystification was the product. Strip it away and the claim being made was always modest to the point of uselessness — this text has features statistically associated with machine generation — a sentence that cannot support a suspension, a failing grade, or an expulsion. The optimistic quarter of 2025 was, in this light, not a period of genuine progress but a period during which the industry’s marketing outran its evidence, and the correction of Q3 was the evidence catching up. The conversation did not discover a new truth. It stopped ignoring an old one.

V. The longer view

What is ending is not the use of these systems but a specific and dangerous grammar around them — the grammar that let a probability masquerade as a verdict. The systems will remain, repurposed downward toward what they can honestly do: flag a paper for a human to read more carefully, surface an anomaly for an investigator to examine, raise a question rather than answer one. That is a smaller job than enforcement, and a truthful one. The institutions now learning this in their classrooms are arriving, late and at the expense of falsely accused students, at the place Crawford’s forty thousand Michiganders reached the hard way — the place where you learn that a system built to boil a person down to a proxy will, sooner or later, boil down the wrong person. A machine can raise a suspicion; it cannot carry a verdict, and every institution that forgot the difference has been quietly walking it back. The detectors were never lie detectors. They were pattern detectors that we agreed, for a few confident quarters, to treat as oracles — and the collapse of that agreement is the healthiest thing to happen to academic integrity in years.

Guilt by Algorithm

I. The question of the week

II. What we’ve been saying

III. What’s been happening

IV. Where they meet, where they miss

V. The longer view

References