“The real danger is not that computers will begin to think like men, but that men will begin to think like computers.”
– Sydney J. Harris
What should a police officer do when the file arrives already carrying a tidy score, a match presented as mathematics that cannot be meaningfully contested? This is the moment when data stops being background and starts structuring suspicion itself.
Somewhere upstream, a pattern has been found, a person flagged, a risk ranked. By the time the case reaches the courtroom, the system has already shaped what officers noticed, what the prosecution foregrounds, and what the defense is forced to explain. A “technical” identification can therefore slide quickly into a coercive sequence of stops, arrests, and sanctions – a dynamic the Strasbourg Court already confronted in public-space facial recognition in Glukhin v Russia.
In criminal justice, that quiet work matters more than almost anywhere else, because the state here decides when to detain, charge and punish. It also decides what kind of subject you are – a citizen to be heard, or a data point to be processed. Small shifts in how suspicion is formed travel through the process, and they compound.
The sharp edge of the automated state rarely appears as a single machine decision with dramatic consequences. More often, it is the slow normalization of outputs that feel objective – a risk category becomes a baseline for reasonableness, a face match a shortcut around doubt, a tool’s recommendation slips into the logic of justification. Even when a judge remains the final decision-maker, the record begins to speak in the system’s language. Fairness thins out, often without announcement.
This post traces that pattern across three pressure points. It begins upstream, where predictive policing turns past records into future patrols; moves to the courtroom, where risk scores and opaque tools strain the right to contest evidence; and asks whether judge-in-the-loop is the safeguard it claims to be. It closes with a “minimum humanity package“, that is, a baseline of conditions that must hold if criminal justice is to stay recognizably human.
From data to suspicion
Predictive policing is often described as a neutral technique for allocating scarce capacity. The mechanics can be more troubling, because models learn from “discovered crime” data such as stops and arrests, and then use that learning to decide where police should go next. As Ensign et al. (2017) explain, that loop can become self-reinforcing. Officers are repeatedly sent back to the same areas regardless of the underlying crime rate, because the system keeps feeding on the data created by its own deployments. Reported incidents can dampen the effect, but they do not remove it.
Data-heavy policing practices are social, institutional, and incentive-driven, and the resulting “objectivity” often functions as a form of cover that makes the practice harder to challenge. Brayne (2021) shows how private platforms, proprietary analytics, and procurement logic can turn everyday policing into a regime of stratified surveillance that resists scrutiny. Ferguson (2017) notes that big data changes the suspicion calculus without changing what the person on the street has actually done.
The criminal file carries these distortions before any judge sees it. Suspicion becomes a product of infrastructure, and the question shifts from what happened to what the system surfaced.
The AI Act draws a line, but a narrow one. Article 5(1)(d) prohibits AI systems that predict an individual’s risk of committing a crime based solely on profiling or personality traits, yet exempts those that support a human assessor relying on objective, verifiable facts already linked to criminal activity. Predictive policing therefore escapes the prohibition; only pure individual prediction is caught.
From suspicion to score
Once a case reaches the courtroom, risk scores often arrive as if they were ordinary information. As Mayson (2019) puts it, prediction functions like a mirror that projects the past forward under status quo conditions. When the state turns that projection into a reason to detain or punish, old inequalities stop being history and become a managed future. The ProPublica COMPAS investigation documents how risk assessments are used at bail, sentencing, and parole, and treated as a proxy for future behavior. In its Broward County analysis, ProPublica found that predictions about violent recidivism were notably unreliable, while error patterns differed sharply across racial groups.
The deeper problem is institutional, because a score can quietly reset the reference point for a judge’s discretion. The defense then litigates against a number that looks objective on a surface, while its inputs, assumptions, and tolerable error rate stay offstage. The CEPEJ Ethical Charter on AI in judicial systems takes a cautious approach on exactly these points, putting non-discrimination, transparency, and user control at the center. That posture fits the reality of criminal adjudication, where the cost of a false positive is measured in days in jail and years in prison, not in a minor administrative inconvenience.
European law has already confronted automated pre-screening. Although the Court of Justice’s PNR rulings arose in an administrative-data context, the architecture they impose (a tightly framed purpose, targeted criteria, bans on sensitive grounds, and a meaningful human check before any “hit” triggers action) is even more important in predictive policing and risk assessment, where the consequence is interference with liberty.
The black-box in the courtroom
A criminal process lives or dies by contestability and automation strains that premise in several ways that often show up together, e.g., through a legal black-box (forms when the record offers conclusions without the reasoning chain that produced them) and a technical black-box (forms when the method is unreadable to the parties). Liu et al. (2019) treats Loomis as a warning about “algorithmization” of government functions, where opacity and delegation of authority can undermine due process and transparent justification.
Trade secret claims make the problem sharper. Wexler (2018) argues that developers have increasingly invoked intellectual property to avoid disclosure in criminal cases, even when the tool’s output is central to proving guilt. A privilege that blocks meaningful defense access overprotects secrecy at the expense of liberty. Citron (2008) makes a related point in her account of technological due process, that is, the automated outputs can devalue hearings when parties lack meaningful notice and decision makers presume the system’s infallibility.
Courts have begun to respond with a demand for fit, not faith. In United States v Ortiz, the court excluded DNA evidence generated by STRmix because the software had not been properly validated for the kind of complex mixture at issue. That logic travels well beyond DNA, because a method should be shown reliable for the particular task, and the defense should have a real chance to probe its limits.
Why judge-in-the-loop is not enough
Human oversight is usually the default answer in policy documents, but the question that remains is who carries it in practice, and with what authority and support. Banks (2026) argues that expecting judges who use high-risk decision-support systems to also oversee them is too much: real oversight requires training, time, technical access, and the institutional power to stop or redesign a system. The AI Act governance architecture points to supervision shared across bodies with expert capacity and enforcement powers. The CEPEJ Charter similarly centers transparency and the possibility of external audit, while insisting that tools remain under user control.
When oversight becomes no one’s specific job, the system’s outputs gradually inherit the authority that once belonged to reasoned judgment, and the person before the court is left arguing against a process that has already forgotten it ever needed to explain itself. The real danger there is not that computers will begin to think like judges, but that judges will begin to think like computers.
A minimum humanity package
A workable baseline requires conditions that keep coercive power answerable in court. When analytics steer patrols or prioritize targets, the sources, criteria, and scope have to be knowable enough for feedback loops to be spotted early and corrected before they harden into routine. If a score, a match, or a model shaped a step in the case, the parties should be told, and the court should be able to see the tool’s purpose, its validated scope, and the point where its reliability ends. Evidence that cannot survive that kind of scrutiny should not be treated as evidence, no matter how clean the output looks, and commercial claims cannot justify a process where the defense is asked to answer a machine without access to the pathway that produced the result. Judges remain responsible for the decision, yet the institution needs a backstop that can monitor deployments, demand documentation, and intervene when a system drifts beyond its lawful purpose.
That is the line between a state that uses machines and a state that has become one – and crossing it quietly, without announcement, is precisely how criminal justice stops being something that can be done to a person and starts being something that simply happens to them.
Janko Munjić
Janko Munjić is a senior judicial assistant at the Appellate Court in Kragujevac and a PhD candidate in criminal law at the Faculty of Law, University of Kragujevac. His research focuses on criminal-law responsibility and digital evidence in AI-mediated contexts, with particular attention to autonomous systems and human-machine interaction. He was awarded the We Robot 2026 Best Paper Award.
