You can read the full paper here.
As generative large language models (LLMs) like GPT demonstrate unprecedented capabilities in legal analysis and reasoning, their potential deployment in courts raises critical questions about accountability and preserving the rule of law. This recently published paper examines this technological frontier through the lens of the EU AI Act’s regulatory framework, revealing both opportunities and potentially overlooked risks for the rule of law.
Drawing insights from the (in)famous COMPAS case, where an AI system used for predicting recidivism risk created an accountability gap in criminal sentencing, the analysis shows how similar challenges persist – and may even intensify – with more sophisticated generative AI. Despite their impressive ability to analyze legal texts and generate human-like responses, these systems can produce “hallucinations”, i.e. plausible but factually incorrect content, that may unduly influence judicial reasoning.
Particularly concerning is how the anthropomorphic nature of generative LLMs can amplify the accountability gap. These systems’ ability to engage in human-like dialogue and generate sophisticated legal reasoning creates an illusion of competence that can mask their fundamental limitations. While they can craft persuasive legal arguments and analyze complex cases, they remain pattern-matching systems that can “hallucinate” convincing but false legal precedents or statutory provisions. The paper demonstrates how this combination of apparent authority and hidden fallibility is especially dangerous in judicial settings. When judges perceive these systems as intelligent legal assistants rather than sophisticated text generators, they may place unwarranted trust in their outputs, effectively delegating judicial authority to systems incapable of genuine legal reasoning or moral judgment. This anthropomorphism widens the accountability gap and potentially undermines the foundation of judicial decision-making.
Against this background, the paper examines how the EU AI Act’s risk-based approach could help prevent the erosion of judicial accountability. It reveals that the mere presence of a “human in the loop” is insufficient protection against automation bias, i.e. the tendency to over-rely on AI recommendations. Instead, (meaningful) human oversight – recalling the GDPR’s Article 22 – requires a comprehensive framework of checks and balances across the entire AI value chain.
It starts by offering a critical analysis of the AI Act’s risk classification rules in Article 6, revealing its unexpected complexities and potential pitfalls. While AI systems used in judicial decision-making are presumptively classified as high-risk under Annex III, Article 6(3) provides a significant derogation: systems that “do not materially influence” judicial decisions may avoid being classified as high-risk. The analysis exposes significant legal uncertainty around what constitutes “material influence” in judicial settings, potentially creating loopholes where providers might attempt to avoid stricter compliance requirements through strategic system classification. Moreover, the paper highlights how even systems self-classified as non high-risk remain subject to post-market surveillance and potential reclassification, creating ongoing compliance challenges for providers. This regulatory ambiguity particularly affects AI systems designed to assist rather than directly influence judicial decision-making, highlighting the need for clearer guidance and more precise classification criteria from a socio-technical perspective.
Finally, the paper provides a novel interpretation of how different accountability frameworks apply based on the AI system’s level of influence on judicial decisions. For high-risk applications that materially influence court decisions, the Act mandates robust human oversight measures (Article 14). For systems playing more limited roles, like legal research assistance, different safeguards – including voluntary ones – apply. This nuanced approach helps balance technological innovation with the imperative to protect fundamental rights.
A key finding is the central role of AI literacy in preserving constitutional safeguards. The paper demonstrates how the AI Act’s requirement of AI literacy (Article 4) is more than just a technical specification, it’s a fundamental guarantee of due process. When judges and court staff understand both the capabilities and limitations of AI systems, they are better equipped to maintain meaningful human control over decision-making and prevent undue delegation of judicial authority to algorithms. Moreover, the AI Act addresses the persistent tension between the proprietary nature of AI systems and the right to contest judicial decisions. By requiring explanations about an AI system’s role in decision-making (Article 86), the regulation creates new pathways for meaningful due process—even when dealing with complex, opaque AI models (also for liability purposes, to be explored further).
The research comes at a crucial moment, as courts worldwide experiment with AI tools for judicial assistance. Colombia has already seen judges use ChatGPT in drafting decisions, while various so-called legal copilot products are entering the market. Against this backdrop, the paper’s analysis of the EU AI Act’s accountability frameworks provides timely guidance for courts, policymakers, and AI developers. Looking beyond technical compliance, the paper argues that the successful integration of AI in courts requires a fundamental shift in how we think about judicial decision-making. Rather than viewing judges as mere overseers of algorithms, the focus must be on empowering them as active decision-makers supported by—but never subordinate to—AI systems.
Irina Carnat
Irina is a PhD candidate in Artificial Intelligence for Society at the University of Pisa and Research Fellow at the LIDER-Lab, Sant'Anna School of Advanced Studies. Her research focuses on accountability frameworks for trustworthy AI systems, with particular emphasis on AI governance, product liability for AI-embedded products, and the implications of Generative AI for fundamental rights. She is also a qualified lawyer in Italy.
