Automation and data-driven algorithmic processing have become common tools for decision-making and the delivery of government services. Software systems participate in government processes in diverse ways, from simple case management, to translating and codifying legislation, to automated determinations of rights, privileges, and statuses. Automated decision-making in the public sector has, however, provoked a range of anxieties around the capacities of technical systems to affect individuals’ lives in ways that are incompatible with rule of law-based justifications for governmental authority. Objections to decision-automation go beyond arguments that technical systems are more prone to error. They include deeper claims that decisions and decision-making criteria become opaque and unchallengeable—making government action less accountable, and that decision-automation elides the forms of human moral action required by liberal political theory. Put differently: automation challenges the humanised vision of political life.
In response to that challenge, ‘human oversight’ has become central to mitigating the effects of algorithmic and automated decision-making on the values of liberal governance. Policy edicts and regulatory requirements specifying a human in the loop, human oversight, or meaningful human control are already well-established governance apparatuses. Often, these rules insist on a final human decision-maker. A growing literature, however, has shown the limits of that approach, and its inconsistency with the material realities of technologically mediated decision systems. Responding more closely to those realities, the focus has shifted to interrogating how humans can be integrated into technical decision-making workflows in ways that still preserve the authorising power of human autonomy. Examples draw on Human-Centred Design and Human Factors Engineering research to implement forms of ‘function allocation’, in which certain aspects of decision-making are given to humans and other factors are given to machines. These efforts tend to accept the limits of human oversight as a governance paradigm, but nonetheless seek ways to optimise ‘efficiency’, ‘safety’, and ‘performance’ while maintaining sufficient human input to morally justify system outcomes.
Unsurprisingly, these approaches embed stylised assumptions about human capacities and behaviours, reducing the human to a unit of political, cognitive and deliberative value that can be sprinkled into an automated system as a type of liberal politico-moral seasoning. But the notion of the human is historically variable. It not only has meant different things at different times, but it carries a certain excess of meaning: a charge of elevated morality, dignity, and capacity for reflection and judgment. In these ‘human strategies’ for governing automation, the human is fashioned as the inherent bearer of an agency that is by definition autonomous in relation to technology; advanced dialectically as an increasingly positive legal concept to counter a rapidly technologising law. This is deficient for a range of reasons.
First, where governance efforts focus on implementing some uninterrogated notion of human agency to counterbalance automation, they ignore the more complex changes associated with decision systems becoming increasingly decentralised, data-driven, and calculation-based. Regulatory attention that focuses on the presence, non-presence, or quality of that agency thus risks compounding problems of transparency, accountability, explanation and oversight, by appearing to alleviate them.
Second, insisting on the moral and political value of human agency in the context of deepening decision automation and algorithmic administration creates a more profound but less obvious risk. To turn reflexively to the human is to condition our imagination of what administrative processes are today, what they might become, and in what new ways they might be regulated and evaluated. The human’s affordances are invoked as a proxy for a past set of constitutional arrangements, and offered as the template-standard for the formation and evaluation of new technological processes. We see this, for instance, in the European context with explicit calls to fashion the standards, processes, rights and protections of automated decision-making after those of existing ‘human decision-making’, and more extremely, for example, in the Australian jurisprudence indicating that a technically mediated administrative decision does not even take place unless it expresses a subjectively held human mental state.
We suggest that, to the extent automation remains part of administrative action, these calls to ‘humanise’ automated decision-making prevent a more necessary and fundamental evaluation of whether the ‘decision’ itself can be sustained as the legal, political, or technical site where values are operationalised. Looking more closely at the jurisprudence of administrative decision reveals that it has some specific process characteristics. A ‘decision’ is a final outcome that materially determines a decision subject’s right or entitlement. This final determination is arrived at after a specific kind of deliberation, using reasoning to apply policy (that has been deliberately enacted in legislation) to a concrete case (that has been deliberately communicated to a decision-making authority for the purpose of determination). Yet in the contemporary administrative state, this image no longer reflects the range of ways that government activity mediates the relation between government policy and affected individuals. To illustrate those changes, we draw attention to three closely connected shifts.
1. Disaggregation not final decisions: Newer decision tools disaggregate elements of decisions into something like decision supply chains, or decision pipelines, that change where the rights or privileges of decision-subjects are affected. Australian Freedom of Information challenges have revealed that computational decision-making systems introduce new sites of decision agency that substantially reorganise how government action takes place. Without actively outsourcing government decisions, these systems distribute data to third parties’ calculative tools through software-as-a-service architectures, introducing complex decision data flows that escape legal architectures of accountability premised on human decision-making. When information was requested about the details of those calculations, Government had to acknowledge that while it made the relevant decisions that affected those decision-subjects, it was unable to understand or have access to information about the calculations that constituted how those decisions were made.
2. Calculation not reasoning: European Article 22 cases like Schufa have addressed this problem by specifying that unless there is meaningful human action at the final decision ‘node’, those computational calculations will be taken to constitute decisions for the sake of transparency and accountability requirements. While closing an accountability gap, this conflates computational calculation with the human interpretation of rules in light of presented facts. But these are not the same. Similarly, using AI to extract salient information from application documents, or relying on data entered by third parties to establish facts, is different from a decision-maker’s consideration of a case file. New data flows and data processing outsource the cognitive action that grounds the moral rationale and value of human oversight. Using Large Language Models as mechanisms for interpreting statutory rules or internal business processes, as increasingly occurs in public service entities, is meaningfully different from human statutory interpretation.
3. Unclear decision-making agency: The majority of automated administrative decision-making tools involve forms of ‘robotic process automation’: ongoing, iterative interactions that guide human decision-makers through decision processes. Such systems funnel human actions through letter/email templates, drop-down boxes, and automated interpretations of statute and ‘business rules’. Scholars in organisational studies refer to this as workplace technology’s ‘regulative’ function, and explicitly describe how embedding organisational patterns and practices in technology constrains decision-maker autonomy and discretion. Here, engineering and design become the disciplines that determine which aspects of a decision are substantive enough to require human attention and which can be excluded from the space of deliberation. Where that human agency is expressed, it is typically limited in order to realise the consistencies and labour efficiencies that motivated the introduction of automation in the first place.
As government administration becomes increasingly technicised, we argue that there are serious limits and risks in attempts to preserve liberal democratic governance norms by reasserting ‘the human’, as well as the corresponding legal paradigm of ‘decision’. In our view, the concept of ‘decision’ has become a poor fit for the reality of how algorithmic decision systems materially determine the rights and entitlements of decision-subjects. If the Australian requirement that administrative decisions be a product of a human mind feels unrealistic, and European approaches deeming scores and calculations to constitute decisions feels awkward, perhaps the common problem is the category of decision itself. Efforts to deem conceptually and organisationally incompatible processes as ‘decisions’, in our view, may fall as short as deeming humans capable of effectively overseeing algorithmic processes. Instead, we regard the contemporary evolutions in administration—which reflect and embed new demands of scale, speed, and the becoming-computational of life—as marking a distinct, historically emergent paradigm of politico-legal normativity that cannot be ignored or papered over by human oversight rules or the expansion of decision as a legal category to include computational processes or mere data entry. Instead, these decision-systems must be taken seriously as a new form of political-administrative action that requires a novel response and approach to governance.
Jake Goldenfein
Jake Goldenfein is a law and technology scholar at Melbourne Law School, University of Melbourne and a Chief Investigator of the Australian Research Council Centre of Excellence for Automated Decision-Making and Society (ADM+S). His recent work on AI and automation in the public sector includes empirical research on AI procurement by state entities, as well as conceptual analyses of the ways that new technology alters the basic structures of government labour and processes.
Connal Parsley
Connal Parsley is Reader in Law at the University of Kent (UK), a UKRI Future Leaders Fellow, and an Affiliate Researcher at the ARC Centre of Excellence for Automated Decision-Making and Society. He leads the Future of Good Decisions project, developing new design and evaluation protocols and review criteria for administrative decision-making involving algorithmic automation. His interdisciplinary scholarship draws on legal studies, political theory, philosophy of technology, critical AI and media studies, and experimental prefigurative research methods.
