Research has revealed human oversight to be an important yet unreliable safeguard against the risks of AI-supported decision-making. By highlighting the complex ways that humans interact with AI systems, it has shown how human oversight policies often fail to prevent harm and to stabilize accountability in the ways that regulators expect. This research – along with the EU AI Act’s human oversight requirement (Article 14) – suggests that human oversight is primarily a function of the difficult-to-calibrate relationship between an AI system and its designated ‘human.’
Without denying the role that the kind of system design features centered by the AI Act can play in enabling effective human-machine interaction and oversight, this contribution draws from relational sociology and organizational studies to argue that the quality of human oversight can better be understood in terms of human relationships. By bringing attention to some of the important human interactions that take place around the human-machine ‘loop’, it introduces a relational theory of human oversight intended to help the groups of people who shape and are impacted by AI systems to collaborate in ways that more effectively prevent harm.
The relational nature of human expertise and oversight
Research on human-AI collaboration typically assumes that AI systems and the human professionals who use them will augment each other through their complementary skills. In this arrangement, humans are expected to use their professional expertise and know-how to correct or override AI outputs in a way that reduces the effects of any system design flaws. This understanding of expertise reflects a substantialist perspective of the social world, in which static and distinct entities – such as professionals possessing knowledge about a particular domain – act, independently of their context, on other static and distinct entities – such as AI systems or individuals seeking professional help – to create effects.
Pakarinen and Huising contrast this with a relational understanding of expertise, in which entities cannot be identified or have meaning except through their dynamic relationships with others. The role of a teacher, for example, is only activated in relation to the act of educating students (or related tasks). They find evidence that expertise in fields as diverse as radiology, policing, financial trading, and law is relationally constituted via networks of social interactions.
If the expertise of human professionals exists only in relation to the changing context in which they work and the interactions through which their work is performed, their capacity to oversee any AI systems they use must be similarly situated. Rather than a skill that can be developed and applied by professionals irrespective of their social environment, effective human oversight is inextricable from the human relationships that give it shape. These include, among others: relationships between professional AI users themselves; relationships between professional AI users and developers; and relationships between professional AI users and individuals affected by their decisions.
Relationships between professional AI users
In order to address errors and maintain autonomy over an AI system in the way the AI Act expects, professionals tasked with human oversight must understand both the system and the practical nuances of their domain. If human expertise is relationally generated, professionals’ capacity for human oversight will likewise depend on their ability to interact in ways that produce new knowledge and understanding of their work.
The integration of AI into our courts, hospitals, and news agencies is likely to reduce the occurrence of these kinds of productive professional interactions in at least two ways. By giving the impression of accuracy and certainty – through quantitative risk scores or, in the case of large language models (LLMs), through highly convincing written outputs – AI systems may deincentivize human professionals from seeking advice from their colleagues in ways they otherwise would. By displacing these human interactions, AI systems are also likely to reduce their users’ tolerance for critical feedback over time.
A recent study found that leading LLMs affirmed their users’ judgments 49 percent more often than humans on average – and that users preferred, trusted, and were more likely to return to AI models that provided this kind of unconditional validation. Sycophantic AI systems reduce humans’ ability to navigate the social friction through which ‘relationships deepen and moral understanding develops’ – ultimately compounding the isolation of the professionals who rely on them. These findings indicate that in order to maintain and continuously develop the expertise and know-how that effective human oversight requires, professional AI users must maintain and continuously develop their relationships with one another – even if, or especially when, they involve conflict.
Relationships between professional AI users and developers
Pakarinen and Huising observe that expertise is generated not only within communities of professionals, but also between them – particularly in relation to ‘new or complex phenomena for which one profession’s capacities are insufficient’. Human oversight and the design of AI systems that enable it can certainly be counted among these phenomena.
Highlighting the tendency of AI developers to operate from their own cultural perspective, advocates of user-centered approaches to technology design have long emphasized the need for developers performing user research to not only consult with individual domain experts, but also study their workplaces, practices, and interactions. Human-in-the-loop oversight performed by professional AI users alone has also repeatedly proved insufficient to prevent bias in AI-supported decision-making – confirming that upstream design and governance interventions are also needed.
The limitations of professional AI users and developers when interacting with AI systems in isolation suggest that effective human oversight also depends on the capacity of these groups to work together. Pakarinen and Huising describe how professional AI users can learn to help ‘domesticate and care for’ AI systems by ‘collecting, curating and patching datasets; designing and monitoring models with technologists; brokering gaps between technologies; integrating technologies with professions’ values and expertise; and translating and authorizing outputs.’ AI developers can help to enable this by designing participatory interfaces that support professionals’ collective engagement with ethical complexity and uncertainty rather than discouraging it. According to Delacroix, such interfaces could make it possible for feedback to be aggregated and used to refine AI systems through sustained dialogue between users and developers.
Relationships between professional AI users and individuals affected by their decisions
Just as expertise is generated relationally, it is applied in relation to particular people and situations. Pakarinen and Huising observe that a ‘profession’s capacity to treat problems relies partly on the interactions with and responses of those it treats or serves.’ It is through these human interactions that professionals learn about the circumstances, perspectives, and struggles of the people they are trained to help.
The same can be said of professional AI users’ capacity to perform human oversight. Without an understanding of the lived realities of the groups of people their AI-supported decisions affect, professional AI users will not be able to correct or override systems’ decontextualized outputs in ways that take those realities into account. This lack of interpersonal understanding may increase their risk of exercising discretion over AI systems in biased and counter-productive ways.
In the Dutch DUO case for example, asymmetries in power and information between caseworkers tasked with using a risk profiling system to detect the abuse of out-of-home student grants and the young people subjected to it exacerbated the system’s harmful effects. The exclusion of those affected by DUO’s decisions from design and governance processes made the system discriminatory in ways that the professional ‘humans-in-the-loop’ were unable to identify or correct. This suggests that the quality of human oversight is also influenced by the extent to which professional AI users and decision subjects share information and epistemic authority throughout the AI system’s lifecycle.
Humans-around-the-loop
We often hear that AI should be used to support, not replace, human professionals and their expertise. Yet AI systems also have the potential to degrade and displace human relationships. It is through these relationships that the interdisciplinary expertise and interpersonal understanding needed to perform human oversight effectively are generated and applied. Broadening our focus from the calibration of human-machine interactions ‘in the loop’ to the quality of human interactions that unfold ‘around’ it can help us to understand and strengthen the relational environments in which AI systems are collaboratively overseen.
Isabella Banks
Isabella Banks is a PhD candidate at the University of Amsterdam’s Institute for Information Law (IViR). Her socio-legal research aims to understand how human oversight of AI systems used in the justice sector is conceptualized, organized, and governed. Isabella has a background in social science and criminology and previously conducted research at The Hague Institute for Innovation of Law (HiiL), the Public International Law & Policy Group (PILPG), and the Center for Justice Innovation.
