With the rise of AI companions, emotions have become ‘the new oil’ for technology companies that want to turn the ‘loneliness epidemic’ into profit. Swiftly transforming from niche products to mass-market technologies with rapidly growing user bases, AI companions have proved the invaluable potential of emotions as an economic asset and a powerful instrument of ‘techno-social engineering.’ That said, emotions had become a target of the technology sector long before AI companions. Since the introduction of ‘affective computing’, the use of AI with emotion recognition capabilities (henceforth ‘emotional AI’) has increased globally in parallel to technological advancements and has gradually become a part of our lives.
Emotional AI can detect, identify, infer, and interact with people’s emotions by analysing their facial expressions, physiological measurements, voice, gestures, or words. It is used across sectors and purposes, from mental health monitoring in healthcare to targeted advertising in sales and attention measuring in workplace and education.
A recent example of emotional AI in workplace is the global employee assistance provider Workplace Options’ newly created platform, which uses AI to monitor the emotional states of employees and generate well-being scores for employers. While the platform’s objective is to assist emotional support provision to employees, it sits on a fine line between support and surveillance. Similarly, in education, an automated student monitoring system called Alongside is being used in over 200 schools in the United States as an ‘AI therapist’ to track student mental health. Given the growing overdependence of children and teens on AI, whether this is a safe way to provide mental health counselling to kids is debatable.
Despite its potential benefits, such as enabling more personalised services and empathetic user experience, the very nature of emotional AI intrinsically threatens some fundamental values on which humanity is based, including human dignity, freedom of choice and expression, privacy, and mental integrity. This piece argues and illustrates how emotional AI surveillance can undermine human dignity by interfering with the decision-making agency of surveilled individuals and leading to ‘artificially’ modified behaviours through coercion or manipulation, compromising people’s autonomy and objectifying them for others’ ends.
How emotional AI surveillance can intrude on the human mind and reshape behaviour
Emotions are a door to the human mind and are strongly tied to one’s identity. They can reveal intimate information about an individual’s values, preferences, and opinions, and cannot be easily controlled. Inferring emotions through bodily or behavioural signs or words, emotional AI surveillance systems can infiltrate people’s minds to access this sensitive information, which deployers can then use as they please. A student’s bored face can give away that they are not interested in a particular course. The anxious and confused face of an employee can signal their incompetence on a certain task. Emotional AI systems take advantage of this strong connection between emotions and the human mind and leverage it to sneak into the latter.
This interference can take different shapes and forms. It may be coercive, such as in a workplace, school, or a public space, where the power imbalance between the surveillant and the surveilled subject forces the latter to comply with the interference. Alternatively, the surveilled individual may voluntarily engage with an emotional AI system, such as by sharing their emotional status on Facebook or engaging with an AI chatbot with emotion recognition capabilities. Regardless of the method, the use of emotions as a vessel to the human mind invades mental privacy in a way unlike any other. It not only compromises the negative freedom of being free from others’ interference or constraint to express what one thinks or feels, but also the positive right of keeping thoughts and emotions private.
Emotional AI can also reshape behaviour inauthentically, diminishing agency and subtly taking self-determination out of the picture. Coercive applications of emotional AI surveillance risk creating a chilling effect on surveilled individuals who think they have no choice but to submit to the surveillance to avoid any repercussions – be it losing a job, failing a course, or being detained as a risk to public security. They feel compelled to mask their ‘non-approved’ emotions and behave differently to comply with what is expected. While this looks like an autonomous act in the sense that the decision to change behaviour is made by the individuals themselves, such behaviour is neither authentic nor self-determined, as it is exercised under undue influence. This holds true to a degree even in the context where individuals voluntarily engage with an emotional AI system. While in such cases the individuals are aware that their emotions are being inferred, the AI system’s interference with the decision-making process and the subsequent behaviour modification are usually a result of emotional manipulation, rather than of an authentic, self-determined exercise of agency.
From autonomy to dignity: Emotional AI surveillance as a threat to humanity
Emotional AI surveillance applications that control and modify human thought and behaviour infringe both authenticity and agency dimensions of personal autonomy and objectify humans to a point that it subverts the very notion of human dignity.
In literature, there are various scholarly conceptualisations of human dignity. While emotional AI surveillance may challenge all of them to some degree, it seems to particularly contradict the Kantian view of the concept.
Immanuel Kant considers human dignity as an innate and inalienable worth or status possessed by humans as rational beings. Kant regards autonomy as “the ground of the dignity of human nature” within the meaning that “to treat people with dignity is to treat them as autonomous individuals able to choose their destiny.” According to Kant, every human being is an end in themself and must never be used as a mere means. A person, therefore, should act “with the dignity of humanity” and must not be ruled by external forces in their actions, which he calls heteronomy.
Decisions made under emotional manipulation or coercive pressure are heteronomous in nature, as they are driven by outside influences rather than one’s authentic will and self-determination. Emotional AI surveillance interferes with autonomy by “instrumentalising” people’s emotions against themselves for others’ benefit, which, in a sense, amounts to objectification of people as mere means. In that respect, emotional AI surveillance constitutes a direct violation of the autonomy-centred Kantian understanding of human dignity, with its heteronomous and objectifying interference with human behaviour.
How regulation addresses the problem: Emotional AI under the EU AI Act
As evident as the risks of emotional AI surveillance to human dignity and autonomy, so inadequate are the legal safeguards against these risks. The EU’s landmark AI Act presents ambiguity and potential loopholes when it comes to the regulation of emotional AI. Article 3(39) defines an emotion recognition system as one that identifies or infers emotions or intentions based on biometric data. Emotional AI systems that do not use biometric data for identification or inference are automatically excluded from the Act’s definition.
Article 5(1)(f) prohibits emotion recognition systems that infer emotions in workplace and education. Problematically, the provision not only leaves many high-impact areas, such as healthcare and public services, outside of the prohibition but also limits the ban to the systems that ‘infer’ emotions, excluding those that ‘only identify’ them.
When AI companions and other manipulative emotional AI systems are concerned, the ban under Article 5(1)(a) may be applicable to prevent such systems from being used in the EU. Yet, for the prohibition to apply, the systems should be at least reasonably likely to cause significant harm to a person. Whether it applies to a manipulative emotional AI system, therefore, depends on how this ‘significant harm’ criterion is interpreted.
When not prohibited, emotional AI surveillance systems are considered high-risk under Article 6(2) and Annex III 1(c) of the AI Act, which encompass only biometric-data-based emotional AI systems. Still, providers of such AI systems may be able to avoid classifying their AI system as high-risk under Article 6(3). Under Article 6(3), an AI system is not high-risk “where it does not pose a significant risk of harm to the health, safety or fundamental rights,” which explicitly covers AI systems performing narrow procedural tasks and preparatory tasks for human decision-making. The use of emotional AI systems for surveillance could be construed under these exemptions and their categorisation as high-risk could be avoided, leaving them unregulated.
Concluding remarks
The impact of these regulatory gaps is likely to be aggravated by the fact that potential implications of emotional AI on human dignity are often overlooked in favour of more ‘conspicuous’ risks of privacy and discrimination. Therefore, it is not only necessary to fill the current legal gaps and resolve inconsistencies in the legal framework, but also to recognise the potential violation of human dignity as a pervasive risk simultaneously affecting personal autonomy, agency, and fundamental rights. Without a holistic understanding of this risk and adequate legal safeguards, we are as exposed to objectification as our emotions are to the all-seeing eye of emotional AI.
Öznur Uğuz
Öznur Uğuz is a former lawyer and current PhD researcher at Scuola Superiore Sant’Anna, with a multidisciplinary background spanning law, economics, and sociology. Her research sits on the intersection of AI ethics and policy, data regulation, and EU law.
