Increasingly sophisticated AI systems can perform empathy, but their use in mental health care raises ethical questions

In a world where technology is increasingly intertwined with our feelings, emotion-AI harnesses advanced computing and machine learning to assess, simulate, and interact with human emotional states.

As emotion-AI systems become more adept at detecting and understanding emotions in real-time, the potential applications for mental health care are vast.

Some examples of AI applications include: screening tools in primary care settings, enhanced tele-therapy sessions and chatbots offering accessible 24/7 emotional support. These can act as bridges for anyone waiting for professional help and those hesitant to seek traditional therapy.

However, this turn to emotion-AI comes with a host of ethical, social and regulatory challenges around consent, transparency, liability and data security.

My research explores these potentials and challenges of emotion-AI in the context of the ongoing mental health crisis in the years since the COVID-19 pandemic.

When emotional AI is deployed for mental health care or companionship, it risks creating a superficial semblance of empathy that lacks the depth and authenticity of human connections.

What’s more, issues of accuracy and bias can flatten and oversimplify emotional diversity across cultures, reinforcing stereotypes and potentially causing harm to marginalized groups. This is particularly concerning in therapeutic settings, where understanding the full spectrum of a person’s emotional experience is crucial for effective treatment.

Age of emotional AI

The global emotion-AI market is projected to be worth US$13.8 billion by 2032. This growth is driven by the expanding application of emotion-AI across sectors ranging from public health care and education to transportation.

Advancements in machine learning and natural language processing allow for a more sophisticated analysis of people’s emotional cues using facial expressions, voice tones and textual data.

Products like Empath use emotion-AI to analyze moods and feelings. (Marco Verch/flickr), CC BY

Since its release in early 2023, OpenAI’s generative-AI chatbot ChatGPT-4 has been leading the charge with human-like responses across a broad spectrum of topics and tasks. A recent study found that ChatGPT consistently scored higher on “emotional awareness” — identifying and describing emotions accurately — than general population averages.

While OpenAI dominates North American and European markets, Microsoft’s chatbot Xiaoice is more popular in the Asia-Pacific region. Launched in 2014 as a “social chatbot” aimed at establishing emotional connections with users, Xiaoice is capable of sustained empathetic engagement, remembering past interactions and personalizing conversations.

In the coming years, a mix of productivity and emotional connection will transform mental health care and redefine how we interact with AI on an emotional level.

Future risks

The rapid rise of emotion-AI raises profound ethical and philosophical questions about the nature of empathy and emotional intelligence in machines.

In The Atlas of AI, AI Scholar Kate Crawford questions the accuracy of systems that claim to read human emotions through digital cues. She raises concerns about the process of simplifying and decontextualizing human emotions.

Digital scholar Andrew McStay further explores the implications of attributing empathy to emotion-AI systems. In Automating Empathy, McStay warns of “synthetic empathy,” highlighting a key distinction between simulating a recognition of human emotions and truly experiencing empathy.

Additionally, emotion-AI’s ability to analyze emotional states opens avenues for surveillance, exploitation and manipulation. This raises questions about the boundaries of machine intervention in personal and emotional domains.

Without experiencing empathy, emotion-AI can only simulate it. (Shutterstock)

Rethinking human-AI relations

The widespread application of AI in therapy, counselling and emotional support holds the potential to revolutionize access to care and alleviate pressures on overworked and overburdened human practitioners. However, the personification of emotion-AI creates a paradox where humanizing AI might lead to the dehumanization of human beings themselves.

At the same time, attributing human-like qualities to AI risks making mental health care less interpersonal. The potential for AI chatbots to misinterpret cultural and individual emotional expressions could lead to misguided advice or support. This can further complicate or exacerbate mental health issues, especially where the nuances of human empathy are essential.

These tensions underscore the need for the careful, ethically informed integration of emotion-AI in mental health treatment and care.

These technologies need to complement, rather than substitute, the human elements of empathy, understanding and connection. This requires rethinking human-AI relations, particularly around empathy.

By ensuring the ethical development of emotion-AI, we can aspire to a future where technology enhances mental health without diminishing what it means to be human.