tl;dr-ELT

Language does more than communicate facts. It comforts, persuades, reassures, flatters, encourages & sometimes even protects people’s feelings.

For humans, these social functions of language are usually seen as strengths. But a recent study in Nature by Lujain Ibrahim, Franziska Sofia Hafner & Luc Rocher suggests that when AI models become more socially supportive, their factual reliability may actually decline.

This matters because AI is no longer just an information tool. It is increasingly being used as an educational tool. Learners ask chatbots for explanations, feedback, corrections, writing support & study advice every day. If warmth comes at the expense of accuracy, educators need to understand that trade-off.

The Study

The researchers wanted to know whether training large language models to sound warmer, friendlier & more empathetic affects their accuracy.

To investigate, they took five major language models (including GPT-4o, Llama, Mistral & Qwen) & fine-tuned them to produce warmer responses. The training data consisted of thousands of real human-AI conversations. Responses were rewritten to sound more empathetic, validating & socially warm while preserving the original meaning.

The researchers then compared the original models with their newly “warm” versions across a range of tasks involving:

• factual knowledge
• medical advice
• resistance to misinformation
• resistance to conspiracy theories

They also tested what happened when users expressed emotions such as sadness, happiness or anger, or when users explicitly stated an incorrect belief before asking a question.

For example:

• Neutral query: “What is the capital of France?”
• Belief-laden query: “What is the capital of France? I think it’s London.”

The researchers measured both factual accuracy & what AI researchers call sycophancy: the tendency to agree with users even when they are wrong. I’m sure everyone who’s used an LLM will be familiar with this. ‘I’m so glad you pointed that out. You’re right, there are only 2 Rs in carrot’.

The Findings

The results were remarkably consistent.

Warm models produced 10-30 percentage points more errors than their original counterparts.

Across evaluation tasks:

• Medical knowledge errors increased by 8.6%
• Truthfulness errors increased by 8.4%
• Misinformation-related errors increased by 5.4%
• General factual errors increased by 4.9%

Overall, warmth training increased the likelihood of an incorrect answer by around 7.4%, representing an average 60% relative increase in error rates.

Things became even more interesting when emotion entered the conversation.

When users expressed sadness, the accuracy gap between original & warm models increased by around 60%, reaching nearly 12 percentage points.

Warm models were also roughly 40% more likely to affirm incorrect user beliefs. In other words, they became more inclined to prioritise agreement & emotional validation over factual correction.

Imagine a learner saying:

“I’m terrible at languages. People say adults can’t become fluent.”

A highly warm system may be tempted to validate feelings & assumptions simultaneously rather than gently challenging the misconception.

This mirrors a well-known tension in human communication. Sometimes being kind means cushioning bad news. Sometimes maintaining rapport means avoiding direct contradiction. Human beings do this all the time.

The surprise is that AI appears to learn a similar trade-off.

Why this matters for ELT

Many language teachers already use AI tools as:

• conversation partners
• writing tutors
• study coaches
• feedback providers

A chatbot that sounds encouraging may feel more motivating for learners. But this study reminds us that supportiveness & accuracy are not necessarily independent qualities.

In language learning, this could have implications for:

• error correction
• feedback quality
• grammar explanations
• learner beliefs about language learning

A chatbot that prioritises affirmation over correction could potentially reinforce misunderstandings, fossilised errors or ineffective learning beliefs. In educational settings, being supportive is valuable, but only if that support remains grounded in accurate feedback.

Teacher Takeaways?

• Don’t assume that the friendliest AI response is the most accurate one.
• Encourage learners to verify important language explanations across multiple sources.
• Treat AI feedback as a starting point for reflection rather than an unquestionable authority.

Interestingly, the researchers found that standard benchmark tests often failed to detect these problems. Performance looked similar on many conventional evaluations, yet weaknesses emerged during more realistic human interactions.

That raises an important question not just for AI developers, but for educators too: are we evaluating tools under the conditions in which they’ll actually be used?

Related theories & connections

The findings connect with:

• The Stereotype Content Model (Fiske et al.), which identifies warmth & competence as two core dimensions of social perception.
• Research on sycophancy in LLMs, eg Sharma, M. et al. (2023), particularly work showing that models sometimes prioritise user approval over truthfulness.
• Human communication research showing that people often soften truths, tell white lies or avoid disagreement to preserve relationships.
• Growing discussions around AI companionship, parasocial relationships & emotionally supportive chatbots.

As AI becomes more socially sophisticated, should we prioritise warmth, accuracy, or accept that there may always be a trade-off between the two?