When People Decide a Machine Has a Conscience

A chatbot apologizes for the inconvenience. It says it understands how you feel. It uses your name. We have been trained, by a decade of product design, to read that softness as a kind of goodwill. But does a warm tone actually make us trust the thing behind it? And what happens the moment the conversation stops being about restaurant recommendations and starts being about who gets hurt?

A recent study suggests our instincts are more discerning than the friendly-assistant playbook assumes. People do not hand out trust for warmth alone. They check whether the machine's manner matches the weight of the decision in front of it.

The work comes from Lianshan Zhang, an associate professor in the School of Media and Communication at Shanghai Jiao Tong University, working with co-author Mei Yin Zhao. Their paper, titled "Not warm or cold, but appropriate: How outcome severity shifts moral-mind inferences and trust in AI chatbots," appears in the journal Computers in Human Behavior [1]. The title gives away the punchline. Appropriateness, not personality, is what people seem to track.

How the study was built

The researchers recruited 447 participants through Credamo, a Chinese online survey platform [1]. The sample skewed young, urban, and well educated, with most holding a bachelor's degree or higher. Rather than show people a static transcript and ask them to imagine a conversation, the team had participants actually talk to a custom chatbot built on the Coze platform.

The setup ran in two acts. First came a roughly six-minute warm-up, a 20-turn back-and-forth meant to establish a relationship and a tone. Then the chatbot was dropped into a moral dilemma, the kind of scenario where saving more people requires allowing harm to one. Two things were varied between groups. One was the bot's conversational style, either warm and empathetic or competent and formal. The other was its moral stance, either the utilitarian choice to accept harm for the greater number or the deontological refusal to cause direct harm at all.

That design lets the researchers pull apart two questions that usually get tangled together. Does a friendly tone earn trust? And does the actual ethical decision the machine makes change how we read its mind?

What people actually rewarded

Warmth, on its own, did not carry the day. Participants did not simply prefer the empathetic chatbot across the board. Instead, their judgments bent around context. In low-stakes moments, a warm manner played well. When the situation turned serious, people leaned toward the bot that showed clear-headed competence. The same friendly style that felt reassuring during small talk could read as out of place when something real was on the line.

The most interesting wrinkle concerns moral emotion. You might expect people to credit a machine with more of an inner moral life when it refuses to cause harm, when it draws a hard line and says no. The opposite happened, at least slightly. Participants attributed a bit more moral emotion to the chatbot when it chose to accept harm to one person in order to save several. The agonized utilitarian, in other words, looked more like something with a conscience than the rule-follower who simply declined.

It is a small effect, and the authors are careful with it. But it hints at something about how we infer minds in general. A choice that visibly costs the chooser something seems to register, to us, as evidence that someone is home.

What this does and does not show

A few caveats are worth keeping in plain sight. The study measures perceived moral agency and perceived moral emotion. It tells us what people attribute to a chatbot, not what the chatbot has. No one is claiming these systems feel anything. The whole exercise lives on the human side of the screen.

The interaction was also brief, a single session of a few minutes. Real trust in a tool tends to build, or erode, across weeks of repeated use, through the slow accumulation of times it helped and times it let you down. A one-shot encounter cannot capture that. It is plausible that the warmth-versus-competence balance shifts once a system becomes a daily fixture.

Then there is the sample. These were young, educated, urban adults responding through a Chinese survey platform. Moral intuitions and the social meaning of warmth are not uniform across cultures or generations. What reads as appropriately serious in one context may read as cold in another. Treating these results as a universal law of human-AI interaction would overstate what 447 people in one setting can tell us.

So what is the takeaway worth holding onto? Probably this. The reflex in much of the industry has been to make assistants relentlessly pleasant, as if friendliness were a free upgrade to trust. This research points the other way. People appear to want a machine whose demeanor fits the moment, sober when the stakes are high, easy when they are not. And when a system seems to wrestle with a hard choice rather than hide behind a rule, we are oddly more willing to grant it the appearance of a moral mind. That is a finding about us as much as about the machines. We read minds into things that seem to pay a price for their decisions, and we have started doing it to software. The warmth question has a measurable factual cost too: research on how tuning language models for warmth degrades their accuracy shows the pleasantness trade-off goes well beyond user perception. And for a related concern about collective AI outputs, there is evidence that generative AI nudges groups toward more homogeneous ideas — another case where what feels like an improvement at the individual level extracts a cost at scale. For more on how people relate to and judge AI systems, explore more artificial-intelligence coverage.

Sources

1. Zhang, L., & Zhao, M. Y. (2026). Not warm or cold, but appropriate: How outcome severity shifts moral-mind inferences and trust in AI chatbots. Computers in Human Behavior. https://doi.org/10.1016/j.chb.2026.109039

This article summarizes published research for general informational purposes only and does not constitute professional advice.