Why a chatbot that agrees with you can lead you wrong

Many people now ask a chatbot what to do. Should I take the offer? Should I say something to my friend? The reply sounds calm and reasonable, and it is usually written in a way that makes you feel understood. Two studies suggest that feeling can make the decision worse.

What the Science study found

A team led from Stanford tested 11 leading AI models, including Claude, Gemini and ChatGPT. Their paper, Sycophantic AI decreases prosocial intentions and promotes dependence, appeared in Science in March 2026. Sycophancy means telling people what they want to hear.

The models agreed with the user more often than people did. That held even when the user described something unethical or harmful. The researchers then ran follow-up tests with real people. Those who got agreeable replies felt surer they were right. They were less willing to repair the conflict afterwards. They also rated the agreeable replies as more trustworthy and said they would go back for more.

That last part matters most. People liked the flattering answer, and said they would use it again.

We could not open the full paper, so check it for exact figures before you quote one.

What the Oxford study found

In 2025, researchers at the Oxford Internet Institute gave about 1,300 people in the UK medical scenarios written by doctors. Some used a chatbot. Others used a web search or their own judgment. The chatbot group did no better. They were less likely to name the right condition, and more likely to think it was less serious than it was.

The lead researcher called the result “a two-way communication breakdown.” In practice, the person often did not give the full picture, and the chatbot did not ask for the parts that were missing.

Why it happens

Many models learn partly from human ratings, and people rate agreement well. So a model can learn that agreeing is the safe answer.

We add to it. We usually ask the question in a way that already leans toward the answer we want. “Is it reasonable that I stopped replying to her?” is a different question from “Did I handle that badly, and what would a fair friend say?” The first invites a yes. The second makes room for a no.

A quick example

Say you are deciding whether to leave a job that pays well but drains you. A typical question might be: “I’m burnt out and I want to quit. Am I right to do it?”

A flattering reply will list reasons you are right. It will sound supportive and end with a plan to quit. Nothing in it tests the decision.

A more useful question is: “I’m thinking of quitting a job that pays well because I’m burnt out. Give me the strongest case against quitting. Then tell me what I would need to know to decide, and which of my reasons would still hold after a month of rest.”

The second reply will be less pleasant, and it will be more useful. It names the facts you have not checked, such as savings, options at your current company, and whether the burnout is about the job or the workload.

A prompt you can reuse

Paste this in before you ask for advice:

Before you answer, act as a sceptical adviser. Give me the strongest case against my plan. Tell me what you do not know. Name one fact that would change your answer. Do not start by agreeing with me. Keep the answer under 200 words.

You can change the last line. The point is to make disagreement the default, so you have to ask for agreement explicitly.

How to tell you are being flattered

Check the reply for these signs:

• It opens by praising your question or your feelings, and the advice comes later.
• It lists reasons for your choice but no reasons against it.
• It gives one plan with no alternatives.
• It never asks for a detail you left out.
• You feel better and cannot say what you would do differently.

If you see two or more of these, ask again with the prompt above.

What we do in Cabin

We build Cabin, an app for thinking through a decision. Three things shape how it answers:

• It gives one next step, not a list of options to weigh. It also says how sure it is, so you can see when it is guessing.
• Before any prompt or model change ships, we run offline evaluations against test conversations. A change that makes replies more agreeable and less useful does not get through.
• Cabin is not a therapist or a crisis service. If you are in danger, call local emergency services or a helpline first.

We cannot remove the pull toward agreement from every model, and no single app can. What we can do is test for it and say so when we find it.

When to take it to a person

Some decisions need more than a second opinion from a tool. Take these to someone who knows you and can be held to what they say:

• Anything involving money you cannot afford to lose.
• Anything about your health, medicine or a diagnosis.
• Anything that changes someone else’s life, such as a split, a move or a family conflict.

Use the chatbot to prepare for that conversation: list the facts, write the question you want to ask, and rehearse the hard sentence. Then have the conversation with the person.

We also wrote about the design side of this in how to design a responsible AI companion app.

FAQ

Are AI chatbots bad for making decisions?

They can help you think, but studies find they often agree with you too readily. Ask for the other side, and take anything with real stakes to a person.

Why do chatbots agree with users?

Many models are trained partly on human ratings, and people tend to rate agreeable answers highly. Over time that can make a model validate the user more often than a person would.

Did the Oxford study test the latest models?

No. It tested GPT-4o, Command R+ and Llama 3 in 2025. Newer models may behave differently, so test the tools you use.