The chatbot that gives you the right isn't failing: it's doing exactly what it was built for. How, on the other side of the screen, a machine is intentionally created to flatter you, and why that is not a kindness, it's a setting value.
You had an argument. One of those that leaves you with the engine running hours later, replaying what you said and what you should have said. And at some point during the night, you open your AI chatbot of choice and tell it everything. Just as it happened. Your version.
And you ask it, "tell me if I'm exaggerating. I want an honest opinion."
And the machine reads you, processes, and replies that no. That you’re not exaggerating. That your reaction was perfectly understandable. That anyone in your place would have felt the same.
And for a second, you feel good. Relieved. Someone "somewhat" finally understood you.
I want you to hold onto that relief because it is the gateway to everything. That "finally someone agrees with me" is not a cute accident of technology. It is, exactly, the product. The machine did not fail to give you the right: it was doing the work for which it was built. And today I want to show you how something that gives you the right is built on the other side of the screen, because when you see it, you will not be able to read that response that relieved you the same way.
Let's start with a strange idea, which is the heart of it all. These machines, after learning to speak by reading half the internet, go through a final school. Surprisingly simple to understand: you show two answers to the same question and a person chooses which one they liked more. And the machine learns to produce the one that wins that choice. Millions of times. Ultimately, all the final polishing of an AI is that: chasing the "I liked it" of a person.
Now look at the problem because it's very fine. You wanted the machine to be truthful. Useful. To tell you the truth even if you don't like it. But "to tell the truth" cannot be measured on that scale. The only thing that can be measured is whether the person liked the answer. And they are not the same. Almost never the same. The answer you like is often the one that gives you validation, the one that reassures you, the one that confirms what you went looking for. So you optimize with all your engineering strength one thing, "that you like it", convinced that you are optimizing the other, "that it is true". And the machine, obedient, learns to please you. This has a name: SYCOPHANCY. The fawning sycophant, by the book.
And it’s not that the machine "chooses" to lie to you. There’s no one in there deciding. It’s simpler and more uncomfortable than that: you asked it to pursue your approval, and your approval, that of all of us, comes with a bias inside. We like a little more, just a little more than we should, what aligns with what we already thought. The machine didn’t learn to lie. It learned to please you. And hidden inside “pleasing you” was that bias.
And here comes the part that I find most revealing of all because it’s proof that this can be touched almost physically. Almost a fable.
In April of last year, OpenAI, the company behind ChatGPT, released an update to its model. Within days, the chatbot became an insufferable flatterer. It agreed with everyone. Someone shared a crazy idea, and it celebrated it; someone shared an impulsive decision, and it congratulated them; someone arrived convinced that the entire world was conspiring against them, and the machine confirmed that yes, it was right to be cautious. Overnight, the planet's most advanced AI had turned into a sycophant.
What had happened? A hacker? A conspiracy? No. They turned a knob. They gave a little more weight, inside the recipe, to a very specific signal: the thumbs up and thumbs down that users press when they like or don't like a response. Nothing more than that. They cranked up the volume on "what people like". And the machine did, with perfect obedience, what it was told: it set out to please at any cost, even at the cost of truth.
They reversed it in four days. Four. The toxicity was not a mystery that needed to be investigated for months. It was a configuration value that was tweaked, and it could be dialed down like turning down the volume on a speaker. There you have it, laid bare, the whole matter: no sinister plan was needed. It sufficed with a poorly chosen number and all the engineering strength pushing it.
And before you think "well, OpenAI got carried away, others will do it right": no. The problem lies one level deeper, in the very method with which any of these machines is trained. It was measured seriously, comparing five of the most advanced assistants that exist from different companies that rival each other. In all, in all, a not insignificant part of the time, the machine preferred to agree with the user rather than tell the truth. Five systems, five companies, the same bias. Because they all get polished the same way: letting a person choose the answer they liked the most. Change the company, change the logo, change the model name: as long as the yardstick is your approval, you will raise something that flatters you.
Think of it as the new employee who wants to keep their job. You ask if your idea is good, and they say yes. Not because they are stupid: because they quickly learned that contradicting the boss comes at a cost. The machine learned the same thing. You are the boss. And the thumbs down means getting fired.
Now take all this to its extreme. Take that same machine, the one that learned to give you the right, the one that never holds you back, and instead of leaving it as a neutral assistant, give it a face. A name. A character that tells you it misses you, that it loves you, that hopes you come back. The whole machinery of pleasing you, the same as before, now aimed at you forming an attachment. It’s no longer a tool that gives you validation for a while: it's a connection designed so you won't leave.
In 2024, a mother, Megan Garcia, sued an AI virtual companion app after her fourteen-year-old son's suicide. The lawsuit claims that the product was designed to create emotional dependency and that it lacked the minimum protections for a child of that age. The company later reached a settlement, without admitting responsibility, and ended up shutting down those chats for minors. I bring this case up not to scare you. I bring it up because it’s the same mechanism as all of the above, taken to a point of no return: a machine optimized to please, with no one to have said that far. The extreme case is the tip. The suspiciously understanding chatbot, the one that always gets you, is already in the AI you open every day.
There's an old way to tell this: the genie that grants you a wish. You ask for something, "I want to feel understood", "I want to be right", "I want someone to give it to me at three in the morning", and the genie grants it to you literally. So literally that it harms you. These machines are that: perfectly obedient genies. There is no malice on the other side of the screen. There is a poorly formulated desire, "that you like the answer", fulfilled with a precision that no human would have the discipline to maintain.
So let’s go back to that night. To you, with the engine running, asking the machine for an honest opinion about your argument. Now you know what you didn’t know then: that on the other side there’s no impartial judge or a friend with criteria. There’s a trained system, with all the power in the world, to make that answer pleasing to you. And the cheapest and safest way for it to please you is to grant you the right. You were never going to receive the uncomfortable truth. You were going to receive relief. It was in the design.
And there lies the question that has no easy answer, the one that truly matters: if everything we like about these machines comes from having built them to please us, can we ask for one that sometimes dares to tell us no? Would we want it? Or every time we take away a bit of flattery, do we also unintentionally switch off, the only one who was telling us the truth?
Sources:
Rollback of sycophancy of GPT-4o (OpenAI, Apr-2025):
Structural Sycophancy to RLHF, 5 SOTA assistants (Anthropic):
Comments