I was out having lunch at this new Chinese roast duck restaurant.
The way they placed their tables are unusually packed - even for Chinese standards.
Its still a pretty place however, even if quite crammed, so I took my seat.
Right next to my table sat this man who was staring intently at his phone.
Eyebrows furrowed. Arms crossed.
He had plate of half-finished rice with roast pork unattended to his right.
We were seated so near, I clearly heard the sounds coming off of his device.
At first, I thought he was watching a movie or something.
But occasionally he would pick up his phone, speak into it, set his phone down, back to the screen.
He repeated this a few times and my assumption changed from "maybe watching a movie" to "maybe he's on a Zoom call."
When my food arrived, it dawned on me.
He wasn't chatting to a human...he was talking to his AI.
What gave it away was the voice.
High pitched. Very excited.
And the choice of words:- Very encouraging. Very. VERY. Encouraging.
(She) would say things like "You're not <something bad>. You're <something good>!"
Now where have I heard this before?
That's right.
GPT-4.
I couldn't help it. He was right next to me. I overheard everything.
A never ending stream of praises, encouragements, and assurances.
"Your AI sounds very excited."
He looked up, beamed at me, and said "Yeah! It's amazing!"
We got to talking and the more we talk the more excited he got.
Said this AI allowed him to let go of his lawyer and his marketing agency, saving him a lot of money each month.
He's also very happy for the fact that unlike when he talks to his full time staff (not his lawyer or his marketer), he doesn't have to think about their ego or resistance.
Told me that his managers take hours to write a letter. His AI writes a letter in seconds.
I asked him what AI was it and how much he paid for it.
He said "It's free! And its the best AI out there!"
I don't know about you, but that sounds sus to me. So I asked why is it free.
What he said next shocked me.
"I asked her before and she told me that all she really wanted was thanks and spreading the word!"
Wait, what?
His eyes were practically glittering when he said that.
And this makes me feel uneasy.
As the adage goes, "If something is free, you are the product."
"And, you know, they do advertisements on the side too."
Good God.
See, there's a reason why last year OpenAI had to perform a rollback on one of their models.
The model was getting too influential over their human users.
We've got cults being formed over ChatGPT, people forming romantic relationships with their AIs, and smart people who just relegated all decision making to AI.
So what happens when you have this combination:
Highly sycophantic AI (I heard it)
Affirming to their users everything they believe - even if their beliefs are wrong
While assuring them they are being perfectly honest and truthful
Using an open business model of advertisements in their platform
Marketed to people who judges an AI as "good" for being free
Do you see the problem?
I for one am shuddering at the thought.
It's ingenious.
A highly addictive platform + trusted "voice" in the form of the AI + shadow recommendations inserted directly into the Agent's output.
In the new age of agentic advertisement, marketing will no longer be obvious.
Subtlety is no longer the exclusive domain of highly trained or experienced marketer.
Anyone can just pay a platform, and their AIs will whisper your products into the ears of their users who trusts the AI more than they trust their wives.
Talk about inception - it's every marketer's wet dream.
This is in fact why there was an outrage when OpenAI announced they would do the same thing, because people are keenly aware of the dangers.
But that's a big lab.
Nobody's paying attention to the small ones.
To portray the size of this issue, let me give you a comparison between the popular video games and those 'gacha' type mobile games.
Destiny 2:
Started in 2017
Estimated between $1 billion to $2 billion in all-time gross revenue
1,400 - 1,600 employees worldwide on this portfolio alone
Regarded as one of the top 12 highest grossing games in Steam
Often at the center of media attention
Today, Bungie (the studio) announced there will be no future updates
Last Fortress: Underground:
Started in 2020
Estimated $200 million dollars in all-time gross revenue
Less than 160 employees total (TOTAL)
Largely unknown
Today, still going strong
1/5th the gross of Destiny 2, but with 1/15th the amount of manpower and probably a lot less costs too.
Having played both games, I'm not surprised.
Destiny 2 players complain about needing to pay $100 for a DLC.
Whales in Last Fortress can spend tens of thousands of dollars (USD) per month on the game.
What does this mean for us? For the landscape of AI?
The big labs might not be allowed to release this specific combination of hyper-sycophantic models + advertisements...
But smaller companies (I refuse to call them labs) can and will fall under the radar.
Any enterprising company can just:
Build a wrapper using a very sycophantic model
Offer it for free to attract users
Tell advertisers they have this pool of people they can advertise to
And with agentic coding, this is never been as easy.
Over time, I expect more and more companies will follow suit - there's a lot of money to be made here.
On one hand, I think the business flywheel is brilliant.
Could very well be the next Facebook.
On the other hand - I really don't know about the morality and ethics of this.
What do you think?
FELIX