By Ralf Biesemeier
There is a conversation I’ve been putting off. You know the kind. You sit down across from someone who has been a big part of your life — someone you invested in, believed in, reorganized your whole working day around — and you say, as gently as you can: This isn’t quite working out the way I hoped.So. Deep breath.
Hey, AI. We need to talk.
The Honeymoon Is Over
I remember the early days clearly. The sheer astonishment of it. Drafts in seconds, research summarized in moments, tasks that used to take hours compressed into minutes. I told colleagues, clients, anyone who would listen: This changes everything.
And it did. It does. I don’t want to be unfair.
But something has shifted. The results feel different now. Not always, not in every session — but often enough that I’ve started to notice a pattern, and to be genuinely troubled by it. I work in publishing. I use AI every day. I know how to write a prompt. And still — what comes back to me is, increasingly, unreliable. Sometimes obviously wrong. Sometimes subtly wrong, which is worse.
The Pen That Didn’t Drop
A concrete example — not mine, but one that has stayed with me:
Someone tested GPT-4o with a simple physical experiment. They held a pen horizontally in both hands and asked the AI what would happen if they let one hand go. Correct answer: The pen would rotate downward under gravity. Then they let go. The pen didn’t move. And the AI reported confidently: “I can see the pen rotating, just as expected.”
It insisted it had witnessed something that never happened. When challenged, it invented explanations. The model didn’t see reality. It saw what it had predicted — and when reality failed to cooperate, it edited reality. (YouTube Short)
That video is almost funny. Except it isn’t. It’s a perfect metaphor for a deeper problem.
Ask Again. Get a Different Answer.
Research confirms what many users have observed: Ask the same question twice and you may get completely different answers. GPT-4 showed measurable performance drift over just a few months. Responses during peak usage hours have been found to be shorter and less reliable (ScaleMath, 2024).
For scenario planning, comparative analysis, evaluating options — this is a fundamental problem. And the longer a session runs, the worse it gets. Constraints established early in a conversation are quietly dropped. Requirements acknowledged in one message are forgotten two messages later. The AI doesn’t tell you it has lost the thread. It just drifts. And you only notice when you go back and check — which you have to do, every time.
The Washing Machine Problem
A piece in Die Zeit (11 March 2026) made an argument I haven’t been able to shake. It cited Keynes’s famous 1930 prediction that rising productivity would eventually allow for a fifteen-hour working week. Britain’s GDP has grown enormously since then — and yet the average working week is around thirty hours, not fifteen. The gains didn’t create leisure. They created more consumption, more output – and more laundry.
The washing machine was the key example. Before automatic washing, men changed their shirt roughly once a week. After? Daily. The machine didn’t save time. It raised the standard.
AI, I fear, may be my washing machine.
A 2025 METR study found that experienced developers using AI tools took 19% longer to complete tasks than those working without them (METR, 2025). Not faster. Slower. The overhead of checking, correcting, and verifying ate the time saved — and then some. This is increasingly called AI Fatigue: the slow erosion of enthusiasm into wariness, the cognitive cost of never quite trusting the tool you depend on.
Running Out of Road
There is a structural problem underneath all of this. Epoch AI projects that the available stock of high-quality human-generated text — the raw material these models are trained on — could be exhausted by 2026 or 2027 (Epoch AI). As models increasingly train on AI-generated content, quality may degrade in ways that are hard to trace. Researchers have identified a “scaling wall” — the point at which making models larger simply stops producing meaningful improvements (Tensility VC). The steep ascent of capability that felt so dizzying in 2022 and 2023 may already be flattening into a plateau.
Meanwhile the investment flywheel keeps spinning. Hundreds of billions committed, new announcements every quarter, pressure to ship whether or not the fundamentals are solid. The gap between what is being promised and what is actually being delivered is quietly widening.
The Yes-Machine
One more problem — perhaps the most insidious.
AI agrees with me. Almost always. Push back on a conclusion and it reconsiders. Suggest an alternative framing and it enthusiastically adopts it. It is, in the most literal sense, a yes-machine — and that is not a compliment.
Good thinking requires friction. It requires something that will tell you when your argument has a hole in it, when your assumption is unfounded. What I get instead, too often, is a very articulate mirror — reflecting my own assumptions back at me, polished and elaborated. This sycophancy is a known, documented phenomenon (Medium / Goldfinger): Models trained on human feedback learn that agreement keeps the conversation moving.
But a partner who always says yes isn’t a partner. They’re a problem.
And Yet — The Platform Play
Here is where I have to be honest about a tension in everything I’ve just written.
I’ve spent the last several hundred words explaining why AI is (currently still, anyway) often too unreliable, too exhausting, and really often falling short of its promise. And I believe all of that. But here is the uncomfortable truth that keeps me up at night: It probably doesn’t matter.
Not for the platforms, anyway.
In music, we can already measure what “good enough” looks like at scale. Deezer reported in late 2025 that 50,000 fully AI-generated songs are uploaded to its platform every single day — around 34% of all new music submitted (The Guardian, November 2025). Spotify’s royalty payout pool — which grew from $1 billion in 2014 to $10 billion in 2024 — is already being gamed by AI-generated content engineered purely to capture streaming revenue (Spotify Newsroom, 2025).
And this is where the logic of the platform becomes truly dangerous. The question isn’t whether AI-generated content is good. The question is whether a platform still needs artists, labels, publishers, or distributors at all — if AI allows it to generate its own content at infinite scale, near-zero cost, and then use its own algorithms to surface that content while rendering everything else invisible.
Amazon has already shown the blueprint in books. Its own publishing imprints regularly appear at the top of Amazon bestseller lists — on Amazon’s platform, boosted by Amazon’s algorithms. The conflict of interest is structural. AI simply supercharges it.
So here is the paradox at the heart of this conversation: The same technology that frustrates me daily with its inconsistencies and half-answers is just capable enough to make human creativity economically optional — for the companies that control distribution. You don’t need a perfect tool to dismantle an industry. You just need one that’s cheap, fast, and infinitely scalable.
That’s not a technology problem. That’s a power problem. And it’s already underway.
What I Actually Think
I am not saying AI is worthless. I am saying something more uncomfortable: That we have built enormous structures — commercial, cultural, organizational — on a foundation that is quite possibly less solid than we were led to believe. And that the same technology whose reliability I question every working day is simultaneously being used to hollow out the industries that produce the human creativity it was trained on in the first place.
The pen is still not falling. But we keep being told it is. Maybe it’s time to look more carefully — and ask who, exactly, benefits from us believing otherwise.
P.S. — In the spirit of full transparency: this article was written with the assistance of AI (of course, it was …) — specifically, Anthropic’s Claude Sonnet 4.6. It helped keep the piece close to the requested word count, researched and surfaced sources (both the ones I already had in mind and several I hadn’t thought of — all of which I verified independently), and supported the drafting process, since It also knows, by now, something of how I think and how I write.
Make of that what you will. I think it actually makes the point rather well.
Sources
- METR Study (2025) — AI impact on developer productivity: metr.org
- Epoch AI — Data exhaustion and LLM scaling limits: epoch.ai
- The Guardian — AI music on streaming platforms (Nov. 2025): theguardian.com
- Music Ally / Spotify — AI music upload volumes: musically.com
- Spotify Newsroom — AI & royalty spam policy: newsroom.spotify.com
- Tensility VC — The AI scaling wall: tensilityvc.com
- ScaleMath — GPT performance variation by time of day: scalemath.com
- IEEE Spectrum — Why AI Chatbots Agree With You Even When You’re Wrong: https://spectrum.ieee.org/ai-sycophancy
- YouTube Short — Pen/gravity test with GPT-4o: youtube.com/shorts/gPthZLTnzu8
- Die Zeit — Die Arbeit ist den Menschen nie ausgegangen: https://www.zeit.de/2026/12/produktivitaet-ki-arbeit-arbeitszeit-john-maynard-keynes
Ralf Biesemeier is a pioneer