Is AI overhyped? If all you have seen is a free chatbot, you are judging the weakest tier of the technology, and you are more likely to underrate it than to overrate it. AI researcher Andrej Karpathy, a founding member of OpenAI, estimates that only about 20 million people use frontier models professionally for code and maths, where they watch agents finish projects that used to take weeks or months from a single prompt. He puts executives explicitly in the group that knows AI second-hand. Among the advanced users, the bottleneck is no longer writing but checking. For a small or mid-sized company, that means two things: judge AI on a real task with a real deadline, and ask how any software an agent wrote was checked.
This piece is based on Karpathy's post on X from 11 October 2026, in which he updates a post of his from April. He calls the numbers napkin math, and there is no survey behind them, so we treat them as an experienced observer's estimate rather than market data. The model of an invite flow further down was written for this article and checked with the TLC model checker. Status: 15 October 2026.
Karpathy's funnel at a glance
| Tier, by Karpathy's estimate | What people there see |
|---|---|
| About 6 billion, roughly 75% of the world | Barely any contact with language models |
| 1 to 2 billion, roughly 20% | Free chatbots used now and then as a better search box or writing aid |
| About 20 million, roughly 0.2% | Agents completing large code and maths projects that used to take weeks or months |
| About 5,000 people inside the AI labs | Internal systems where thousands of agents work for weeks on large software projects |
The distance between the tiers is growing, Karpathy writes. What used to be two groups talking past each other is now a sharp funnel, and the world outside the labs has only seen the preview.
Why executives usually sit in the second tier
Karpathy places many professionals outside code and maths in the second tier, executives included. They spend much of their day talking to people. They understand what is happening intellectually, but second-hand. That is a description of the job, not a criticism.
He also names what drives the funnel, and it explains why a free chatbot makes a poor test. The payoff grows with the ambition, size and time horizon of a task. A question whose answer fits in a paragraph barely stresses the system. Access matters too: free tier, paid subscription or a lab's internal systems. Ask the free version a short question and you are testing the smallest task with the weakest tool.
Capability is also uneven. According to Karpathy, models are strongest where work is digital, verifiable and economically valuable, because that is what they are trained on with verifiable rewards. Our practical reading: you will see the biggest jumps first in software, data analysis and rule-based documents, and later in negotiations or customer conversations.
A verdict like “overhyped” therefore comes from a test that cannot show the difference. It gets expensive when decisions rest on it. An automation project is turned down because the chatbot once talked nonsense, or a proposal is signed that nobody in the company can assess. Both mistakes share a cause: nobody in the company has watched the technology do real work.
Letting an agent write is not the same as being able to check
The third tier is not one block either. Some users let the agent write code and check whether the demo runs. Others can explain why the result is correct. The first group grows fast, the second much more slowly. That split is our reading. Karpathy himself writes that human review and comprehension are starting to fall behind.
When an agent writes in an afternoon what used to take a month, writing stops being the bottleneck. Checking becomes the job, and few people have trained for it. For you as a client, this changes the most important question to put to a supplier. Whether they use AI will soon go without saying. What matters is how they check what the AI wrote.
Example: two clicks on the same invite link
Bugs that no demo shows tend to appear where two things happen at once. Take a closed space, such as a customer portal or a team workspace. Whoever enters first becomes the owner, and everyone else needs a one-time invite code. The obvious implementation reads the current state first and writes the result afterwards.
We described that implementation and a locked variant in TLA+, Leslie Lamport's specification language. The TLC model checker went through every possible ordering for three people and one code.
| Variant | TLC result |
|---|---|
| No lock: read and write as two steps | 81 reachable states. After 4 steps there are two owners; after 6 steps one code has let two people in |
| With a lock: check and write as one step | 16 reachable states, and both rules hold in every one |
The first bug takes four steps to tell. Person A and person B open the link at almost the same moment, and both read that the space is empty. Then A is saved as owner, and so is B, because B read the same empty state. In the second bug, two invitees both read the code as unused before either of them spends it. In a demo, one person clicks once and neither case ever happens. In production, it takes two colleagues opening the same link in the same second.
The core of the model, shortened:
\* No lock: read the state and remember it
Read(u) == /\ pc[u] = "start"
/\ seenEmpty' = [seenEmpty EXCEPT ![u] = (members = {})]
/\ seenUnused' = [seenUnused EXCEPT ![u] = ~codeUsed]
\* later, write using the remembered and possibly stale state
Write(u) == /\ pc[u] = "read"
/\ Apply(u, seenEmpty[u], seenUnused[u])
\* With a lock: check and write in one step
JoinLocked(u) == /\ pc[u] = "start"
/\ Apply(u, members = {}, ~codeUsed)
\* Rules that must hold in every state
OneFounder == Cardinality(founders) <= 1
CodeOnce == Cardinality(viaCode) <= 1
Tests check the cases somebody thought of. A model checker covers every ordering within the bounds of the model, including the ones nobody imagined. The fix itself is dull: check and write happen in one step, in the database for example through a transaction with a lock or a uniqueness constraint. An agent will happily write such a model if you ask, but hardly anyone does.
The limits belong next to the result. The model checks the design, not the finished code. Whether the code really implements the lock has to be shown by a test with concurrent requests. TLC is also only exhaustive within the size you choose, here three people and one code. For this example that is enough, because both bugs already appear with two or three people.
The 5,000 in the labs are not your problem yet
The fourth tier is the most dramatic. Karpathy describes swarms of thousands of agents working for weeks on large software projects, finding security holes and pushing mathematics forward. No mid-sized company gets access, and for your planning that is not the point.
For your decisions this year, the gap between the second and third tiers is what counts. Third-tier tools are on sale today as paid products, and many companies still do not use them for real work. What the labs use internally hints at what will later reach those products. That is our inference from Karpathy's picture of the preview, not a vendor's promise.
How to judge AI on a real task
Set aside a week and pick a task you actually care about.
- Choose a project that has been postponed for months and has real users, even if they are three colleagues.
- Work with a paid frontier model or an agent, not the free chat version. Have it write a plan first, and read the plan before anything is built.
- Review every result like the work of a new hire who never doubts themselves.
- Put the result in front of people who use it. A demo video does not count.
- Where two things can happen at once, such as bookings, invitations, stock levels or invoice numbers, have the flow modelled first and coded second.
If you do not write code, the same rule applies with less ceremony: a real task with a real deadline, judged by the result rather than by how the conversation felt. When you commission software, ask the supplier two questions. How do you check what the agent wrote? Which flows in our system can happen at the same time, and how are they protected? How to set acceptance criteria before signing is shown in our AI chatbot requirements document.
A manageable task for a first attempt of your own is Karpathy's LLM wiki, where an agent turns your sources into a wiki. How that works with Obsidian, and what our test showed, is in Obsidian second brain with AI.
Frequently asked questions
Is AI overhyped?
It depends on which AI is being judged. A free chatbot answering a short question shows little of what paid frontier models do as agents on large tasks. AI is overrated mainly where its output is accepted without checking.
What is the difference between an AI agent and a chatbot?
A chatbot answers one message at a time. An agent gets a goal, plans steps, uses tools such as files, code or APIs, and keeps working until the task is done or it needs to ask something. How an agent is put together is covered in our article on the five layers of an AI agent.
Is the free version of ChatGPT enough for a business?
For occasional help with wording, often yes. For larger tasks it lacks the stronger models and agent features of the paid tiers. Once customer data is involved, you also need contractual data protection terms, as described in our article on running AI in the EU.
How do you review code that AI wrote?
The same way as human code, only more consistently: read the plan before it is built, review every change, run automated tests and model concurrent flows up front. Tests cover the cases you thought of, a model also covers the orderings nobody thought of.
Where do Karpathy's numbers come from?
From his own napkin math; there is no study behind them. The orders of magnitude work as a way to think about the situation, not as market data for a business plan.
If a task has been sitting on your list for months, we can look at it together and work out whether an agent can take it on and how the result gets checked before it reaches your customers. Bring the task, the systems involved and a list of the flows that can happen at the same time. How we build AI solutions with language models is described on our service page.