विमर्श Vimarsh · Udyamnīti
Compute vs Memory: Who Really Rules AI?
AI chips spend most of their time waiting for memory. A Stanford paper says it can make that wait disappear. What it really shows, what it means for Nvidia and the memory makers, where India stands, and what founders should build for.
Nikhil Sharma · उद्यमनीति Udyamnīti · 9 November 2026 · 9 min read
This essay goes with my film of the same name. Every number is linked to its source. It is for education and is not investment advice.
Every time you ask ChatGPT a question, a chip that costs as much as a car spends most of its time doing nothing. It is waiting.
That wait has become the most expensive problem in technology. It decides who gets rich from AI. In October a Stanford team published a paper that claims it can make much of the wait disappear. Some readers called it the end of Nvidia’s grip on memory. I read the whole paper. Here is what it shows and what it does not.
The claim
The paper is called Next-Concept Modeling, or NCM. It comes from Chris Ré’s lab at Stanford (arXiv 2610.14162, Hazy Research). Its promise is simple to state. Instead of writing an answer one word at a time it writes in a handful of passes, broad meaning first and exact words last.
To see why that matters you need to understand what an AI chip actually spends its time doing. The answer surprises most people.
Two engines: the chef and the pantry
Think of a kitchen. Every AI chip does two jobs. Compute is the chef. It does the maths, trillions of multiplications a second. Memory is the pantry. It holds the model itself and everything said in the conversation so far.
Between them runs a corridor, and a runner carries ingredients up and down it. The speed of that runner is memory bandwidth. A brilliant chef with a slow runner spends most of the night standing still.
The memory wall
Researchers led by Amir Gholami at UC Berkeley measured how wide the gap has grown. Over two decades peak server compute grew about three times every two years. Memory bandwidth grew about 1.6 times. From 1996 to 2023 that adds up to roughly 60,000 times more compute and about 100 times more bandwidth (Gholami et al., IEEE Micro 2024).
Engineers call this the memory wall. The reason is physical. Faster maths comes from shrinking transistors, and the industry is very good at that. Moving data off the chip depends on wires, pins and distance. Those do not shrink nearly as well.
Why AI waits
Today’s chat models write one token at a time. A token is roughly a word or part of one. To write each new token the chip has to read the whole model again. A thousand-word answer means about a thousand trips down the corridor.
The model also keeps notes on everything said so far so that it does not lose the thread. Those notes are called the KV cache. They grow with every word and with every extra user. Servers try to keep the chef busy by cooking many orders together. But each order brings its own notes, and the pantry runs out of shelf space long before the chef runs out of hands.
So the chef waits. And the most valuable part of an AI server becomes the memory sitting beside the chip.
The fix: HBM
The industry’s answer so far has been to move the pantry next to the stove. High Bandwidth Memory, or HBM, stacks memory chips like the floors of a building and sits them on the same package as the processor. Thousands of tiny connections make the corridor very short.
Only three companies make HBM at scale: SK hynix, Samsung and Micron. SK hynix took the early lead. Analyst forecasts give it about half of the newest HBM supply (SemiconSam). Memory went from a cheap commodity to one of the scarcest parts in AI.
Others took a different route. Groq and Cerebras put fast memory directly on the chip so the runner barely moves. It is very quick, but on-chip memory is small and costly, so big models must be spread across many chips. Every design is a different trade between the chef and the pantry.
Nvidia
Now look at who sits on top. In the quarter that ended in July 2026 Nvidia’s data centre business earned $89.0 billion (Nvidia, Q2 FY2027 results). Its strength is not one chip. It is the whole system: the chips, the CUDA software that millions of developers already know, the networking that joins thousands of chips into one machine, and tight control of the memory around them.
The software is the real moat. A rival can build a faster chip and still find that nobody wants to rewrite twenty years of code for it.
In August Nvidia went further and announced its own HBM design, NVHBM. It says the design gives up to 30 per cent more bandwidth per stack than the next memory standard. For now those are design claims, not results from a shipping product (Tech Times). So when someone says a paper could make memory unnecessary, they are really saying it could break the bottleneck Nvidia is building around.
Enter NCM
There is already a rival to word-by-word writing called a diffusion language model. It starts with a page of noise and refines every word at once over many rounds. It is parallel, but the paper shows quality falls fast when you cut the number of rounds, and each round still processes the whole text.
NCM works more like an artist: rough outline first, then detail. It compresses text into layers of codes. The top layer holds the broad meaning and each layer below adds finer detail down to the exact words. A model then fills in one layer at a time with every position in that layer at once. The number of passes depends on the number of layers, not on the length of the answer. Not a thousand trips. A handful.
The results are real. Against the strongest diffusion language models of the same size NCM ran 21 times faster with better quality on the LM1B benchmark, 10 times faster on OpenWebText and 31 times faster on XSum, which tests summaries of news articles. On Sudoku, a test of reasoning, it beat both word-by-word and diffusion models at every level of difficulty (arXiv 2610.14162).
Reality check
Now the fine print. The main models in the paper have about 100 million parameters. The largest test reached 725 million. The models you use every day are hundreds of times bigger. And NCM was compared with diffusion models, not with ChatGPT, Claude, Gemini or Grok.
That gap matters. Many ideas that shine at small scale behave differently when models grow a thousand times. Nobody, including the authors, knows yet whether NCM keeps its lead at frontier size. The authors list their own limits: it needs a separately trained compressor, it uses the same number of layers even for short answers, and the joins between layers can still be rough.
So this is a strong research step. It is not yet a product and it is not the end of memory. But parallel writing is already real. Inception says its diffusion models serve answers at around 1,000 tokens a second, a company claim (The Decoder).
If it works
Suppose ideas like NCM do scale. Fewer trips down the corridor mean the chef stops waiting. The limit becomes how much maths you can do each second and how much power you can feed it. The bottleneck does not disappear. It moves. That still favours chips built for dense compute, Nvidia’s included.
It could also let much smaller machines, even a good gaming graphics card, run models that today need a data centre. In time the same answer could come from a laptop or a phone. Fast, private and nearly free to run. That is the real prize inside papers like this one.
Memory does not lose its job. The model still has to live in memory. Long documents still need room. Training still moves enormous amounts of data. And when something gets cheaper people use more of it. Cheaper answers mean more answers and more total demand for both compute and memory.
If chat stays word by word, the memory makers sell the bandwidth. If it goes parallel, they sell the capacity. The pantry is never going out of business.
The crown
How long does Nvidia rule? Three forces push against it. Its biggest customers are building their own chips: Google has its TPUs and Amazon has Trainium. New model designs, like the parallel methods above, change which hardware matters most. And the hard limits are now power and supply as much as chip design.
My read: Nvidia rules for as long as it owns the bottleneck, wherever the bottleneck moves. So far it has followed it every time, from graphics to AI, from chips to whole systems and now into memory itself. The day it stops owning the bottleneck is the day the crown slips. None of this is investment advice.
Engine and road
So will memory rule instead? Compute is the engine. Memory is the road. A faster engine on a broken road goes nowhere, and a perfect road with no engine carries nothing. The winners of the next decade will be the best at moving data between the two.
- Nvidia: Compute
- SK hynix: Memory · HBM
- Samsung: Memory · HBM
- Micron: Memory · HBM
- Tata Electronics: India’s fab
And India?
In February 2026 Micron’s plant at Sanand in Gujarat began shipping India’s first made-in-India memory modules. It does not make memory wafers. It assembles and tests chips made elsewhere (IBS Electronics). The pantry now has a door in India.
The chef is coming too. Tata Electronics and Taiwan’s PSMC are building a ₹91,000 crore fab, about $11 billion, at Dholera (Outlook Business). The government once expected first chips by the end of 2026. In July 2026 the IT minister said commercial operations are now expected in mid-2028 (Angel One, citing Bloomberg). The film uses the earlier estimate. This essay uses the newer one.
And the IndiaAI Mission has put more than 38,000 GPUs on a shared portal for Indian startups and researchers (Devdiscourse).
None of this is HBM. India does not make the scarcest part yet. But if the cost of intelligence keeps falling, India does not need to own the most expensive kitchen. It needs to cook for a billion people. That is the opportunity, and it plays to what India already does well: software, services and scale.
For founders
The lesson is simple. The cost of intelligence keeps falling, from the data centre down to the laptop on your desk. A capable model already runs on an ordinary Mac with the internet switched off. Two years ago that needed a data centre. Build your business for where that cost is going, not for where it is today.
What founders should do
1. Price for the cost curve, not today’s bill. Assume the cost of each answer falls every year. Design margins that improve as it falls and avoid long commitments locked at today’s prices.
2. Start small and route up. Give each task the smallest model that does it well, on device or on your own machine where privacy and speed matter. Send only the hard cases to the biggest models.
3. Measure cost per task and keep memory lean. Long context fills the pantry. Trim prompts, cache what repeats and batch what can wait. The cheapest token is the one you never send.
Download the Founder’s Guide to AI Costs (PDF)
Questions people ask
What is the memory wall?
The growing gap between how fast chips can compute and how fast memory can feed them. From 1996 to 2023 peak server compute grew about 60,000 times while memory bandwidth grew about 100 times (Gholami et al., IEEE Micro 2024). AI chips that write one word at a time spend much of their time waiting for data.
What is NCM?
Next-Concept Modeling is a Stanford method published in October 2026 (arXiv 2610.14162). It compresses text into layers of codes from broad meaning to exact words and fills in one layer at a time with every position in parallel. The number of passes depends on the number of layers, not the length of the answer.
Will NCM hurt Nvidia?
Not on today’s evidence. NCM was tested at about 100 million to 725 million parameters and compared with diffusion models, not chat assistants. If parallel writing scales, the bottleneck moves from memory to compute and power, which still suits chips built for dense compute. Nvidia leads for as long as it owns the bottleneck. This is not investment advice.
Who wins, compute or memory?
Both, and the memory makers win either way. If chat stays word by word they sell bandwidth. If it goes parallel they sell capacity for bigger models and longer context. Cheaper answers also mean more answers and more total demand.
Where does India stand in AI chips?
Early. Micron’s Sanand plant shipped the first made-in-India memory modules in February 2026 (assembly and test). Tata Electronics and PSMC are building a ₹91,000 crore fab at Dholera, with commercial operations now expected in mid-2028. The IndiaAI Mission offers more than 38,000 GPUs on a shared portal. India does not make HBM yet.
The film, word for word
Full transcript
0:00 The claim
Every time you ask ChatGPT a question, a chip that costs as much as a car spends most of its time doing nothing. Waiting. That wait has become the most expensive problem in technology, and it decides who gets rich from AI. This month, a Stanford paper claimed it could make that wait disappear. Some called it the end of Nvidia's memory chips. I read the paper. All of it. Here is what it really shows. And stay to the end, because the real winner is not who you think.
0:32 Two engines
Start with a kitchen. Every AI chip does two jobs. Compute is the chef. It does the maths, trillions of multiplications every second. Memory is the pantry. It holds the model itself, and everything said in the conversation so far. Between them runs a corridor, and a runner carries ingredients up and down it. The speed of that runner is called memory bandwidth. Here is the problem. A brilliant chef with a slow runner spends most of the night standing still. Waiting for the next tray.
1:05 The memory wall
Researchers at UC Berkeley measured how bad this has become. Over two decades, peak server compute grew about three times every two years. Memory bandwidth grew about one point six times. Add that up from nineteen ninety-six to twenty twenty-three. Compute rose roughly sixty thousand times. Bandwidth rose about a hundred. The chef became sixty thousand times faster. The runner, a hundred times. Engineers call this the memory wall. And AI ran straight into it. Why does this happen? Making the maths faster is a matter of shrinking transistors, and the industry is very good at that. Moving data off the chip is a matter of wires, pins and distance. Those do not shrink nearly as well. Physics favours the chef.
1:54 Why AI waits
Here is why. Today's chat models write one token at a time. A token is roughly a word, or part of one. To write each new word, the chip has to read the entire model again. A thousand-word answer means about a thousand trips down that corridor. And the model keeps notes on everything said so far, so it does not lose the thread. Those notes are called the KV cache. They grow with every word, and with every extra user. Servers try to keep the chef busy by serving many people at once, cooking many orders together. But every extra user brings their own notes, and the pantry runs out of shelf space long before the chef runs out of hands. So the chef waits. The pantry fills up. And the most valuable part of an AI server becomes the memory sitting next to the chip.
2:46 The fix: HBM
The industry's answer so far has been simple. Move the pantry next to the stove. High Bandwidth Memory, or HBM, stacks memory chips on top of each other like floors of a building, and sits them on the same package as the processor. Thousands of tiny connections, a very short corridor. Only three companies make it at scale. SK hynix, Samsung and Micron. SK hynix took the early lead, and analyst forecasts give it about half of the newest HBM supply. Memory went from a cheap commodity to one of the scarcest parts in AI. Some companies took a different route. Groq and Cerebras put fast memory directly on the chip itself, so the runner barely has to move. It is very quick, but on-chip memory is small and expensive, so big models must be spread across many chips. Every design is a different trade between the chef and the pantry.
3:43 Nvidia
Now look at who sits on top of all this. In the quarter that ended in July, Nvidia's data centre business made eighty-nine billion dollars. More than double a year earlier. Nvidia's strength is not one chip. It is the whole system. The chips. The CUDA software that millions of developers already know. The networking that joins thousands of chips into one machine. And tight control of the memory around them. That software is the real moat. Twenty years of tools, libraries and habits. A rival can build a faster chip, and still find that nobody wants to rewrite their code for it. In August it went further. It announced its own HBM design, called NVHBM. Nvidia says it gives up to thirty per cent more bandwidth per stack than the next memory standard. For now, those are design claims, not yet tested in a shipping product. So when someone says a paper could make memory unnecessary, they are really saying it could break the bottleneck Nvidia is building around.
4:49 Enter NCM
Here is the paper. It is called Next-Concept Modeling, or NCM, from a Stanford team in Chris Ré's lab, published in October. To see why that matters, you need one more idea. There is already a rival to word-by-word writing, called a diffusion language model. It starts with a page of noise and refines every word at once, over many rounds. It is parallel, but the paper points out that quality drops fast when you cut the number of rounds, and every round still processes the whole text. NCM does something different. It works like an artist. Rough outline first. Then detail. First it compresses text into layers of codes. The top layer holds the broad meaning. Each layer below adds finer detail, down to the exact words. Then a model fills in one layer at a time, and every position in that layer at once. So the number of passes depends on the number of layers, not on the length of the answer. Not a thousand trips. A handful. The results are real. Against the strongest diffusion language models of the same size, it ran twenty-one times faster with better quality on one benchmark. Ten times faster on another. Thirty-one times faster on summarising news articles. And on Sudoku puzzles, a test of reasoning, it beat both word-by-word models and diffusion models at every level of difficulty.
6:18 Reality check
Now the fine print. The main models in this paper have about one hundred million parameters. The largest test reached seven hundred and twenty-five million. The models you use every day are hundreds of times bigger. And NCM was compared with diffusion models, not with today's leading chat assistants. That gap matters. Many ideas that shine at small scale behave differently when models grow a thousand times. Nobody, including the authors, knows yet whether NCM keeps its lead at the size of a frontier model. The authors are clear about their own limits. It needs a separately trained compressor. It uses the same number of layers even for short answers. And the joins between layers can still be rough. So this is a strong research step. It is not yet a product. And it is not the end of memory. But the idea behind it, writing many words in parallel, is already real. Diffusion models from startups like Inception say they serve answers at around a thousand tokens a second.
7:22 If it works
So suppose ideas like NCM do scale. What changes? Fewer trips down the corridor means the chef stops waiting. The limit becomes how much maths you can do every second, and how much power you can feed it. The bottleneck does not disappear. It moves. That helps chips built for dense compute, and that still includes Nvidia. It could also let much smaller machines, even a good gaming graphics card, run models that today need a data centre. Think about what that means. The same answer that today needs a rack of servers could, in time, come from a laptop or a phone. Fast, private and nearly free to run. That is the real prize hiding inside papers like this one. Memory does not lose its job. The model still has to live in memory. Long documents still need room. And training still moves enormous amounts of data. And there is one more rule. When something gets cheaper, people use more of it. Cheaper answers mean more answers, and more total demand for both compute and memory. Which means the memory makers win either way. If chat stays word by word, they sell the bandwidth. If it goes parallel, they sell the capacity for bigger models and longer memories. The pantry is never going out of business.
8:41 The crown
So how long does Nvidia rule? Three forces push against it. First, its biggest customers are building their own chips. Google has its TPUs. Amazon has Trainium. Second, new model designs, like the parallel methods we just saw, change what hardware matters most. Third, the hard limits are now power and supply, not just chip design. My read is this. Nvidia rules for as long as it owns the bottleneck, wherever that bottleneck moves. So far it has followed it every time. From graphics to AI. From chips to whole systems. And now into memory itself. The day it stops owning the bottleneck is the day the crown slips. And please, treat none of this as investment advice.
9:29 Engine and road
So will memory rule instead? Think of it this way. Compute is the engine. Memory is the road. A faster engine on a broken road goes nowhere. A perfect road with no engine carries nothing. The winners of the next decade will not be the best at one of them. They will be the best at moving data between them.
9:51 And India?
So where does India fit in this race? In February, Micron opened its plant in Sanand, Gujarat. It does not make memory wafers. It assembles and tests memory chips made elsewhere. But India's first made-in-India memory modules shipped from there. The pantry now has a door in India. The chef is coming too. Tata and Taiwan's PSMC are building an eleven-billion-dollar chip fab at Dholera, and the government expects its first chips around the end of this year. And the IndiaAI Mission has put more than thirty-eight thousand GPUs on a shared portal for Indian startups. None of this is HBM yet. India does not make the scarcest part. But if the cost of intelligence keeps falling, India does not need to own the most expensive kitchen. It needs to cook for a billion people. That is the opportunity.
10:46 For founders
For founders, the lesson is simple. The cost of intelligence keeps falling, from the data centre all the way down to the laptop on your desk. Build your business for where that cost is going, not for where it is today. You can already see it. A capable model now runs on an ordinary Mac with the internet switched off. Two years ago that needed a data centre. The model on your laptop keeps getting better every few months. Plan for that. So, compute or memory? Tell me in the comments which side you would bet on, and why. Every source for every number is in the description. I am Nikhil Sharma. See you in the next one.
Compute or memory? Watch the film and tell me in the comments which side you would bet on.
Sources, checked 9 November 2026. The paper: Kumbong, Khare, Mlauzi, Woen, Ding, Buchanan, Arora, Mirhoseini and Ré, “Next-Concept Modeling”, arXiv 2610.14162 and the Hazy Research blog. Memory wall: Gholami et al., IEEE Micro 2024. Nvidia: Q2 FY2027 results. NVHBM: Tech Times (design claims). HBM share: SemiconSam (analyst forecasts). Inception: The Decoder (company claim). India: IBS Electronics on Micron Sanand, Outlook Business on Tata Dholera, Angel One citing Bloomberg on the 2028 timeline, Devdiscourse on IndiaAI GPUs. Logos belong to their companies. Not investment advice.