Why AI Runs on GPUs

Every AI system you have used runs, almost certainly, on hardware whose ancestors were built to render explosions in video games. That accident of history is now a trillion-dollar structural fact, and understanding why the graphics chip won explains more about the industry - its costs, its bottlenecks, its geopolitics - than any single announcement.

The shape of the work

Neural networks, for all their mystique, spend their lives doing one thing: multiplying enormous grids of numbers together. A single response from a large model is billions of multiply-and-add operations, none individually interesting, all independent enough to happen at once. This is precisely the shape of work graphics demanded - computing millions of pixels, each a small identical calculation - and so the GPU evolved as the opposite of a CPU: instead of a handful of brilliant, flexible cores that excel at one complicated thing after another, thousands of simple cores that do one simple thing everywhere simultaneously. When deep learning arrived, its arithmetic slotted into that architecture as if designed for it, because in a sense it had been - researchers shaped their algorithms around the hardware that existed.

The real bottleneck is memory

The less famous half of the story is that modern AI chips are defined as much by memory as by arithmetic. A model’s weights - the billions of numbers that constitute it - must stream through the processor for every token generated, and moving data costs far more time and energy than multiplying it. This is why accelerators bolt stacks of high-bandwidth memory directly onto the package, why “memory bandwidth” appears in spec sheets next to raw compute, and why inference performance often tracks bandwidth more faithfully than FLOPs. It is also why the supply chain has two choke points, not one: advanced logic fabrication, and advanced memory - each concentrated in a handful of facilities on earth.

Software is the moat

Hardware alone does not explain the market’s concentration. A decade of tooling - drivers, kernels, libraries, and the muscle memory of every ML engineer - accreted around one vendor’s ecosystem, so that switching silicon means porting and re-optimising a living codebase, not swapping a part. Rivals’ chips are routinely competitive on paper and adopted slowly in practice; the gap is measured in software engineers, not transistors. Any claim that a new accelerator will “break the monopoly” should therefore be read at the software layer first: what runs on it today, unmodified, at what fraction of advertised speed?

What to take from this

Three durable consequences. Compute costs track a narrow supply chain, so shortages and export controls propagate straight into what models get trained. Memory bandwidth quietly bounds inference economics, so watch it in spec sheets the way you watch FLOPs. And architectural incumbency lives in software, so hardware challengers are software stories in disguise. The chip in the headline is never just a chip; it is a position in all three of these games at once.