Announcements count chips because chips are countable. But training a frontier model is not ten thousand computers doing ten thousand jobs - it is ten thousand computers pretending, at great expense, to be one. The pretence is manufactured by the network, and the network is where modern clusters are actually won or lost.
Why training is a conversation
Large-scale training splits a model and its data across every accelerator in the building, and after each tiny step of learning, all of them must exchange and reconcile results before anyone may proceed. The consequence is brutal: the whole cluster advances at the pace of its slowest, chattiest link. Double your chips with an inadequate fabric and you have bought idle silicon - the machines finish their arithmetic and wait, in synchronised silence, for the network. This is why serious operators speak of interconnect topology with the reverence outsiders reserve for chip counts, and why “GPUs” and “GPUs that can train together” are different quantities.
Two tiers of wiring
Cluster fabrics come in two tiers. Inside a server or rack, proprietary short-range links let a handful of accelerators share memory at staggering speed, forming the tight islands where the heaviest traffic stays. Between islands stretches the datacentre fabric - specialised lossless networking or heavily engineered Ethernet - whose bandwidth, latency and, above all, topology decide how many islands can cooperate before coordination eats the gains. The engineering art is keeping communication local: sharding models so the loudest conversations never leave a rack. When a lab claims a cluster of some impressive size, the load-bearing question is what fraction of it can actually train one model - the answer lives in the fabric diagram nobody publishes.
Failure is a network property too
At ten-thousand-device scale, something is always broken - a link flapping, an optic dying, a switch rebooting - and a synchronised workload turns any single failure into everyone’s stall. Mature operators engineer for it the way airlines do: redundant paths, straggler detection, checkpointing tuned so a fault costs minutes rather than days. Reliability engineering of this kind is invisible in press releases and decisive in economics; it is a large share of why identical hardware yields wildly different real throughput in different hands.
The reading rule
So translate the next cluster announcement. Chip count is procurement. The number that describes a computer is sustained throughput on a real training run - and between the two sit the fabric, the topology and the failure engineering that no photo of racks can show. When a team volunteers utilisation figures and largest-single-job size, they are showing you the network. When they volunteer aisles, they are showing you a warehouse.
Reading a fabric claim
Cluster networking has its own marketing dialect, translatable with four questions. Bandwidth per accelerator, not per switch - aggregate figures flatter; what each chip can actually push to a distant peer is the binding number. Topology - a fat-tree that gives full bandwidth between any two points costs far more than an oversubscribed design that shares it, and “up to” in a spec sheet usually marks the difference. Largest single job - the honest capacity question is the biggest model the fabric has trained as one workload, not the sum of chips in the building. And the software - collective-communication libraries, schedulers and routing tuned to the topology are where identical hardware diverges by double-digit percentages, which is why the serious labs publish networking papers and the rest publish photographs. A fabric described in those four terms is an engineering claim; described in aggregate petabits, it is a brochure.
THE UTILISATION TRUTH SERUM
Model FLOPs Utilisation - the fraction of a cluster’s theoretical arithmetic a training run actually achieves - is the metric that exposes fabric quality, and published large-run figures habitually land in the 30-50% band. Half the silicon bill, in other words, is routinely spent waiting on communication, stragglers and restarts. A cluster pitch quoting peak FLOPs without an MFU is quoting the speedometer’s top number.
THE NUMBERS THAT MATTER
- Per-accelerator bandwidth to a distant peer - not the switch aggregate.
- MFU on the largest real run - 30-50% is the published norm; claims above it deserve methods.
- Largest single job completed - the only honest capacity statement a fabric can make.
