Case study · One design across several GPUs
A 10.7-billion-gate chip, split across four GPUs
The transformer inference chip, grown to 1,024 experts in each of its 16 layers and keeping its weights in memories, is 10.67 billion gates. Xerolane split it into four parts, one on each GPU of a rented machine with four NVIDIA L40S cards, and ran it as one design: 64 data sets through 82,109 clock cycles in 5 hours, every data set matching the chip's C model.
one connected design, 2.67 billion gates on each GPU.
NVIDIA L40S, 48 GB each, in one machine, one design between them.
64 data sets × 82,109 clock cycles, about $23 of rented GPU time.
data sets matching the C model, every output byte.
The chip
It is the same transformer stage as in the case study: attention, feed-forward, LayerNorm and a router, all in exact integer arithmetic, so the gates either match the C model byte for byte or they do not. Here each stage is 16 wide with one attention head, every layer holds 1,024 experts that all run every block, and 16 layers are pipelined one after another.
- Weights in memoriesEach stage keeps its weights and parameters in memories rather than flip-flops. They are loaded from a memory image before the run, and each stage copies them into place in its first clock cycles after reset.
- One design, not copiesEvery expert of every layer is wired into the chip, and the output of each layer is the input of the next. What is checked is the whole chip's answer, against the C program that defines it.
- Each data set its own tokensAll 64 data sets run the same model on different tokens, at the same time, and each is checked against its own expected output.
- Built from one stageOne stage is synthesized once, and a tool wires copies of its netlist into the full chip, writing it directly as the four parts.
How it is split
The chip is cut between layers: each GPU holds four whole layers, 2.67 billion gates. Each GPU runs its part in its own process. At every clock cycle, the signals that cross from one part to the next are exchanged, so the four parts behave as the one chip they came from.
A part could equally be something other than a GPU running gates: the same exchange at every clock is how a C model or another simulator can sit beside the chip.
The run
| Design | Gates | GPUs | Data sets | Cycles per data set | Simulation | Result |
|---|---|---|---|---|---|---|
| 1,024 experts × 16 layers, width 16 | 10.67 billion | 4 × NVIDIA L40S | 64 | 82,109 | 5 h 1 min | 64 of 64 exact |
- Checked end to endEvery output of every data set, over every clock cycle, was compared with the C model's expected output.
- CostThe four cards rent for $4.50 an hour; the simulation took 5 hours.
- The largest design on one GPUA single NVIDIA B200 ran 10.03 billion gates, 1,200 independent copies of one block. This chip is about the same size, but one connected design, with signals crossing between GPUs every cycle. Every design past 2 billion gates →
- Scale by adding GPUsEach GPU added holds more layers, so a design is no longer limited to what one card can hold.
Scaling two ways
A 2.41-billion-gate chip on NVIDIA L40S GPUs. Four GPUs give either four times the data sets, or each data set about four times sooner.
| Run | GPUs | Data sets | Per clock cycle | 20,000 cycles |
|---|---|---|---|---|
| Whole chip | 1 × L40S | 64 | 114 ms | 38 min |
| A copy on each GPU | 4 × L40S | 256 | 114 ms | 38 min |
| Chip split across four | 4 × L40S | 64 | 28.8 ms | 9.6 min, 3.97× faster |
- ExactThe 256 outputs of the four-copy run are bit-for-bit identical to the one-GPU run.
- Splitting is nearly freeEach GPU holds about 603 million gates and still runs at the speed of one GPU alone. The split adds 0.005% gates; 12,000 to 14,000 signals cross between GPUs each cycle.
Next: 20 billion gates on eight GPUs
The same chip with 2,048 experts a layer, about 20 billion gates, split across eight GPUs and checked the same way, against its C model on every data set.