Disaggregated Inference From The Ground Up

Skip to main content

Go to homepage

General Compute Selects Cerebras to Bring Ultra-Fast Inference to Agentic Coding >>

Oct 01 2026

Disaggregated Inference From The Ground Up

Isaac Tai Zhenwei Gao

Increasing inference throughput traditionally means adding hardware or serving more requests concurrently. But pushing concurrency can slow token generation for each user.

By using a technique called disaggregation, we've increased throughput by 5× in early results with the same number of Cerebras systems and no loss in token generation speeds.

At Cerebras, we’re leading the development of heterogeneous disaggregation, which is combining multiple types of chips in one inference system. By assigning different hardware to the segments of inference that are memory- or compute-bound, we unlock gains beyond homogeneous disaggregation.

This is the first installment in a series on this topic. If you’ve heard the term “disaggregation” or the claim that “prefill is compute-bound and decode is memory-bound” and wondered what either actually means, this article is for you.

We’ll build this intuition from the ground up:

• how accelerators balance compute and memory movement
• what the different stages of inference are
• why they place different demands on hardware
• how disaggregation improves the system as a whole

How Accelerators Work

An accelerator is hardware built to speed up particular kinds of computation. GPUs are one example. We can think of accelerators as two parts working together to execute a program.

• Compute units perform the arithmetic themselves—operations such as addition, multiplication, and bitwise logic.
• Memory stores data. These are typically user inputs, model weights, intermediate values, and results.

GPU

MemoryStores inputs, model weights, intermediate values, and results.Memory

bandwidthData moved

per second(GB/s or TB/s)

ComputeEach square is a compute unit.They perform arithmetic in parallel.

Arithmetic only happens in an accelerator’s compute units, not in memory. So, data movement is necessary for every computation. To use an accelerator efficiently, engineers try to get as much arithmetic as possible from each transfer: _once data has been moved from memory, how many calculations can we perform before moving more?_

Arithmetic intensity is the measure of that efficiency. It's the amount of computational "work" for the amount of data moved. More precisely, the number of floating-point operations divided by the number of bytes transferred.

Arithmetic intensity equals FLOPs divided by bytes transferred.Arithmetic intensity =FLOPsBytes transferred

Arithmetic intensity tells us how much calculation we do for each byte moved; it does not directly tell us how fast an operation runs. When the processor spends much of its time waiting for data, memory bandwidth matters most. When it stays busy calculating, compute capacity matters more.

Grokking arithmetic intensity is key to understanding an LLM's different computational stages. Let's dive into some real examples.

Adding Two Matrices

Consider a simple matrix addition. To perform it on an accelerator, we need to read two matrices from memory, do the addition in the compute cores, and save the result back.

Let's do a real example.

Here, we load two numbers from memory (2 bytes each), execute a single FLOP to add them, then write the 2-byte result back to memory. This gives us 1 FLOP over 6 transferred bytes, for an arithmetic intensity of 0.167 FLOPs per byte.

Arithmetic intensity: 1 FLOPs divided by 6 bytes transferred: 4 bytes loaded into compute and 2 bytes saved to memory.1 FLOPs6 Bytes transferred = 0.167 FLOP/byte

Memory

213352421135132413135241413524241352

321524142352213135352413135241413524

Load

4 bytes

Save

2 bytes

Compute

534876563487345548487654548765654876

−+

Each value uses 2 bytes. Each addition reads two values and writes one result: 1 FLOP per 6 bytes. Hover or tap any cell to trace its matching inputs and output. Use arrow keys within each matrix; Escape clears the selection.

Notice that as you increase the matrix size with the slider, both the calculation count and the bytes transferred grow in proportion. So in this example, although the total number of FLOPs increases for larger matrices, the relative arithmetic intensity stays the same.

Matrix Multiplication

Let's examine a different example. Matrix multiplication follows the same basic path: load two input matrices from memory, multiply and add their values in the compute units, then save the result matrix back to memory.

But unlike addition, each output value is built from an entire row of the first matrix and an entire column of the second. Hover over an output value to see the input values that contribute to it:

Arithmetic intensity: 45 FLOPs divided by 54 Bytes transferred45 FLOPs54 Bytes transferred = 0.833 FLOP/byte

Memory

213352421135132413135241413524241352

321524142352213135352413135241413524

Load

36 bytes

Save

18 bytes

Compute

131113161825161711272125101613162320161922193235191515262233122113232721

−+

Each value uses 2 bytes. We count one read per input and one write per result. For matrices with N rows and columns, each result takes N multiplications and N − 1 additions. Hover or tap any input to see affected outputs, or an output to see contributing inputs. Use arrow keys within each matrix; Escape clears the selection.

Notice what changes as you increase the matrix size. The accelerator still loads each input value once and writes each result once, but every loaded value can now contribute to more output values. The amount of data moved grows with the number of matrix entries, but the amount of arithmetic grows _faster_.

With matrix multiplications, we can increase arithmetic intensity even if only one of the input matrices changes. In the next example, we'll only make the second matrix interactive. Try the slider once more:

Arithmetic intensity: 15 FLOPs divided by 30 Bytes transferred15 FLOPs30 Bytes transferred = 0.500 FLOP/byte

Memory

213352421135132413135241413524241352

321524142352213135352413135241413524

Load

24 bytes...