Pipeline -- How to Make CPUs Faster

Without increasing clock frequency, can CPUs still become faster? The answer is yes. Pipeline technology is one of the most important methods. This lecture uses Python code and Gantt-chart-style visual output to show how pipelining greatly improves instruction execution efficiency.

Analogy from daily life: car wash assembly line

Imagine you run a car wash. There are two ways to operate:

Method 1: Single-person all-in-one (no pipeline)

One worker handles all washing steps for one car from start to finish—rinse, foam, scrub, dry. Only after finishing one car can they start the next. Each car takes 40 minutes, so only 1.5 cars can be washed per hour.

Method 2: Pipeline division of labor (with pipeline)

Four workers each handle one step—rinse, foam, scrub, dry. After the first car enters the rinse stage, the rinser can start rinsing the second car, while the foamer is foaming the first car. All four people are busy at the same time, and a car is completed every 10 minutes.

The second method does not make any single step faster (rinsing still takes 10 minutes), but by overlapping the steps of different cars, the overall output rate is greatly improved.

This is the core idea of the CPU pipeline: overlapping the execution of different stages of different instructions.

Serial execution vs. pipelined execution

The problem of serial execution

Reviewing Lecture 12, each instruction goes through four stages: Fetch (IF), Decode (ID), Execute (EX), Write-back (WB).

InSerial execution (no pipeline)In this mode, the CPU must fully execute all four stages of the current instruction before it can start fetching the next instruction.

周期:  1    2    3    4    5    6    7    8    9    10   11   12
I1:   [IF] [ID] [EX] [WB]
I2:                      [IF] [ID] [EX] [WB]
I3:                                          [IF] [ID] [EX] [WB]

3 条指令 = 12 个周期

Advantages of pipelined execution

InPipeline executionIn this mode, once instruction I1 completes the fetch stage (IF), the CPU's fetch unit becomes idle. Rather than letting it sit idle, you can immediately start fetching the next instruction I2.

Similarly, when I1 completes decode (ID) and enters the execute stage, the decode unit begins processing I2, while the fetch unit begins processing I3.

周期:  1    2    3    4    5    6    7
I1:   [IF] [ID] [EX] [WB]
I2:        [IF] [ID] [EX] [WB]
I3:             [IF] [ID] [EX] [WB]

3 条指令 = 6 个周期（比串行快了一倍！）

Note the key numbers:

• No pipeline: n instructions = 4n cycles
• With pipeline: n instructions = 4 + (n-1) = n + 3 cycles (i.e., after the pipeline is filled, one instruction completes each cycle)

When n is large, the speedup from pipelining approaches 4 (the number of pipeline stages).

Pipelining does not make a single instruction execute faster (each instruction still needs 4 cycles), but it greatly improves "throughput"—the number of instructions completed per unit time.

Pipeline "hazard" problem

Pipelining is good, but it does not come without cost. There are three typical "hazards" that can stall the pipeline:

Modern CPUs use various techniques to mitigate these hazards:

• Data Bypass (Data Forwarding): Without waiting for the result to be written back to the register, directly "steal" the data from the ALU output for the next instruction to use
• Branch PredictionBranch prediction: guess the branch direction and fetch ahead. If guessed correctly, you gain; if incorrectly, flush the pipeline and start over.
• Out-of-Order ExecutionOut-of-order execution: do not follow program order; instead, "execute whichever instruction's data is ready first."

Interactive demo: Pipeline Simulator (Gantt chart style)

The Python code below simulates instruction execution in both pipelined and non-pipelined modes, and prints output in Gantt chart style, allowing you to intuitively compare the differences between the two approaches.

Run the code above, and you will see:

• Gantt chart comparisonObservation: In non-pipelined mode, each instruction occupies 4 consecutive cycles, with large idle gaps between instructions. In pipelined mode, the stages are tightly arranged with almost no idle cycles.
• Speedup trendObservation: The more instructions there are, the closer the speedup approaches the pipeline depth (here, 4). But with only 1 instruction, pipelining provides no speedup at all, because the fill and drain overhead accounts for 100%.
• ThroughputObservation: After the pipeline is "filled" (starting from the 4th cycle), one instruction is completed every cycle. This is the core value of pipelining.

Interactive demo: pipeline Gantt chart comparison

Below, we use a Chart.js horizontal stacked bar chart to draw Gantt charts, directly comparingWithout pipeline (serial)andWith pipeline (4-stage)the time distribution of each stage for 4 instructions under the two modes. Each color block represents a stage (IF=Fetch, ID=Decode, EX=Execute, WB=Write-back), and the x-axis is clock cycles.

Comparison of pipeline stage depths

The "depth" of the pipeline determines the theoretical upper limit of speedup, but the deeper the pipeline, the more severe the hazard problems:

It can be seen that pipeline depth is a trade-off: too many stages make the cost of a single branch misprediction too high; too few stages prevent frequency from going high enough.

The mainstream trend in modern CPU design is“wide and shallow”— moderate pipeline depth + ultra-wide issue width (able to start executing multiple instructions per cycle), rather than blindly pursuing more stages.