Computer Organization and Design /fj^ The Hardware/Software Interface ^h±<y	
		Chapter 4
		The Processor
Introduction
CPU performance factors
■ Instruction count
> Determined by ISA and compiler
■ CPI and Cycle time
> Determined by CPU hardware
We will examine two MIPS implementations
■ A simplified version
■ A more realistic pipelined version
Simple subset, shows most aspects
■ Memory reference: lw, sw
■ Arithmetic/logical: add, sub, and, or, sit
■ Control transfer: beq, j
M 14
Chapter 4 — The Processor —
Instruction Execution
PC -> instruction memory, fetch instruction Register numbers -> register file, read registers Depending on instruction class
■ Use ALU to calculate
Arithmetic result
> Memory address for load/store
> Branch target address
■ Access data memory for load/store
■ PC <- target address or PC + 4
M 14
Chapter 4 — The Processor — 3
CPU Overview
Add
pc pH Address Instruction
Instruction memory
Add.
Data
Register #
Registers
Register #
Register #
Address
Data memory
Data
M 14
Chapter 4 — The Processor — 4
Multiplexers
PC     Address Instruction
Instruction memory
Add,
Can't just join wires together
■ Use multiplexers
Data
Register #
Registers
Register #
Register #
Address
Data memory
Data
4
Chapter 4 — The Processor — 5
Control
Branch
M
u
X
Add
PC
Address Instruction
Instruction memory
Add,
Data
Register #
Registers
Register #
Registers RegWrite
M
u
X
y> ALU
4
Zero
MemWrite	
Address	
Data	
memory	
Data	
MemRead	
Chapter 4 — The Processor — 6
Logic Design Basics
Information encoded in binary
■ Low voltage = 0, High voltage = 1
■ One wire per bit
■ Multi-bit data encoded on multi-wire buses
Combinational element
■ Operate on data
■ Output is a function of input
State (sequential) elements
■ Store information
M 14
Chapter 4 — The Processor —
Combinational Elements
AND-gate . Y = A & B
a — b —
y
Multiplexer
■ Y = S ? 11 : I0
I0		
11		u
		
		s
y
Adder .Y = A+ B
a
b
+
y
Arithmetic/Logic Unit
. Y = F(A, B)
a
b
SaluUy
Chapter 4 — The Processor — 8
Sequential Elements
Register: stores data in a circuit
■ Uses a clock signal to determine when to update the stored value
■ Edge-triggered: update when Clk changes from 0 to 1
M 14
Chapter 4 — The Processor — 9
Sequential Elements
Register with write control
■ Only updates on clock edge when write control input is 1
■ Used when stored value is required later
Q
Clk
Write D Q
x
X
X
Chapter 4 — The Processor — 10
Clocking Methodology
Combinational logic transforms data during clock cycles
■ Between clock edges
■ Input from state elements, output to state element
■ Longest delay determines clock period
Clock cycle
M 14
Chapter 4 — The Processor — 11
Building a Datapath
Datapath
■ Elements that process data and addresses in the CPU
■ Registers, ALUs, mux's, memories, ...
We will build a MIPS datapath incrementally
■ Refining the overview design
M 14
Chapter 4 — The Processor —
Instruction Fetch
PC
Read address
Instruction
Instruction memory
Increment by 4 for next instruction
M 14
Chapter 4 — The Processor — 13
R-Format Instructions
Read two register operands Perform arithmetic/logical operation Write register result
Register numbers
Data
a. Registers
Read	
register 1	Read
Read	data 1
register 2	
Write Re3iste's	
register	Read
Write	data 2
Data	
RegWrite
> Data
ALU operation
Zero alu alu
result
b. ALU
Chapter 4 — The Processor — 14
Load/Store Instructions
Read register operands
Calculate address using 16-bit offset
■ Use ALU, but sign-extend offset
Load: Read memory and update register Store: Write register value to memory
MemWrite
Write data
Address
Data memory
Read data
MemRead
a. Data memory unit
b. Sign extension unit
M 14
®
Chapter 4 — The Processor — 15
Branch Instructions
■ Read register operands
■ Compare operands
■ Use ALU, subtract and check Zero output
' ■ Calculate target address
■ Sign-extend displacement
■ Shift left 2 places (word displacement) - Add to PC + 4
■ Already calculated by instruction fetch
M 14
Chapter 4 — The Processor — 16
Branch Instructions
Just re-routes wires
Instruction
4
PC+4 from instruction datapath
Read	
register 1	Read
Read	data 1
register 2	
Write RegiSterS	
register	Read
Write	data 2
data	
RegWrite
16
ALU operation
Branch target
To branch control logic
Sign-bit wire replicated
Chapter 4 — The Processor
Composing the Elements
First-cut data path does an instruction in one clock cycle
■ Each datapath element can only do one function at a time
■ Hence, we need separate instruction and data memories
Use multiplexers where alternate data sources are used for different instructions
M 14
Chapter 4 — The Processor — 18
R-Type/Load/Store Datapath
Instruction
Read	
register 1	Read
Read	data 1
register 2	
Registers Write register	Read data 2
Write	
data	
ALU operation
Address
MemWrite
MemtoReg
Read data
^ Write data
Data memory
1 M u x
MemRead
Chapter 4 — The Processor — 19
Full Datapath
PCSrc
ALU Add result
PC
Read address
Instruction
Instruction memory
Read register 1
Read data 1
Read register 2
Write Registers Read register data 2
M u
X
w
4
ALU operation
Zero alu alu
result
MemWrite
Address	Read
	data
Write	Data
data memory	
MemRead
MemtoReg
M u
X
Chapter 4 — The Processor — 20
ALU Control
ALU used for
■ Load/Store: F = add
■ Branch: F = subtract
■ R-type: F depends on funct field
ALU control	Function
0000	AND
0001	OR
0010	add
0110	subtract
0111	set-on-less-than
1100	NOR
Chapter 4 — The Processor — 21
ALU Control
Assume 2-bit ALUOp derived from opcode
■ Combinational logic derives ALU control
opcode	ALUOp	Operation	funct	ALU function	ALU control
Iw	00	load word	XXXXXX	add	0010
sw	00	store word	XXXXXX	add	0010
beq	01	branch equal	XXXXXX	subtract	0110
R-type	10	add	100000	add	0010
		subtract	100010	subtract	0110
		AND	100100	AND	0000
		OR	100101	OR	0001
		set-on-less-than	101010	set-on-less-than	0111
M 14
Chapter 4 — The Processor — 22
The Main Control Unit
Control signals derived from instruction
R-type	0	rs	rt	rd shamt			funct	
	31:26          25:21        20:16 U				5:11         10:6 5:0			
Load/ Store	35 or 43	rs	rt		address			
	31:26          25:21        20:16  \ \				15:0			
Branch	4	rs	rt		\ address			
	31:26          25:21 20:16 V-v-J  V-v-' K-v-'				^              15:0 I			
	opcode	always read	read, except for load		write for R-type and load		sign-extend and add	
								
M 14
Chapter 4 — The Processor — 23
Datapath With Control
Chapter 4 — The Processor — 24
R-Type Instruction
Read address
Instruction [31-0]
Instruction memory
Instruction [25-21]
Instruction [20-16]
Instruction [15-11]
(0~\ M u
X
1
Instruction [15-0]
4
Read
register 1 Read data 1
register 2
Write register
Read data 2
Write
data Registers
Zero ALU ALU
result
Address
Read data
Write
Data
data memorV
M u
x
Chapter 4 — The Processor — 25
Load Instruction
Read address
Instruction [31-0]
Instruction memory
Instruction [25-21]
Instruction [20-16]
Instruction [15-11
(0~\ M u
X
1
Instruction [15-0]
<
Read
register 1 Read data 1
register 2
Write register
Read data 2
Write
data Registers
Zero ALU ALU
result
Address
Read data
Write
Data
data memory
M u
x
Chapter 4 — The Processor — 26
Branch-on-Equal Instruction
Read
address
Instruction [31-0]
Instruction memory
Instruction [25-21]
Instruction [20-16]
Instruction [15-11'
(0~\ M u
X
1
Instruction [15-0]
4
Read
register 1 Read data 1
register 2
Write register
Read data 2
Write
data Registers
Zero
ALU A|_u result
Address
Read data
Write
Data
data memorV
M u
x
Chapter 4 — The Processor — 27
Implementing Jumps
Jump
address
31:26 25:0
Jump uses word address Update PC with concatenation of
. Top 4 bits of old PC ■ 26-bit jump address . 00
Need an extra control signal decoded from opcode
Chapter 4 — The Processor — 28
Datapath With Jumps Added
4
Chapter 4 — The Processor — 29
Performance Issues
Longest delay determines clock period
■ Critical path: load instruction
■ Instruction memory -» register file -> ALU -» data memory    register file
Not feasible to vary period for different instructions
Violates design principle
■ Making the common case fast
We will improve performance by pipelining
M 14
Chapter 4 — The Processor — 30
Pipelining Analogy
■ Pipelined laundry: overlapping execution
■ Parallelism improves performance
Time
Task order
A B C D
Time -
Task order
A B C D
6 PM
9 10        11 12 1 2AM
□
□
■1
6 PM 7
9 10 11 12 1 2AM
□
»1
□
^1
Q
i
□
0
Four loads:
■ Speedup
= 8/3.5 = 2.3
Non-stop:
■ Speedup
= 2n/0.5n + 1.5 «4 = number of stages
Chapter 4 — The Processor — 31
MIPS Pipeline
Five stages, one step per stage
1. IF: Instruction fetch from memory
2. ID: Instruction decode & register read
3. EX: Execute operation or calculate address
4. MEM: Access memory operand
5. WB: Write result back to register
M 14
Chapter 4 — The Processor — 32
Pipeline Performance
Assume time for stages is
■ 10Ops for register read or write
■ 200ps for other stages
Compare pipelined datapath with single-cycle datapath
Instr	Instr fetch	Register read	ALU op	Memory access	Register write	Total time
Iw	200ps	100 ps	200ps	200ps	100 ps	800ps
sw	200ps	100 ps	200ps	200ps		700ps
R-format	200ps	100 ps	200ps		100 ps	600ps
beq	200ps	100 ps	200ps			500ps
M 14
Chapter 4 — The Processor — 33
Pipeline Performance
Program execution order (in instructions)
Single-cycle (Tc= 800ps)
Time
200
—i—
400
—i—
600
800
—i—
1000    1200    1400     1600 1800
lw $1, 100{$0) lw $2, 200{$0) lw $3, 300{$0)
Instruction fetch
Reg ALU
Data access
Reg
800 ps
Instruction fetch
		Data
Reg	ALU	
		access
Reg
800 ps
Instruction fetch
Program execution order (in instructions)
Pipelined (Tc= 200ps)
800 ps
Time
200
—i—
400
—i—
600
800
—i—
1000    1200 1400
lw $1, 100($0)	Instruction fetch		Reg	ALU		Data access		Reg				
lw $2, 200($0) 200 ps		Instruction fetch			Reg	ALU		Data access		Reg		
lw $3, 300($0)		200 ps		Instruction fetch			Reg	ALU		Data access		Reg
4
200 ps 200 ps 200 ps  200 ps 200 ps
Chapter 4 — The Processor
Pipeline Speedup
■ If all stages are balanced
■ i.e., all take the same time
■ Time between instructionspjpe|jned
= Time between instructionsnonpjpe|jned Number of stages
■ If not balanced, speedup is less
■ Speedup due to increased throughput
■ Latency (time for each instruction) does not decrease
M 14
Chapter 4 — The Processor — 35
Pipelining and ISA Design
MIPS ISA designed for pipelining
■ All instructions are 32-bits
Easier to fetch and decode in one cycle
■ c.f. x86: 1- to 17-byte instructions
■ Few and regular instruction formats
■ Can decode and read registers in one step
■ Load/store addressing
■ Can calculate address in 3rd stage, access memory in 4th stage
■ Alignment of memory operands
■ Memory access takes only one cycle
M 14
Chapter 4 — The Processor — 36
Hazards
Situations that prevent starting the next instruction in the next cycle
Structure hazards
■ A required resource is busy
Data hazard
■ Need to wait for previous instruction to complete its data read/write
Control hazard
■ Deciding on control action depends on previous instruction
Chapter 4 — The Processor — 37
Structure Hazards
Conflict for use of a resource
In MIPS pipeline with a single memory
■ Load/store requires data access
■ Instruction fetch would have to stall for that cycle
■ Would cause a pipeline "bubble"
Hence, pipelined datapaths require separate instruction/data memories
■ Or separate instruction/data caches
M 14
Chapter 4 — The Processor — 38
Data Hazards
An instruction depends on completion of data access by a previous instruction
- add   $s0, $t0, $tl sub   $t2, $s0, $t3
Time
200
—I—
400
add $s0, $t0, $t1
IF
sub $t2, $s0, $t3
ID
>EX
600		800	1000	1200	1400	1600
i		i	I	I	I	i *
	MEM		WB			
						
(Tbubble) (/bubble) Gobble) (/bubble) (/bubble)
XÄJ  XjdU XjsxJ XjsxJ XjsxJ
(/bubble) ("bubble) Tbubble) Tbubble) (/bubble)
XjzkJ XjsxJ XjSkJ XjoU XjzkJ
IF
ID
>EX
	MEM		WB
			
Chapter 4 — The Processor — 39
Forwarding (aka Bypassing)
■ Use result when it is computed
■ Don't wait for it to be stored in a register
■ Requires extra connections in the datapath
Program execution
order Time (in instructions)
add SsO, StO, $t1
sub $t2, $s0, St3
M 14
Chapter 4 — The Processor — 40
Load-Use Data Hazard
Can't always avoid stalls by forwarding
■ If value not computed when needed
■ Can't forward backward in time!
Program execution order Time (in instructions)
Iw $s0, 20($t1)
200        400        600        800        1000       1200 1400
IP	H
ir	
id
>EX
MEM
WB
(/bubble) (/bubble) Cbubble)\ ((bubble) ((bubble)
sub $t2, $s0, $t3
				
i	f	r	id	
				
>EX
MEM
WB
M 14
Chapter 4 — The Processor — 41
Code Scheduling to Avoid Stalls
Reorder code to avoid use of load result in the next instruction
C code for A = B + E; C = B + F;
stall
stall
lw   $tl, 0($t0) lw ($t2)^4($t0) add $t3, $tlT7^$t2 sw   $t3, 12($t0) lw ($t4X^$tO) add $t5, $tl7($t< sw   $t5, 16($t0)
13 cycles
lw $tl, 0($t0) lw (|t2^($t0) lw ($t4) Sl^tO) add $t3^\^l7X$t2. sw $t3, lZ^StO add $t5, $tl,X$t4v sw   $t5, 16($t0)
11 cycles
Chapter 4 — The Processor — 42
Control Hazards
Branch determines flow of control
■ Fetching next instruction depends on branch outcome
■ Pipeline can't always fetch correct instruction
■ Still working on ID stage of branch
In MIPS pipeline
■ Need to compare registers and compute target early in the pipeline
■ Add hardware to do it in ID stage
M 14
Chapter 4 — The Processor — 43
Stall on Branch
Wait until branch outcome determined before fetching next instruction
Program execution order (in instructions)
Time
200
400
600
800
1000    1200 1400
T
T
add $4, $5, $6 beq $1, $2, 40
orS7, $8, $9
Instruction fetch		Reg	ALU		Data access	Reg					
	Instruction			Reg	ALU	Data		Reg			
- 200 ps	fetch					access					
		(i^le^C			()Ub^(	bubb^C		bubbles		^^^^	
					Instruction fetch					Data access	
		400 ps					Reg	ALU			Reg
M 14
Chapter 4 — The Processor — 44
Branch Prediction
Longer pipelines can't readily determine branch outcome early
■ Stall penalty becomes unacceptable
Predict outcome of branch
■ Only stall if prediction is wrong
In MIPS pipeline
■ Can predict branches not taken
■ Fetch instruction after branch, with no delay
M 14
Chapter 4 — The Processor — 45
MIPS with Predict Not Taken
Prediction correct
Program execution order (in instructions)
Time
200
400
600
800
1000     1200 1400
add $4, $5, $6	Instruction fetch		Reg	ALU		Data access		Reg				
beq $1, $2, 40		Instruction			Reg	ALU		Data		Reg		
	200 ps	fetch						access				
				Instruction fetch						Data access		
Iw $3, 300($0)		200 ps					Reg	ALU				Reg
Prediction incorrect
Program execution order (in instructions)
Time
200      400      600      800     1000     1200 1400
add $4, $5, $6 beq $1, $2, 40
Instruction fetch		Reg	ALU		Data access	Reg			
	Instruction			Reg	ALU	Data		Reg	
200 ps	fetch					access			
or $7, $8, $9
C^Xble^C	bubble)(	^Xble^			C^^^leJ	)
	Instruction fetch				Data access	
400 ps			Reg	ALU		Reg
Chapter 4 — The Processor — 46
More-Realistic Branch Prediction
Static branch prediction
■ Based on typical branch behavior
■ Example: loop and if-statement branches
Predict backward branches taken
> Predict forward branches not taken
Dynamic branch prediction
■ Hardware measures actual branch behavior
> e.g., record recent history of each branch
■ Assume future behavior will continue the trend
When wrong, stall while re-fetching, and update history
M 14
Chapter 4 — The Processor — 47
Pipeline Summary
The BIG Picture
Pipelining improves performance by increasing instruction throughput
■ Executes multiple instructions in parallel
■ Each instruction has the same latency
Subject to hazards
■ Structure, data, control
Instruction set design affects complexity of pipeline implementation
Chapter 4 — The Processor — 48
MIPS Pipelined Datapath
WB: Write back
Right-to-left flow leads to hazards
Chapter 4 — The Processor — 49
Pipeline registers
Need registers between stages
■ To hold information produced in previous cycle
M 14
Chapter 4 — The Processor — 50
Pipeline Operation
Cycle-by-cycle flow of instructions through the pipelined datapath
■ "Single-clock-cycle" pipeline diagram
■ Shows pipeline usage in a single cycle Highlight resources used
■ c.f. "multi-clock-cycle" diagram
■ Graph of operation over time
We'll look at "single-clock-cycle" diagrams for load & store
M 14
Chapter 4 — The Processor — 51
IF for Load, Store,
M 14
Chapter 4 — The Processor — 52
ID for Load, Store, ...
M 14
Chapter 4 — The Processor — 53
EX for Load
Iw
Execution
Chapter 4 — The Processor — 54
MEM for Load
lw
Memory
Instruction memory
Read	
register 1	Read
	data 1
Read	
register 2	
Registers	Read
Write	data 2
register	
Write	
data	
u
	Read
Address	data
Data	
memory	
Write	
data	
J7\
Chapter 4 — The Processor — 55
WB for Load
lw
Write back
		
M		
u		PC
X		
		
Instruction memory
Wrong
register
number
Read	
register 1	Read
	data 1
Head	
r°qister 2	
Registers R6a(j	
Write	data 2
regis+jr	
Write	
data	
f Shift \ I left 2 I-*"
>Add Add. ' result
ALU result
	Read
Address	data
Data	
memory	
Write	
data	
(7\
M 14
Chapter 4 — The Processor — 56
Corrected Datapath for Load
M 14
Chapter 4 — The Processor — 57
EX for Store
sw
Execution
Chapter 4 — The Processor — 58
MEM for Store
sw
Memory
Instruction memory
Read	
register 1	Read
	data 1
Read	
register 2	
Registers	Read
Write	data 2
register	
Write	
data	
	Read
Address	data
Data	
memory	
Write	
data	
J7)
M
u -
X
Chapter 4 — The Processor — 59
WB for Store
sw
Write-back
ID/EX
EX; MEM
Address	
Instruction	
memory	
Read	
register 1	Read
	data 1
Read	
register 2	
Registers	Read
Write	data 2
register	
Write	
data	
'S f Sign- ^ \     I extend
	Read
Address	data
Data	
memory	
Write	
data	
Pi
M
u -
Chapter 4 — The Processor — 60
Multi-Cycle Pipeline Diagram
Form showing resource usage
Time {in clock cycles) -*■
CC 1       CC 2      CC 3      CC 4      CC 5      CC 6      CC 7      CC 8      CC 9
Program
execution
order
(in instructions)
lw$10, 20($1)
sub $11, $2, $3
add $12, $3, $4
lw$13, 24($1)
add $14, $5,
. I .
IM I--Weg
i-DM	
1-	r
—Regi
3aI_uL- tJdmW Mteg!
u 1
"D
IM |—I l-^Reg
DM	
	r
-pegi
dm
Chapter 4 — The Processor — 61
Multi-Cycle Pipeline Diagram
Traditional form
Time (in clock cycles) -•
CC 1       CC 2       CC 3       CC 4       CC 5       CC 6       CC 7       CC 8       CC 9
Program
execution
order
(in instructions)
lw$10, 20($1)	Instruction fetch	Instruction decode	Execution	Data access	Write back				
sub $11, $2, $3		Instruction fetch	Instruction decode	Execution	Data access	Write back			
add $12, $3, $4			Instruction fetch	Instruction decode	Execution	Data access	Write back		
lw$13, 24($1)				Instruction fetch	Instruction decode	Execution	Data access	Write back	
add $14, $5, $6					Instruction fetch	Instruction decode	Execution	Data access	Write back
Chapter 4 — The Processor — 62
Single-Cycle Pipeline Diagram
State of pipeline in a given cycle
add $14, $5, $6	Iw $13, 24 ($1)	add $12, $3, $4	sub $11, $2, $3	lw$10, 20($1)
Instruction fetch	Instruction decode	Execution	Memory	Write-back
		
M u		PC
x xü		
Instruction memory
IF/ID
Read	
register 1	Read
	data 1
Read	
register 2	
Registers	Read
Write	data 2
register	
Write	
data	
ID/EX
sum]     ( reM5
I left 2 I-
EX/MEM
MEM/WB
	Read
Address	data
Data	
memory	
Write	
data	
	
M li
Chapter 4 — The Processor — 63
Pipelined Control (Simplified)
PCSrc
MEM/WB
Chapter 4 — The Processor — 64
Pipelined Control
■ Control signals derived from instruction
■ As in single-cycle implementation
Instruction
WB
IF/ID
ID/EX
EX/MEM
MEM/WB
M 14
Chapter 4 — The Processor — 65
Pipelined Control
14
Chapter 4 — The Processor
Data Hazards in ALU Instructions
Consider this sequence:
sub $2, $1,$3 and $12,$2,$5 or $13,$6,$2 add $14,$2,$2 sw $15,100($2)
We can resolve hazards with forwarding
■ How do we detect when to forward?
M 14
Chapter 4 — The Processor —
Dependencies & Forwarding
Time (in clock cycles)--
Value of cc 1 cc 2 cc 3 cc 4 cc 5 CC 6 CC 7 CC 8 CC 9 register $2:     10 10 10 10       10/-20      -20        -20       -20 -20
Program
execution
order
Detecting the Need to Forward
Pass register numbers along pipeline
■ e.g., ID/EX.RegisterRs = register number for Rs sitting in ID/EX pipeline register
ALU operand register numbers in EX stage are given by
. ID/EX.RegisterRs, ID/EX.RegisterRt
Data hazards when
1a. EX/MEM.RegisterRd = ID/EX.RegisterRs 1b. EX/MEM.RegisterRd = ID/EX. Reg isterRt 2a. MEM/WB.RegisterRd = ID/EX.RegisterRs 2b. MEM/WB.RegisterRd = ID/EX. Reg isterRt
Fwd from EX/MEM pipeline reg
Fwd from MEM/WB pipeline reg
Chapter 4 — The Processor — 69
Detecting the Need to Forward
But only if forwarding instruction will write to a register!
. EX/MEM. Reg Write, MEM/WB. Reg Write
And only if Rd for that instruction is not $zero
. EX/MEM.RegisterRd * 0, MEM/WB.RegisterRd ± 0
M 14
Chapter 4 — The Processor — 70
Forwarding Paths
ID/EX
EX/MEM
MEM/WB
Registers
M u
x
w
f Forward A
M u
x
ForwardB
M u
x
Forwarding Y+-unit
EX/MEM.RegisterRd
M u
x
MEM/WB.RegisterRd
b. With forwarding
M 14
Chapter 4 — The Processor
Forwarding Conditions
EX hazard
. if (EX/MEM.RegWrite and (EX/MEM.RegisterRd * 0) and (EX/MEM.RegisterRd = ID/EX.RegisterRs))
ForwardA = 10
. if (EX/MEM.RegWrite and (EX/MEM.RegisterRd * 0) and (EX/MEM.RegisterRd = ID/EX.RegisterRt))
Forward B = 10
MEM hazard
. if (MEM/WB.RegWrite and (MEM/WB.RegisterRd * 0) and (MEM/WB.RegisterRd = ID/EX.RegisterRs))
ForwardA = 01
. if (MEM/WB.RegWrite and (MEM/WB.RegisterRd * 0) and (MEM/WB.RegisterRd = ID/EX.RegisterRt))
Forward B = 01
M 14
Chapter 4 — The Processor — 72
Double Data Hazard
Consider the sequence:
add $1,$1,$2 add $1,$1,$3 add $1,$1,$4
Both hazards occur
■ Want to use the most recent
Revise MEM hazard condition
■ Only fwd if EX hazard condition isn't true
M 14
Chapter 4 — The Processor — 73
Revised Forwarding Condition
MEM hazard
if (MEM/WB. Reg Write and (MEM/WB.RegisterRd 4 0) and not (EX/MEM. Reg Write and (EX/MEM.RegisterRd 4 0)
and (EX/MEM.RegisterRd = ID/EX.RegisterRs)) and (MEM/WB.RegisterRd = ID/EX.RegisterRs)) ForwardA = 01
if (MEM/WB. Reg Write and (MEM/WB.RegisterRd 4 0) and not (EX/MEM. Reg Write and (EX/MEM.RegisterRd * 0)
and (EX/MEM.RegisterRd = ID/EX.RegisterRt)) and (MEM/WB.RegisterRd = ID/EX.RegisterRt)) Forward B = 01
Chapter 4 — The Processor — 74
Datapath with Forwarding
MEM/WB
PC-
Instruction memory
Chapter 4 — The Processor — 75
Load-Use Data Hazard
Time (in clock cycles) -
CC 1        CC 2        CC 3        CC 4        CC 5        CC 6        CC 7        CC 8        CC 9
Program
execution
order
Load-Use Hazard Detection
■ Check when using instruction is decoded in ID stage
■ ALU operand register numbers in ID stage are given by
. IF/ID.RegisterRs, IF/ID.RegisterRt
■ Load-use hazard when
■ ID/EX. Mem Read and
((ID/EX.RegisterRt = IF/ID.RegisterRs) or (ID/EX.RegisterRt = IF/ID.RegisterRt))
■ If detected, stall and insert bubble
M 14
Chapter 4 — The Processor — 77
How to Stall the Pipeline
Force control values in ID/EX register toO
■ EX, MEM and WB do nop (no-operation)
Prevent update of PC and IF/ID register
■ Using instruction is decoded again
■ Following instruction is fetched again
■ 1-cycle stall allows MEM to read data for lw
■ Can subsequently forward to EX stage
M 14
Chapter 4 — The Processor — 78
Stall/Bubble in the Pipeline
Time (in clock cycles)-
CC 1       CC2      CC3      CC4      CCS      CC 6      CC 7      CC 8      CC 9      CC 10
Program
execution
order
Stall/Bubble in the Pipeline
Time (in clock cycles)-*~
CC 1       CC2      CC3      CC4      CCS      CC 6      CC 7      CC 8      CC 9      CC 10
Program
execution
order
(in instructions)
Iw $2, 20($1)
IM -
and becomes nop
and $4, $2, $5 stalled in ID
or $8, $2, $6 stalled in IF
add $9, $4, $2
Or, more accurately.
Chapter 4 — The Processor —
Datapath with Hazard Detection
Ü
CL
PC
Instruction memory
ID
Hazard detection unit
ID/EX.MemRead
Registers
IF/ID.RegisterRs
IF/ID.RegisterRt
IF/ID.RegisterRt
IF/ID.RegisterRd
ID/EX.RegisterRt
EX/IVO
M u
x
+_
M u
V
EL
RcL
Rs
M u
x
WB
M
Rt
Forwarding unit
MEM/WB
Data memory
WB
M u
X
W
Chapter 4 — The Processor — 81
Stalls and Performance
The BIG Picture
■ Stalls reduce performance
■ But are required to get correct results
■ Compiler can arrange code to avoid hazards and stalls
■ Requires knowledge of the pipeline structure
M 14
Chapter 4 — The Processor — 82
Branch Hazards
If branch outcome determined in MEM
Time (in clock cycles)
CC 1        CC 2 CC 3       CC 4        CC 5        CC 6        CC 7        CC 8        CC 9
Program
execution
order
(in instructions)
40 beq $1, $3, 28
44 and $12, $2, $5
48 or $13, $6, $2
52 add $14, $2, $2
IM--QReg__
I
■ DM —
egj_
IM —
„72 Iw $4, 50($7)
-Reg'
DM — — Reg
JReg
* IM
DM
- ^Regl
PC
3Reg
DM —
-Reg'
1
DM —
Flush these instructions (Set control values to 0)
j
I—I
Chapter 4 — The Processor — 83
Reducing Branch Delay
Move hardware to determine outcome to ID stage
■ Target address adder
■ Register comparator
Example: branch taken		
36:	sub	$10, $4, $8
40:	beq	$1,    $3, 7
44:	and	$12, $2, $5
48:	or	$13, $2, $6
52:	add	$14, $4, $2
56:	sit	$15, $6, $7
72:	■    ■ ■ lw	$4, 50($7)
M 14
Chapter 4 — The Processor — 84
Example: Branch Taken
and $12, $2, $5
IF.Flush
	
u	
X	7z
beq$1,$3, 7
sub $10, $4, $8
before<1 >
_I 44
Instruction meniory
IF/ID
Clock 3
Hazard
detection
V    unit /
44
28k
ID/EX
M u
X
	
Registers	
	
$1
$3.
WB
EX
■ J
10
M u
x
M u
EX/MEM
before<2>
S4
>ALU
tv
Forwarding unit
MEIVJ/WB
Data memory
WB
M
x
Chapter 4 — The Processor — 85
Example: Branch Taken
IF.Flush
Iw $4, 50($7)
Bubble (nop)
beq $1, $3, 7
sub $10,
before<1 >
		
M		
u		PC
	76	
7G
IR
ID
Instruction memory
w
Hazard detection unit
Clock 4
M u
X
id/ex
Registers
EX
M u
X
w
I
M u
X
EX/I^EM
WB
Forwarding unit
MEM/WB
-1'
Data memory
M u
X
Chapter 4 — The Processor — 86
Data Hazards for Branches
If a comparison register is a destination of 2nd or 3rd preceding ALU instruction
add $1, $2, $3 add $4, $5, $6
beq $1, $4, target
Can resolve using forwarding
Chapter 4 — The Processor — 87
Data Hazards for Branches
If a comparison register is a destination of preceding ALU instruction or 2nd preceding load instruction
■ Need 1 stall cycle
lw   $1, addr add $4, $5, $6
beq stalled
beq $1, $4, target
WB
EX
MEM
WB
Chapter 4 — The Processor — 88
Data Hazards for Branches
If a comparison register is a destination of immediately preceding load instruction
■ Need 2 stall cycles
lw   $1, addr if
beq stalled beq stalled beq $1, $0, target
id
if
EX
MEM
WB
id
n\
GOG
id
CP G G
id
EX
MEM
WB
Chapter 4 — The Processor — 89
Dynamic Branch Prediction
In deeper and superscalar pipelines, branch penalty is more significant
Use dynamic prediction
■ Branch prediction buffer (aka branch history table)
■ Indexed by recent branch instruction addresses
■ Stores outcome (taken/not taken)
■ To execute a branch
> Check table, expect the same outcome
> Start fetching from fall-through or target If wrong, flush pipeline and flip prediction
M 14
Chapter 4 — The Processor — 90
1-Bit Predictor: Shortcoming
Inner loop branches mispredicted twice!
outer:
inner:
beq ..., ..., inner
beq ..., ..., outer
Mispredict as taken on last iteration of inner loop
Then mispredict as not taken on first iteration of inner loop next time around
4
Chapter 4 — The Processor — 91
2-Bit Predictor
Only change prediction on two successive mispredictions
Calculating the Branch Target
Even with predictor, still need to calculate the target address
■ 1-cycle penalty for a taken branch
Branch target buffer
■ Cache of target addresses
■ Indexed by PC when instruction fetched
If hit and instruction is branch predicted taken, can fetch target immediately
M 14
Chapter 4 — The Processor — 93
Exceptions and Interrupts
■ "Unexpected" events requiring change in flow of control
■ Different ISAs use the terms differently
■ Exception
■ Arises within the CPU
■ e.g., undefined opcode, overflow, syscall,...
■ Interrupt
■ From an external I/O controller
■ Dealing with them without sacrificing performance is hard
M 14
Chapter 4 — The Processor —
Handling Exceptions
In MIPS, exceptions managed by a System Control Coprocessor (CPO)
Save PC of offending (or interrupted) instruction
■ In MIPS: Exception Program Counter (EPC)
Save indication of the problem
■ In MIPS: Cause register
■ We'll assume 1-bit
0 for undefined opcode, 1 for overflow
Jump to handler at 8000 00180
M 14
Chapter 4 — The Processor — 95
An Alternate Mechanism
■ Vectored Interrupts
■ Handler address determined by the cause
■ Example:
■ Undefined opcode: C000 0000 - Overflow: C000 0020
■ ...: C000 0040
■ Instructions either
■ Deal with the interrupt, or
■ Jump to real handler
M 14
Chapter 4 — The Processor — 96
Handler Actions
Read cause, and transfer to relevant handler
Determine action required If restartable
■ Take corrective action
■ use EPC to return to program
Otherwise
■ Terminate program
■ Report error using EPC, cause, ...
M 14
Chapter 4 — The Processor — 97
Exceptions in a Pipeline
Another form of control hazard Consider overflow on add in EX stage
add $1, $2, $1
■ Prevent $1 from being clobbered
■ Complete previous instructions
■ Flush add and subsequent instructions
■ Set Cause and EPC register values
■ Transfer control to handler
Similar to mispredicted branch
■ Use much of the same hardware
M 14
Chapter 4 — The Processor — 98
Pipeline with Exceptions
Chapter 4 — The Processor — 99
Exception Properties
Restartable exceptions
■ Pipeline can flush the instruction
■ Handler executes, then returns to the instruction
Refetched and executed from scratch
PC saved in EPC register
■ Identifies causing instruction
■ Actually PC + 4 is saved
■ Handler must adjust
M 14
Chapter 4 — The Processor — 100
Exception Example
Exception on add in
40	sub	$11,	$2, $4
44	and	$12,	$2, $5
48	or	$13,	$2, $6
4C	add	$1,	$2, $1
50	sit	$15,	$6, $7
54	lw	$16,	50($7)
■ ■ ■ Handler			
80000180 sw $2 5, 1000($0) 80000184     sw     $26, 1004($0)
M 14
Chapter 4 — The Processor — 101
Exception Example
lw$16, 50($7) IF.FIush
Sit $15, $6, $7
add $1, $2, $1
EX.FIush
Clock 6
Chapter 4 — The Processor — 102
Exception Example
IF
sw$25, 1000($0) Flush
- u -
4x]h
bubble (nop)
PC
4 j/
IF,
Instruction memory
Clock 7
ID.
58
Hazard detection unit
ID.FIush
bubble
EX.FIush
o-»-
ID/EX 0 [7771
0
EX
M u
X
0Ü
000
epc 0—A
riters   Q i- _t_J I—[La
M
u
X
bubble
EX/MEM
WB
u
x
)ALU
M
or$13,
Forwardingy unit
MEMAA/B
WB
Data memory
13
Chapter 4 — The Processor — 103
Multiple Exceptions
Pipelining overlaps multiple instructions
■ Could have multiple exceptions at once
Simple approach: deal with exception from earliest instruction
■ Flush subsequent instructions
■ "Precise" exceptions
In complex pipelines
■ Multiple instructions issued per cycle
■ Out-of-order completion
■ Maintaining precise exceptions is difficult!
M 14
Chapter 4 — The Processor — 104
Imprecise Exceptions
Just stop pipeline and save state
■ Including exception cause(s)
Let the handler work out
■ Which instruction(s) had exceptions
■ Which to complete or flush
■ May require "manual" completion
Simplifies hardware, but more complex handler software
Not feasible for complex multiple-issue out-of-order pipelines
M 14
Chapter 4 — The Processor — 105
Instruction-Level Parallelism (ILP)
Pipelining: executing multiple instructions in parallel
To increase ILP
■ Deeper pipeline
> Less work per stage =^> shorter clock cycle
■ Multiple issue
> Replicate pipeline stages =^> multiple pipelines
> Start multiple instructions per clock cycle
- CPI < 1, so use Instructions Per Cycle (IPC)
> E.g., 4GHz 4-way multiple-issue
16 BIPS, peak CPI = 0.25, peak IPC = 4 But dependencies reduce this in practice
M 14
Chapter 4 — The Processor
Multiple Issue
Static multiple issue
■ Compiler groups instructions to be issued together
■ Packages them into "issue slots"
■ Compiler detects and avoids hazards
Dynamic multiple issue
■ CPU examines instruction stream and chooses instructions to issue each cycle
■ Compiler can help by reordering instructions
■ CPU resolves hazards using advanced techniques at runtime
M 14
Chapter 4 — The Processor — 107
Speculation
"Guess" what to do with an instruction
■ Start operation as soon as possible
■ Check whether guess was right
> If so, complete the operation
> If not, roll-back and do the right thing
Common to static and dynamic multiple issue Examples
■ Speculate on branch outcome
Roll back if path taken is different
■ Speculate on load
Roll back if location is updated
M 14
Chapter 4 — The Processor — 108
Compiler/Hardware Speculation
Compiler can reorder instructions
■ e.g., move load before branch
■ Can include "fix-up" instructions to recover from incorrect guess
Hardware can look ahead for instructions to execute
■ Buffer results until it determines they are actually needed
■ Flush buffers on incorrect speculation
M 14
Chapter 4 — The Processor — 109
Speculation and Exceptions
What if exception occurs on a speculatively executed instruction?
■ e.g., speculative load before null-pointer check
Static speculation
■ Can add ISA support for deferring exceptions
Dynamic speculation
■ Can buffer exceptions until instruction completion (which may not occur)
M 14
Chapter 4 — The Processor — 110
Static Multiple Issue
■ Compiler groups instructions into "issue packets"
■ Group of instructions that can be issued on a single cycle
■ Determined by pipeline resources required
■ Think of an issue packet as a very long instruction
■ Specifies multiple concurrent operations
■ => Very Long Instruction Word (VLIW)
M 14
Chapter 4 — The Processor — 111
Scheduling Static Multiple Issue
Compiler must remove some/all hazards
■ Reorder instructions into issue packets
■ No dependencies with a packet
■ Possibly some dependencies between packets
Varies between ISAs; compiler must know!
■ Pad with nop if necessary
M 14
Chapter 4 — The Processor — 112
MIPS with Static Dual Issue
Two-issue packets
■ One ALU/branch instruction
■ One load/store instruction
■ 64-bit aligned
> ALU/branch, then load/store Pad an unused instruction with nop
Address	Instruction type	Pipeline Stages						
n	ALU/branch	IF	ID	EX	MEM	WB		
n + 4	Load/store	IF	ID	EX	MEM	WB		
n + 8	ALU/branch		IF	ID	EX	MEM	WB	
n + 12	Load/store		IF	ID	EX	MEM	WB	
n + 16	ALU/branch			IF	ID	EX	MEM	WB
n + 20	Load/store			IF	ID	EX	MEM	WB
M 14
Chapter 4 — The Processor — 113
MIPS with Static Dual Issue
80000180
M u
X
M 14
Chapter 4 — The Processor— 114
Hazards in the Dual-Issue MIPS
More instructions executing in parallel EX data hazard
■ Forwarding avoided stalls with single-issue
■ Now can't use ALU result in load/store in same packet
. add   $t0, $s0, $sl load $s2, 0($t0)
> Split into two packets, effectively a stall
Load-use hazard
■ Still one cycle use latency, but now two instructions
More aggressive scheduling required
M 14
Chapter 4 — The Processor — 115
Scheduling Example
Schedule this for dual-issue MIPS
Loop:
lw     $t0, 0($sl)
addu $t0, $t0, $s2
sw     $t0, 0($sl)
addi $sl, $sl,-4
# $tO=array element
# add scalar in $s2
# store result
# decrement pointer
bne   $sl, $zero, Loop # branch $sl!=0
	ALU/branch	Load/store	cycle
Loop:	nop	lw     $t0, 0($sl)	1
	addi $sl, $sl,-4	nop	2
	addu $t0, $t0, $s2	nop	3
	bne   $sl, $zero, Loop	sw     $t0, 4($sl)	4
IPC = 5/4 = 1.25 (c.f. peak IPC = 2)
4
Chapter 4 — The Processor — 116
Loop Unrolling
Replicate loop body to expose more parallelism
■ Reduces loop-control overhead
Use different registers per replication
■ Called "register renaming"
■ Avoid loop-carried "anti-dependencies"
Store followed by a load of the same register
Aka "name dependence" Reuse of a register name
M 14
Chapter 4 — The Processor — 117
Loop Unrolling Example
	ALU/branch	Load/store	cycle
Loop:	addi $sl, $sl,-16	lw     $t0, 0($sl)	1
	nop	lw     $tl, 12($sl)	2
	addu $t0, $t0, $s2	lw     $t2, 8($sl)	3
	addu $tl, $tl, $s2	lw     $t3, 4($sl)	4
	addu $t2, $t2, $s2	sw     $t0, 16($sl)	5
	addu $t3, $t4, $s2	sw     $tl, 12($sl)	6
	nop	sw     $t2, 8($sl)	7
	bne   $sl, $zero, Loop	sw     $t3, 4($sl)	8
IPC = 14/8 = 1.75
■ Closer to 2, but at cost of registers and code size
M 14
Chapter 4 — The Processor — 118
Dynamic Multiple Issue
"Superscalar" processors
CPU decides whether to issue 0, 1,2, each cycle
■ Avoiding structural and data hazards
Avoids the need for compiler scheduling
■ Though it may still help
■ Code semantics ensured by the CPU
M 14
Chapter 4 — The Processor — 119
Dynamic Pipeline Scheduling
Allow the CPU to execute instructions out of order to avoid stalls
■ But commit result to registers in order
Example
lw $t0, 20($s2)
addu $tl, $t0, $t2
sub $s4, $s4, $t3
slti $t5, $s4, 20
■ Can start sub while addu is waiting for lw
M 14
Chapter 4 — The Processor — 120
Dynamically Scheduled CPU
			Instruction fetch and decode unit			Ii		
								
1						r		r
Reservation station		Reservation station			Reservation station		Reservation station	
								
In-order issue
Functional units
Integer
Integer
J
Floating point
Out-of-order execute
Reorders buffer for register writes
Commit unit
n-order commit
4
Can supply operands for issued instructions
Preserves dependencies
Hold pending operands
Results also sent
to any waiting reservation stations
Chapter 4 — The Processor — 121
Register Renaming
Reservation stations and reorder buffer effectively provide register renaming
On instruction issue to reservation station
■ If operand is available in register file or reorder buffer
■ Copied to reservation station
No longer required in the register; can be overwritten
■ If operand is not yet available
■ It will be provided to the reservation station by a function unit
■ Register update may not be required
Chapter 4 — The Processor — 122
Speculation
Predict branch and continue issuing
■ Don't commit until branch outcome determined
Load speculation
■ Avoid load and cache miss delay
Predict the effective address
■ Predict loaded value
Load before completing outstanding stores
■ Bypass stored values to load unit
■ Don't commit load until speculation cleared
M 14
Chapter 4 — The Processor — 123
Why Do Dynamic Scheduling?
Why not just let the compiler schedule code?
Not all stalls are predicable
■ e.g., cache misses
Can't always schedule around branches
■ Branch outcome is dynamically determined
Different implementations of an ISA have different latencies and hazards
M 14
Chapter 4 — The Processor — 124
Does Multiple Issue Work?
The BIG Picture
Yes, but not as much as we'd like
Programs have real dependencies that limit ILP
Some dependencies are hard to eliminate
■ e.g., pointer aliasing
Some parallelism is hard to expose
■ Limited window size during instruction issue
Memory delays and limited bandwidth
■ Hard to keep pipelines full
Speculation can help if done well
Chapter 4 — The Processor — 125
Power Efficiency
Complexity of dynamic scheduling and speculations requires power
Multiple simpler cores may be better
Microprocessor	Year	Clock Rate	Pipeline Stages	Issue width	Out-of-order/ Speculation	Cores	Power
i486	1989	25MHz	5	1	No	1	5W
Pentium	1993	66MHz	5	2	No	1	10W
Pentium Pro	1997	200MHz	10	3	Yes	1	29W
P4 Willamette	2001	2000MHz	22	3	Yes	1	75W
P4 Prescott	2004	3600MHz	31	3	Yes	1	103W
Core	2006	2930MHz	14	4	Yes	2	75W
UltraSparc III	2003	1950MHz	14	4	No	1	90W
UltraSparcTI	2005	1200MHz	6	1	No	8	70W
M 14
Chapter 4 — The Processor — 126
The Opteron X4 Microarchitecture
Instruction cache
Branch prediction
Instruction prefetch and decode
T
RISC-operation queue
T
Dispatch and register remaining
Integer and floating-point operation queue
Integer ALU. Multiplier
Integer ALU
Integer ALU
Floating point Adder /SSE
Floating point
Multiplier /SSE
Load/Store queue
Data cache
4
Commit unit
72 physical registers
Chapter 4 — The Processor — 127
The Opteron X4 Pipeline Flow
For integer operations
RISC-operation queue
Number of clock cycles
Reorder buffer allocation + register renaming
Reorder buffer
Scheduling + dispatch unit
Execution J	1 .	Data Cache/ Commit
		
■ FP is 5 stages longer
■ Up to 106 RISC-ops in progress
Bottlenecks
■ Complex instructions with long dependencies
■ Branch mispredictions
■ Memory access delays
14
Chapter 4 — The Processor — 128
Fallacies
Pipelining is easy (!)
■ The basic idea is easy
■ The devil is in the details
> e.g., detecting data hazards
Pipelining is independent of technology
■ So why haven't we always done pipelining?
■ More transistors make more advanced techniques feasible
■ Pipeline-related ISA design needs to take account of technology trends
e.g., predicated instructions
M 14
Chapter 4 — The Processor — 1
Pitfalls
Poor ISA design can make pipelining harder
■ e.g., complex instruction sets (VAX, IA-32)
Significant overhead to make pipelining work IA-32 micro-op approach
■ e.g., complex addressing modes
■ Register update side effects, memory indirection
■ e.g., delayed branches
■ Advanced pipelines have long delay slots
M 14
Chapter 4 — The Processor — 130
Concluding Remarks
ISA influences design of datapath and control
Datapath and control influence design of ISA
Pipelining improves instruction throughput using parallelism
■ More instructions completed per second
■ Latency for each instruction not reduced
Hazards: structural, data, control
Multiple issue and dynamic scheduling (ILP)
■ Dependencies limit achievable parallelism
■ Complexity leads to the power wall
M 14
Chapter 4 — The Processor —
Exercises
Answer the following exercises, and send your answers as a PDF attachment to the email address listed below
xamiri@fi.muni.cz Leave body of the email blank Deadline is May 13th Exercises
4.1.1(b), 4.2.4(a), 4.7.3(a), 4.9.2(a)~4.9.4(a), 4.10.4(a), 4.11.1(a)~4.11.4(a), 4.13.1(a), 4.13.5(a), 4.16.4(b), 4.18.1(b), 4.21.2(a), 4.21.4(a), 4.22.1(b), 4.23.2(a), 4.24.2(b)
M 14
Chapter 1 — Computer Abstractions and Technology — 132