Dependence Analysis: Understanding Parallelism and Data Dependences in Programming, Slides of Computer Science

Download Dependence Analysis: Understanding Parallelism and Data Dependences in Programming and more Slides Computer Science in PDF only on Docsity!

Dependence Analysis

Michael O’BoyleFebruary, 2012

M. O’Boyle

Dependence Analysis

1

Course Structure

•^

5 lectures on high level restructuring for parallelism and memory

-^

Dependence Analysis

-^

Program Transformation

-^

Automatic vectorisation

-^

Automatic parallelisation

-^

Speculative parallelisation

-^

Then adaptive compilation M. O’Boyle

Dependence Analysis

3

References

•^

R. Allen and K Kennedy Optimizing compilers for modern architectures:

A

dependence based approach Morgan Kaufmann 2001. Main reference for thissection of lecture notes

-^

Michael Wolfe High Performance Compilers for Parallel Computing Addison-Wesley 1996.

-^

H. Zima and B. Chapman. Supercompilers for Parallel and Vector Computers.ACM Press Frontier Series 1990

-^

Today : The Omega Test: a fast and practical integer programming algorithmfor dependence analysis Supercomputing 1992 M. O’Boyle

Dependence Analysis

February, 2012

4

Programming Parallel Computers

•^

Two extremes: User specifies parallelism and mapping

-^

Compiler parallelises and maps “dusty deck” sequential codes- Debatable how far this can go

-^

A popular approach is to break the transformation process into stages

-^

Transform

to

maximise

parallelism

i.e

minimise

critical

path

of

program

execution graph

-^

Map

parallelism

so

as

to

minimise

“significant”

machine

costs

i.e.

communication/ non-local access etc. M. O’Boyle

Dependence Analysis

February, 2012

6

Loop Parallelism / Array Parallelism

Original loop

Parallel loop

Do i = 1 , n

a(i) = b(i) EndDo

Doall i = 1 , n

a(i) = b(i) EndDo

•^

All iterations of the iterator

i

can be performed independently

•^

Independence implies Parallelism

-^

We will concentrate on loop parallelism O(n) potential parallelism. Statementand Operation - O(1).

-^

Recursive

parallelism

is

rich

but

very

dynamic.

Exploited

in

functional

computational models M. O’Boyle

Dependence Analysis

February, 2012

7

Parallelism and Data Dependence

Do i = 1,

a(i) = b(i) + c(i) EndDo

1

2

3

10

Do i = 2,

a(i) = a(i-1)

• function_call(i)

EndDo

1

2

3

10

Completely parallel each iteration is totally independentCompletely serial each iteration depends on the previous iterationNote: iterations NOT array elements M. O’Boyle

Dependence Analysis

9

Data Dependence

•^

In general we need to know if two usages of an array access the same memorylocation and what type of dependence

-^

Helpful as this can be done relatively cheaply for simple programs

-^

General dependence is intractable - equivalent to Hilbert’s tenth problem a (f

(i

a

(g

(i

for arbitrary

f

,^

g

•^

Decidable (NP) if

f, g

linear

M. O’Boyle

Dependence Analysis

10

Dependence in loops

Do i =1,N a(f(i)) =

= a(g(i))

EndDo^ •

Conditions for flow dependence from iteration

I

w

to

I

r

I

w

I

r^

N

f

Iw

g

(I

)r

•^

Conditions for anti dependence from iteration

I

r^

to

I

w

I

r^

< I

w

N

g

(I

) =r

f

Iw

•^

Conditions for output dependence from iteration

I

w

1

to

I

w

2

I

w

1

< I

w

2

N

f

(I

w

f

(I

w

M. O’Boyle

Dependence Analysis

12

Dependence distance and direction: (approx) summarising dependence Do i =1,N

Do j = 1,M

a(i,j) = a(i-1,j+1) + EndDo EndDo^ •

Flow dependence

,^ 2)

,^ 1)

,^ (

,^ 3)

,^ 2)

,^ (

,^ 2)

,^ 1)

•^

Dependence

Iw

, J

w

)^

Ir

, J

) :r

I

r^

I

w

, J

r^

J

w

•^

Distance vector is [1,-1].

Direction vector [+,-] or [

] sign of direction.

Any: [*], 0 [=],Positive [

], Negative [

]

•^

First non zero vector element cannot be negative - why? M. O’Boyle

Dependence Analysis

13

Hiearchical Computing of Dependence Directions in loops.

Do i =1,N a(f(i)) =

= a(g(i))

EndDo^ •

Test for any dependence from iteration

I

w

to

I

:r

I

w

, I

r^

N

f

Iw

g

(I

)r

•^

Use this test to test any direction[*].

-^

If solutions add additional constraints:

[<

]^

direction: add

I

w

< I

,r

[=]

add

I

w

I

.r

•^

Extend for multi loops,

[

]^

then

[

],

[=

,^ ∗

]^

etc - hierarchical testing

M. O’Boyle

Dependence Analysis

15

Separable SIV test

Do i =1,N Do j= 1,N Do k = 1,N

x(aI+b,..,..) = x(cI+d,..,..)

EndDo^ •

If equations for one iterator appear in only one subscript, we can separate itand solve independently. • a

×

I

w

b

c

×

I

r^

d

Strong SIV,

a

c

, so

I

r^

I

w

b^

d

)/a

•^

If a divides b-d and result is in range of I, then we have the dependencedistance. Weak SIV : a or c =0. M. O’Boyle

Dependence Analysis

16

General SIV test or Greatest Common Divisor

Do i =1,N Do j= 1,N Do k = 1,N

x(aI+b,..,..) = x(cI+d,..,..)

EndDo^ •

We have

a

×

I

w

b

c

×

I

r^

d

•^

If gcd(a,c) does not divide d-b no solution try a=c=2, d=1,b=

-^

ELSE ... potentially many solutions. M. O’Boyle

Dependence Analysis

18

Example using flow constraint

Do i =1,

a(2*i+3) = a(i+7)

-^

We have

Iw

I

r^

, h

Iw

I

r^

and

I

w

I

r^

•^

Min h = (21 -100 -4) = -102, Max h = (2100-1-4)=195 195

hence solution

•^

Simple example can be extended. Technical difficulties with complex iterationspaces

-^

Performed sub-script at a time, Used for MIV M. O’Boyle

Dependence Analysis

19

Omega Test - Read the paper!

•^

Most compilers still use classification and special tests for dependence

-^

However

Pugh’s

Omega

Test

can

solve

exactly

using

integer

linear

programming.

-^

Basically state constraints and put into a smart Fourier-Motzkin eliminationbased solver

-^

Shown that worst case double exponential cost on manipulating Presburgherformula is frequently low-end polynomial M. O’Boyle

Dependence Analysis

February, 2012