Download Embarrassingly Parallel Computation - Notes | CSC 4310 and more Study notes Computer Science in PDF only on Docsity!
Slides for Parallel Programming: Techniques and Applications using Networked Workstations and Parallel Computers Barry Wilkinson and Michael Allen Prentice Hall, 1999. All rights reserved.
Processes
Results
Input data
Figure 3.1 Disconnected computational graph (embarrassingly parallel problem).
Chapter 3 Embarrassingly Parallel
Computations
A computation that can be divided into a number of completely independent parts, each of which can be executed by a separate processor.
Figure 3.2 Practical embarrassingly parallel computational graph with dynamic process creation and the master-slave approach.
Send initial data
Collect results
Master
Slaves
spawn()
recv()
send()
recv()
send()
Slides for Parallel Programming: Techniques and Applications using Networked Workstations and Parallel Computers Barry Wilkinson and Michael Allen Prentice Hall, 1999. All rights reserved.
Embarrassingly Parallel Examples
Low level image operations: (a) Shifting Object shifted by ∆ x in the x -dimension and ∆ y in the y -dimension: x ′ = x + ∆ x y ′ = y + ∆ y where x and y are the original and x ′ and y ′ are the new coordinates.
(b) Scaling Object scaled by a factor Sx in the x -direction and Sy in the y- direction: x ′ = xSx y ′ = ySy
(c) Rotation Object rotated through an angle θ about the origin of the coordinate system: x ′ = x cosθ + y sinθ y ′ = − x sinθ + y cosθ
(a) Square region for each process
Figure 3.3 Partitioning into regions for individual processes.
Process
Map
x
y
Slides for Parallel Programming: Techniques and Applications using Networked Workstations and Parallel Computers Barry Wilkinson and Michael Allen Prentice Hall, 1999. All rights reserved.
Slave
recv(row, Pmaster); / receive row no. / for (oldrow = row; oldrow < (row + 10); oldrow++) for (oldcol = 0; oldcol < 640; oldcol++) {/ transform coords / newrow = oldrow + delta_x; / shift in x direction / newcol = oldcol + delta_y; / shift in y direction / send(oldrow,oldcol,newrow,newcol, Pmaster);/ coords to master / }
Analysis
Sequential
ts = n^2 = Ο( n^2 )
Parallel
Communication t comm = t startup + mt data t comm = p ( t startup + 2 t data) + 4 n^2 ( t startup + t data) = Ο( p + n^2 ) Computation
= Ο( n^2 / p ) Overall Execution Time tp = t comp + t comm
For constant p , this is Ο( n^2 ). However, the constant hidden in the communication part far exceeds those constants in the computation in most practical situations.
t comp 2 n
2 ---- p -
= ^
Slides for Parallel Programming: Techniques and Applications using Networked Workstations and Parallel Computers Barry Wilkinson and Michael Allen Prentice Hall, 1999. All rights reserved.
Mandelbrot Set
Set of points in a complex plane that are quasi-stable (will increase and decrease, but not exceed some limit) when computed by iterating the function zk +1 = zk^2 + c where zk +1 is the ( k + 1)th iteration of the complex number z = a + bi and c is a complex number giving the position of the point in the complex plane. The initial value for z is zero.
Iterations continued until magnitude of z is greater than 2 or number of iterations reaches arbitrary limit. Magnitude of z is the length of the vector given by z length = a^2 + b^2
Computing the complex function, zk +1 = zk^2 + c , is simplified by recognizing that z^2 = a^2 + 2 abi + bi^2 = a^2 − b^2 + 2 abi or a real part that is a^2 − b^2 and an imaginary part that is 2 ab.
The next iteration values can be produced by computing: z real = z real^2 - z imag^2 + c real z imag = 2 z real z imag + c imag
Slides for Parallel Programming: Techniques and Applications using Networked Workstations and Parallel Computers Barry Wilkinson and Michael Allen Prentice Hall, 1999. All rights reserved.
Real Figure 3.4 Mandelbrot set.
− 2 0 + 2
− 2
0
Imaginary
Parallelizing Mandelbrot Set Computation
Static Task Assignment
Master for (i = 0, row = 0; i < 48; i++, row = row + 10)/ for each process/ send(&row, Pi); /* send row no./ for (i = 0; i < (480 * 640); i++) {/ from processes, any order / recv(&c, &color, PANY); / receive coordinates/colors / display(c, color); / display pixel on screen / }*
Slave (process i) recv(&row, Pmaster); / receive row no. / for (x = 0; x < disp_width; x++)/ screen coordinates x and y / for (y = row; y < (row + 10); y++) { c.real = min_real + ((float) x * scale_real); c.imag = min_imag + ((float) y * scale_imag); color = cal_pixel(c); send(&c, &color, Pmaster);/ send coords, color to master / }
Slides for Parallel Programming: Techniques and Applications using Networked Workstations and Parallel Computers Barry Wilkinson and Michael Allen Prentice Hall, 1999. All rights reserved.
Work pool
( xc , yc )
( xa , ya )
( xb , yb ) ( xd ,^ yd )
( xe , ye )
Figure 3.5 Work pool approach.
Task Return results/ request new task
Dynamic Task Assignment
Work Pool/Processor Farms
Coding for Work Pool Approach
Master count = 0; / counter for termination/ row = 0; /* row being sent / for (k = 0; k < procno; k++) { / assuming procno<disp_height / send(&row, Pk, data_tag); / send initial row to process / count++; / count rows sent / row++; / next row / } do { recv (&slave, &r, color, PANY, result_tag); count--; / reduce count as rows received / if (row < disp_height) { send (&row, Pslave, data_tag); / send next row / row++; / next row / count++; } else send (&row, Pslave, terminator_tag); / terminate / rows_recv++; display (r, color); / display row / } while (count > 0);*
Slides for Parallel Programming: Techniques and Applications using Networked Workstations and Parallel Computers Barry Wilkinson and Michael Allen Prentice Hall, 1999. All rights reserved.
Analysis
Sequential
ts ≤ max × n = Ο( n )
Parallel program
Phase 1: Communication - Row number is sent to each slave t comm1 = s ( t startup + t data) Phase 2: Computation - Slaves perform their Mandelbrot computation in parallel
Phase 3: Communication - Results passed back to master using individual sends
Overall
t comp≤^ max ------------------ s^ × n -
t comm2 =^ n^ -- s -^ ( t startup + t data)
t (^) p ≤^ max ------------------ s^ × n - + ns --- + s ^ ( t startup + t data)
Figure 3.7 Computing π by a Monte Carlo method.
Area = π
Total area = 4
2
2
Monte Carlo Methods
Basis of Monte Carlo methods is the use of random selections in calculations.
Example - To calculate π
A circle is formed within a square. Circle has unit radius so that square has sides 2 × 2.
Slides for Parallel Programming: Techniques and Applications using Networked Workstations and Parallel Computers Barry Wilkinson and Michael Allen Prentice Hall, 1999. All rights reserved.
The ratio of the area of the circle to the square is given by
Points within the square are chosen randomly and a score is kept of how many points happen to lie within the circle.
The fraction of points within the circle will be π/4, given a sufficient number of randomly selected samples.
Area of circle Area of square-----------------------------------^
π ( 1 )^2 ------------- 2 × 2 - π = = 4 ---
x
(^1) y = 1 – x 2
f ( x )
Figure 3.8 Function being integrated in computing π by a Monte Carlo method.
Computing an Integral
One quadrant of the construction in Figure 3.7 can be described by the integral
A random pair of numbers, ( xr , yr ) would be generated, each between 0 and 1, and then counted as in circle if ; that is,.
∫ 01 1 –^ x^2 dx = π 4 ---
yr ≤ 1 – xr^2 yr^2 + xr^2 ≤ 1
Slides for Parallel Programming: Techniques and Applications using Networked Workstations and Parallel Computers Barry Wilkinson and Michael Allen Prentice Hall, 1999. All rights reserved.
Master
Slaves
Random number process
Random number
Partial sum Request
Figure 3.9 Parallel Monte Carlo integration.
Parallel Implementation
Pseudocode
Master
for(i = 0; i < N/n; i++) { for (j = 0; j < n; j++) / n=no of random numbers for slave / xr[j] = rand(); / load numbers to be sent / recv(PANY, req_tag, Psource); / wait for a slave to make request / send(xr, &n, Psource, compute_tag); } for(i = 0; i < slave_no; i++) { / terminate computation / recv(Pi, req_tag); send(Pi, stop_tag); } sum = 0; reduce_add(&sum, Pgroup);
Slides for Parallel Programming: Techniques and Applications using Networked Workstations and Parallel Computers Barry Wilkinson and Michael Allen Prentice Hall, 1999. All rights reserved.
Slave
sum = 0; send(Pmaster, req_tag); recv(xr, &n, Pmaster, source_tag); while (source_tag == compute_tag) { for (i = 0; i < n; i++) sum = sum + xr[i] * xr[i] - 3 * xr[i]; send(Pmaster, req_tag); recv(xr, &n, Pmaster, source_tag); }; reduce_add(&sum, Pgroup);
Random Number Generation
The most popular way of creating a pueudorandom number sequence: x 1 , x 2 , x 3 , …, xi − 1 , xi , xi + 1 , …, xn − 1 , xn , is by evaluating xi +1 from a carefully chosen function of xi , often of the form xi +1 = ( axi + c ) mod m
where a , c , and m are constants chosen to create a sequence that has similar properties to truly random sequences.
