Shared Memory Parallel Programming - Lecture Slides | COSC 4397, Study notes of Computer Science

Download Shared Memory Parallel Programming - Lecture Slides | COSC 4397 and more Study notes Computer Science in PDF only on Docsity!

Shared Memory Parallel

Programming

Shared Memory SystemsIntroduction to OpenMP

Parallel Architectures

• Distributed Memory Machine (DMP)• Shared Memory Machine (SMP)DMP Multicomputer Architecture

ProcessorM

ProcessorM

ProcessorM Message Passing Network

•••••••••••••

Shared Memory

P^

C^

P^

C^

P^

C

SMP Multiprocessor Architecture

Bus or Switch Network

Sun Fire 6800 Server

Up to 24 UltraSPARC III processorsin SMP architecture192GB memory9.6GB bandwidth4-way associative on-chip 64KB dataand 32KB instruction cache8MB E-cache

Shared Memory Architectures

Built during the 1980’s, now popular againMultiple processors that access

same main memory

•^

local cache, connected to main memory via bus

-^

When a memory location is updated, any copies inthe local cache of other processors must beinvalidated

-^

Bus is bottleneck; traditional wisdom is that onlysmall systems are possible
 

^

  

More Realistic View of Shared

Memory Architecture

For small numbers of processors

proc

proc

proc

procN

Shared memory

cache

cache

cache

cacheN

Distributed Shared Memory

Systems

• Architecture designed to overcome size

limits of shared memory systems

• Distributed memory system but memory is

globally addressed

• Hardware supports cache coherency• So looks like shared memory systemExamples: HP Exemplar, SGI Origin, Altix

From Clusters of SMPs to Distributed

Global Address Space … …

grids

   

Future HPC Hardware

• Feature set?
  • Massive parallelism – Global address space – High bandwidth to memory – Thread-based components – Efficient synchronization, context switching
    • Execution model?
      • Multi-level parallelism,

multithreading

Multicore Complexity

-^

Resources (L2 cache, memory bandwidth): shared or separated

-^

Each core: single thread or multithreaded, complex or simplified

-^

Among cores: symmetric or asymmetric (Heterogeneous)





 



 





















Challenges Posed By New

Architectures

•^

Hierarchical and hybrid parallelism^ –

Clusters, SMPs, CMP (multicores), SMT(simultaneous multithreading),

…

•^

Diversity in kind and extent of resourcesharing, potential for thread contention^ –

ALU/FP units, cache, MCU, data-path, memorybandwidth

-^

Homogeneous or heterogeneous

-^

Deeper memory hierarchy

-^

Size and scale

^

 



What is OpenMP?

• New standard for

shared memory

parallel

programming for scientific applications

• Focus is on scientific applications• Rapidly gaining acceptance among vendors

and application writers

• For Fortran, C and C++• http://www.openmp.org

The OpenMP ARB

•^

OpenMP is maintained by the OpenMP ArchitectureReview Board (the ARB).

-^

The ARB:

• Interprets OpenMP• Writes new specifications - keeps OpenMP relevant• Works to increase the impact of OpenMP

•^

Members are organizations - not individuals– Current members

• Permanent: Fujitsu, HP, IBM, Intel, NEC, PGI, SGI, Sun• Auxiliary: ASCI, cOMPunity, EPCC, KSL, NASA, RWTH Aachen

OpenMPOpenMPC/C++ 1.0C/C++ 1.

OpenMP Release History

1998

OpenMP^ OpenMPFortran 1.1Fortran 1.

1999

OpenMP^ OpenMPC/C++ 2.0C/C++ 2.

2002

OpenMP^ OpenMPFortran 1.0Fortran 1.

1997

OpenMP

OpenMP

2005 2.

OpenMP^ OpenMPFortran 2.0Fortran 2.

2000

C/C++ Fortran

OpenMP Overview

• A set of compiler directives inserted in the

source program

• Also some library functions• Ideally, compiler directives do not affect

sequential code– pragmas in C / C++– (specially written) comments in Fortran code