Introduction - Parallel Processing - Lecture Slides, Slides of Parallel Computing and Programming

Download Introduction - Parallel Processing - Lecture Slides and more Slides Parallel Computing and Programming in PDF only on Docsity!

Introduction to Parallel

Processing

Outline

• Moore's Law and its limits
• Different uni-processor performance enhancement

techniques and their limits

• Classification of parallel computations
• Classification of parallel architectures - Distributed and

Shared memory

• Simple examples of parallel processing
• Example applications
• Future advances
• Summary

Quest for Performance

• Pipelining
• Superscalar Architecture
• Out of Order Execution
• Caches
• Instruction Set Design Advancements
• Parallelism
  • Multi-core processors
  • Clusters
  • Grid

This is the future

Pipelining

• Illustration of Pipeline using the fetch, load, execute, store stages.
• At the start of execution – Wind up.
• At the end of execution – Wind down.
• Pipeline stalls due to data dependency (RAW, WAR), resource conflict, incorrect branch prediction – Hit performance and speedup.
• Pipeline depth – No of cycles in execution simultaneously.
• Intel Pentium 4 – 35 stages.

Cache

• Desire for fast cheap and non volatile memory
• Memory speed growth at 7% per annum while processor growth at 50% p.a.
• Cache – fast small memory.
• L1 and L2 caches.
• Retrieval from memory takes several hundred clock cycles
• Retrieval from L1 cache takes the order of one clock cycle and from L2 cache takes the order of 10 clock cycles.
• Cache ‘hit’ and ‘miss’.
• Prefetch used to avoid cache misses at the start of the execution of the program.
• Cache lines used to avoid latency time in case of a cache miss
• Order of search – L1 cache -> L2 cache -> RAM -> Disk
• Cache coherency – Correctness of data. Important for distributed parallel computing
• Limit to cache improvement: Improving cache performance will at most improve efficiency to match processor efficiency

(exs. of limited data parallelism)

(exs. of limited & low-level functional parallelism)

(single-instr. multiple data)

: instruction-level parallelism—degree generally low and dependent on how the sequential code has been written, so not v. effective

• Thus need development of explicit parallel algorithms that are

based on a fundamental understanding of the parallelism inherent in a problem, and exploiting that parallelism with minimum interaction/communication between the parallel parts

(simultaneous multi- threading)

(multi-threading)

Applications of Parallel Processing