Neural Networks on Parallel Architectures: Learning and Parallelism, Slides of Computer Architecture and Organization

Download Neural Networks on Parallel Architectures: Learning and Parallelism and more Slides Computer Architecture and Organization in PDF only on Docsity!

Neural Network Implementations on

Parallel Architectures

Index

• Neural Networks
• Learning in NN
• Parallelism
• Characteristics
• Mapping Schemes & Architectures

Artificial Neural Networks-

• weights may be positive or negative
• each unit computes a simple function of its inputs,which are weighted outputs from other units

Artificial Neural Networks-

• a threshold value is used in each neuron to determine the activation of the output
• learning in NN:
  • finding the weights and threshold values
  • training set
• multi-layer,feedforward networks: input layer,hidden layer,output layer

Parallelism

• further speed-up of training
• neural networks exhibit high degree of parallelism(distributed set of units operating simultaneously)
• process of parallelism:
  • what type of machine?
  • how to parallelize?

Characteristics

• theoretical analysis of the inherent algorithm
• portability
• ease of use
• access to ANN model description

Historical Data Integration

• prediction of the sensor output
• two parallelism methods:
  • parallel calculation of weighted sum
    • time increases with the number of processors
  • parallel training of each seperate NN
    • time decreases with the number of processors
• 8 RISC processors
• 4 MB cache memory
• 512 RAM

Distributed Training Data

A library on MIMD machines-

• several communication and syncronization schemes
  • message passing or shared memory
    • thread programming with shared memory has the best performance
  • every data is shared but handled only by one processor
• training of a Kohonen map of 100*100 neurons with 100000 iterations with 8 processors are 7 times faster than the sequential execution.

• Learning in ANN-
• Method-
• Method-
• A library on MIMD machines-

AP1000 Architecture-

• MIMD computer with distributed memory
• vertical slicing of the network
• 3 methods for communication
  • One to one communication
  • Rotated messages in horizontal and vertical rings
  • Parallel routed messages
• different neural network implementations

AP1000 Architecture-

• different mappings according to the network and the training data
• heuristic on training time
• combine multiple degrees of parallelism
  • training set parallelism
  • node parallelism
  • pipelining parallelism