Artificial Neural Networks, Slides of Artificial Intelligence

Download Artificial Neural Networks and more Slides Artificial Intelligence in PDF only on Docsity!

Artificial Neural Networks

Part 1/

Slides modified from Neural Network Design by Hagan, Demuth and Beale Berrin Yanikoglu DA514– Machine Learning

Biological Inspirations

Human Brain

• The brain is a highly complex, non-linear, and parallel computer, composed of some 1011 neurons that are densely connected (~ 104 connection per neuron). We have just begun to understand how the brain works...

• A neuron is much slower ( 10

  sec) compared to a silicon logic gate ( 10 -9sec), however the massive interconnection between neurons make up for the comparably slow rate. - Complex perceptual decisions are arrived at quickly (within a few hundred milliseconds)
• 100-Steps rule: Since individual neurons operate in a few milliseconds, calculations do not involve more than about 100 serial steps and the information sent from one neuron to another is very small (a few bits)
• Plasticity: Some of the neural structure of the brain is present at birth, while other parts are developed through learning, especially in early stages of life, to adapt to the environment (new inputs).

Biological Neuron

A variety of different neurons exist (motor neuron,

on-center off-surround visual cells…), with different

branching structures.

The connections of the network and the strengths of

the individual synapses establish the function of the

network.

Artificial Neural Networks Computational models inspired by the human brain:

• Massively parallel, distributed system, made up of simple processing units (neurons)
• Synaptic connection strengths among neurons are used to store the acquired knowledge.
• Knowledge is acquired by the network from its environment through a learning process

Properties of ANNs Learning from examples

• labeled or unlabeled Adaptivity
• changing the connection strengths to learn things Non-linearity
• the non-linear activation functions are essential Fault tolerance
• if one of the neurons or connections is damaged, the whole network still works quite well Thus, they might be better alternatives than classical solutions for problems characterised by:
• high dimensionality, noisy, imprecise or imperfect data; and
• a lack of a clearly stated mathematical solution or algorithm

Artificial Neuron Model Neuron i Activation Output Input Synaptic Weights

x

0

x 1 x 2 x 3 x m

w

i

w

im

a

f i

function

b

i :Bias

Bias n ai = f (ni) = f (Σwijxj + bi) j = 1 An artificial neuron:

• computes the weighted sum of its input (called its net input )
• adds its bias
• passes this value through an activation function We say that the neuron “fires” (i.e. becomes active) if its output is above zero.

Activation functions Also called the squashing function as it limits the amplitude of the output of the neuron. Many types of activations functions are used:

• linear: a = f(n) = n
• threshold: a = {1 if n >= 0 (hardlimiting) 0 if n < 0
• sigmoid: a = 1/(1+e -n )
• ...

Activation Functions

Different Network Topologies Single layer feed-forward networks

• Input layer projecting into the output layer

Input Output

layer layer

Different Network Topologies Multi-layer feed-forward networks

• One or more hidden layers.
• Input projects only from previous layers onto a layer. typically, only from one layer to the next

Input Hidden Output

layer layer layer

2-layer or

1-hidden layer

fully connected

network

Applications of ANNs ANNs have been widely used in various domains for:

• Pattern recognition
• Function approximation
• Associative memory
• ...

Artificial Neural Networks Early ANN Models:

• Perceptron, ADALINE, Hopfield Network Current Models:
• Deep Learning Architectures
• Multilayer feedforward networks (Multilayer perceptrons)
• Radial Basis Function networks
• Self Organizing Networks
• ...