1
CogNova
Technologies
Theory and Application of
Theory and Application of
Artificial Neural Networks
Artificial Neural Networks
with
with
Daniel L. Silver, PhD
Daniel L. Silver, PhD
Copyright (c), 2014
All Rights Reserved
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Guidelines and tips
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Natural language processing (NLP)., Thesis of Computer-Aided Power System Analysis
Natural language processing (NLP)........
Typology: Thesis
2018/2019
1 / 101
This page cannot be seen from the preview
Don't miss anything!
Related documents
Partial preview of the text
Download Natural language processing (NLP). and more Thesis Computer-Aided Power System Analysis in PDF only on Docsity!
CogNova
Theory and Application of
Theory and Application of
Artificial Neural Networks
Artificial Neural Networks
with with
Daniel L. Silver, PhD
Daniel L. Silver, PhD
Copyright (c), 2014
All Rights Reserved
CogNova
Seminar Outline
Seminar Outline
DAY 1
DAY 1
ANN Background and Motivation
ANN Background and Motivation
Classification Systems and
Classification Systems and Inductive
Inductive
Learning Learning
From Biological to Artificial Neurons
From Biological to Artificial Neurons
Learning in a Simple Neuron
Learning in a Simple Neuron
Limitations of Simple Neural Networks
Limitations of Simple Neural Networks
Visualizing the Learning ProcessVisualizing the Learning Process
Multi-layer Feed-forward ANNs
Multi-layer Feed-forward ANNs
The Back-propagation Algorithm
The Back-propagation Algorithm
DAY 2 DAY 2
Generalization in ANNs
Generalization in ANNs
How to Design a Network
How to Design a Network
How to Train a NetworkHow to Train a Network
Mastering ANN Parameters
Mastering ANN Parameters
The Training Data
The Training Data
Post-Training AnalysisPost-Training Analysis
Pros and Cons of Back-propPros and Cons of Back-prop
Advanced issues and networks
Advanced issues and networks
CogNova
Background and
Background and
Motivation
Motivation
Growth has been explosive since 1987
Growth has been explosive since 1987
education institutions, industry, military
education institutions, industry, military
> 500 books on subject
> 500 books on subject
> 20 journals dedicated to ANNs
> 20 journals dedicated to ANNs
numerous popular, industry, academic
numerous popular, industry, academic
articles
articles
Truly inter-disciplinary area of study
Truly inter-disciplinary area of study
No longer a
No longer a
flash in the pan
flash in the pan
technology
technology
CogNova
Background and
Background and
Motivation
Motivation
Computers and the Brain: A Contrast
Computers and the Brain: A Contrast
Arithmetic:
Arithmetic: 1 brain = 1/10 pocket calculator
1 brain = 1/10 pocket calculator
Vision:
Vision: 1 brain = 1000 super computers
1 brain = 1000 super computers
Memory of arbitrary details: computer wins
Memory of arbitrary details: computer wins
Memory of real-world facts: brain wins
Memory of real-world facts: brain wins
A computer must be programmed explicitly
A computer must be programmed explicitly
The brain can learn by experiencing the world
The brain can learn by experiencing the world
CogNova
Background and
Background and
Motivation
Motivation
Inherent Advantages of the Brain:
Inherent Advantages of the Brain:
distributed processing and
distributed processing and
representation
representation ”
Parallel processing speeds
Parallel processing speeds
Fault tolerance
Fault tolerance
Graceful degradation
Graceful degradation
Ability to generalize
Ability to generalize
I
O
f(x)
x
CogNova
Background and
Background and
Motivation
Motivation
History of Artificial Neural Networks
History of Artificial Neural Networks
Creation:
Creation:
1890: William James - defined a neuronal process of learning
1890: William James - defined a neuronal process of learning
Promising Technology:
Promising Technology:
1943: McCulloch and Pitts - earliest mathematical models
1943: McCulloch and Pitts - earliest mathematical models
1954: Donald Hebb and IBM research group - earliest simulations
1954: Donald Hebb and IBM research group - earliest simulations
1958: Frank Rosenblatt - The Perceptron
1958: Frank Rosenblatt - The Perceptron
Disenchantment:
Disenchantment:
1969: Minsky and Papert - perceptrons have severe limitations
1969: Minsky and Papert - perceptrons have severe limitations
Re-emergence:
Re-emergence:
1985: Multi-layer nets that use back-propagation 1985: Multi-layer nets that use back-propagation
1986: PDP Research Group - multi-disciplined approach 1986: PDP Research Group - multi-disciplined approach
CogNova
Classification Systems
Classification Systems
and Inductive Learning
and Inductive Learning
CogNova
Classification Systems
Classification Systems
and Inductive Learning
and Inductive Learning
Basic Framework for Inductive Learning
Basic Framework for Inductive Learning
Inductive
Learning System
Environment
Training
Examples
Testing
Examples
Induced
Model of
Classifier
Output Classification
(x, f(x))
(x, h(x))
h(x) = f(x)?
A problem of representation and
search for the best hypothesis, h(x).
CogNova
Classification Systems
Classification Systems
and Inductive Learning
and Inductive Learning
Vector
Vector Representation & Discriminate Functions
Representation & Discriminate Functions
x
x
Height
Age
2
1
o
o
o
o
o
A
B
Linear Discriminate
Function
f(X)=(x
1
,x
2
w
0
+w
1
x
1
+w
2
x
2
or WX = 0
f(x
1
,x
2
) > 0 => A
f(x
1
,x
2
) < 0 => B
-w
0
/w
2
CogNova
Classification Systems
Classification Systems
and Inductive Learning
and Inductive Learning
f(X) = WX =
f(X) = WX = will discriminate
will discriminate
class A from B,
class A from B,
BUT ... we do not know the
BUT ... we do not know the
appropriate values for :
appropriate values for :
w
0
, w
1
, w
2
CogNova
From Biological to Artificial
From Biological to Artificial
Neurons
Neurons
CogNova
From Biological to Artificial
From Biological to Artificial
Neurons
Neurons
The Neuron - A Biological Information Processor
The Neuron - A Biological Information Processor
dentrites
dentrites
- the receivers
- the receivers
soma
soma
- neuron cell body (sums input signals)
- neuron cell body (sums input signals)
axon
axon
- the transmitter
- the transmitter
synapse
synapse
- point of transmission
- point of transmission
neuron activates after a certain
neuron activates after a certain threshold
threshold is
is
met
met
Learning occurs via electro-chemical changes in
Learning occurs via electro-chemical changes in
effectiveness of
effectiveness of synaptic junction
synaptic junction .
.
CogNova
From Biological to Artificial
From Biological to Artificial
Neurons
Neurons
An Artificial Neuron - The Perceptron
An Artificial Neuron - The Perceptron
Basic function of neuron is to sum inputs, and
Basic function of neuron is to sum inputs, and
produce output given sum is greater than threshold
produce output given sum is greater than threshold
ANN node produces an output as follows:
ANN node produces an output as follows:
Multiplies each component of the input pattern
Multiplies each component of the input pattern
by the weight of its connection
by the weight of its connection
Sums all weighted inputs and subtracts the
Sums all weighted inputs and subtracts the
threshold value =>
threshold value => total weighted input
total weighted input
Transforms the total weighted input into the
Transforms the total weighted input into the
output using the activation function
output using the activation function
CogNova
From Biological to Artificial
From Biological to Artificial
Neurons
Neurons
Hidden Nodes
Output Nodes
Input Nodes
I1 I2 I3 I
O1 O
Distributed processing
Distributed processing
and representation
and representation ”
3-Layer Network
has
2 active layers