11
1
Backpropagation
Docsity.com
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Guidelines and tips
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Backpropagation Algorithm in Neural Networks, Slides of Banking and Finance
An in-depth explanation of the backpropagation algorithm used in neural networks. It includes diagrams and formulas for calculating sensitivities, weight updates, and transfer function derivatives. The document also includes an example of function approximation using a sine wave network.
Typology: Slides
2012/2013
1 / 29
This page cannot be seen from the preview
Don't miss anything!
Related documents
Partial preview of the text
Download Backpropagation Algorithm in Neural Networks and more Slides Banking and Finance in PDF only on Docsity!
Backpropagation
Docsity.co
Multilayer Perceptron
R – S
1
- S
2
- S
3
Network
Docsity.co
Elementary Decision Boundaries
First Subnetwork
First Boundary:
a
1 1
hardlim
1
0
p
(
)
=
Second Boundary:
1
2
a
2 1
hardlim
0
1
p
(
)
=
p
1
a
1 2
n
1
2
Inputs
p 2 - 1 a 1
1
n
1
1
a
2
1
n
2
1
1
1
AAAA
Σ
AA
Σ
AA
Σ
AA
1
AA AAAA
0
0
1
1 1
Individual Decisions
AND Operation
Docsity.co
Elementary Decision Boundaries
3
4
Fourth Boundary:Third Boundary:
Second Subnetwork
a
3 1
hardlim
1 0
p
(
)
=
a
4 1
hardlim
0 1
p
(
)
=
p
1
a
1
4
n
1
4
Inputs
p
2
1
a
1
3
n
1
3
a
2
2
n
2
2
1
1
AAAA
Σ
AA
Σ
AA
Σ
AA
1
AA AAAA
0
0 1
1 1
Individual Decisions
AND Operation
Docsity.co
Function Approximation Example
p
a
1 2
n
1
2
Input
w
1
1,
a
1
1
n
1
1
w
2
1,
b
1 2
b
1 1
b
2
a
2
n
2
1 1
1
AAAA
Σ
AAAA
Σ
AA
Σ
w
1
2,
w
2
1,
AAAA AAAA
Log-Sigmoid Layer
AA
Linear Layer
a
1
=
logsig
(
W
1
p
b
1
)
a
2
=
purelin
(
W
2
a
1
b
2
)
f
1
n
e
n
f
2
n
(
)
n
=
w
1 1
,
1
w
2 1
,
1
b
1 1
b
2 1
w
1 1
,
2
w
1 2
,
2
b
2
0
=
Nominal Parameter Values
Docsity.co
Nominal Response
0
1
2
0 1 2 3
Docsity.co
Multilayer Network
a
m
1
f
m
1
W
m
1
+ a m b m 1
(
)
=
m
0 2
…
M
1
,
,
,
=
a
0
p
=
a
a
M
=
Docsity.co
Performance Index
p
1
t
1
{ , } p 2 t 2
{ , } … p Q t Q
{
,
}
,
,
,
Training Set
F
x
(
)
E e
2
]
[
=
E
t
a
(
)
2
]
[
=
Mean Square Error
F
x
( ) E e T e ]
[
=
E
t
a
(
)
T
t
a
(
) ]
[
=
Vector Case
F ˆ
x
(
)
t
k
(
)
a
k
(
)
(
)
T
t
k
(
)
a
k
(
)
(
)
e
T
k
(
)
e
k
(
)
=
=
Approximate Mean Square Error (Single Sample)
w
i
j
,
m
k
w
i j
,
m
k
α
F
w ∂
i
j
,
m
b
i m
k
b
i m
k
α
F
b
i m
Approximate Steepest Descent
Docsity.co
Gradient Calculation
n
i m
w
i
j
,
m
a
j m
1
j
1
=
S
m
1
∑
b
i m
n
i m
w
i
j
,
m
a
j m
1
n
i m
b
i m
s
i m
F
n
i m
Sensitivity
F
w ∂
i
j
,
m
s
i m
a
j
m
1
F
b
i m
s
i m
Gradient
Docsity.co
Steepest Descent
w
i j
,
m
k
w
i j
,
m
k
α
s
i m
a
j m
1
b
i m
k
b
i m
k
α
s
i m
W
m
k
1
(
)
W
m
k
( ) α s m a m 1
(
)
T
=
b
m
k
1
(
)
b
m
k
(
)
α
s
m
=
s
m
F
n
m
F
n
1 m
F
n
2 m
F …
n
S
m
m
Next Step: Compute the Sensitivities (Backpropagation)
Docsity.co
11
Backpropagation (Sensitivities)
s
m
F
n
m
n
m
1
n
m
T
F
n
m
1
F
˙
m
n
m
W
m
1
T
F
n
m
1
s
m
F
m
n
m
W
m
1
( ) T s m 1
The sensitivities are computed by starting at the last layer, and
then propagating backwards through the network to the first layer.
s
M
s
M
1
- … s 2 s 1
→
→
→
→
Docsity.co
Initialization (Last Layer)
s
i M
F
n
i M
t
a
T
t
a
n
i M
t
j
a
j
2
j
1
=
S
M
∑
n
i M
t
i
a
i
a
i
n
i M
s
M
2
F
M
n
M
(
)
t
a
(
)
=
a
i
n
i M
a
i M
n
i M
f
M
n
i
M
n
i M
f
˙
M
n
i
M
s
i M 2 t i a i
f
M
n
i M
Docsity.co
Example: Function Approximation
g p
4 --π
p
sin
Network
t
a
e
p
Docsity.co
Network
p
a
1 2
n
1
2
Input
w
1
1,
a
1
1
n
1
1
w
2
1,
b
1 2
b
1 1
b
2
a
2
n
2
1 1
1
AAAA
Σ
AAAA
Σ
AA
Σ
w
1
2,
w
2
1,
AAAA AAAA
Log-Sigmoid Layer
AA
Linear Layer
a
1
=
logsig
(
W
1
p
b
1
)
a
2
=
purelin
(
W
2
a
1
b
2
)
Network
a
p
Docsity.co