back propagation for, Study Guides, Projects, Research of Computer science

Download back propagation for and more Study Guides, Projects, Research Computer science in PDF only on Docsity!

DS 4400

David Liu

Khoury College of Computer Science

Northeastern University

April 2 2024

Machine Learning and Data Mining I

Spring 2024

Outline

• Training Neural Networks
  • Demo
  • Backpropagation
  • Parameter Initialization
  • Derivation for feed-forward neural network for

binary classification (sigmoid activation)

• Stochastic Gradient Descent
  • Gradient descent variants
• Regularization methods for neural networks
  • Weight decay
  • Dropout

Reminder: Logistic Regression

Cross-entropy loss

N

N

[𝑦

log ℎ

)log 1 − ℎ

]

loss

loss

Gradient Descent

• Converges for convex objective
• Could get stuck in local minimum for non-convex objectives

Training Neural Networks

• Training data 𝑥

!

, 𝑦

!

, … 𝑥

"

, 𝑦

"

• Training example 𝑥

= 𝑥

#!

, … 𝑥

#$

, label 𝑦

• One forward pass through the network
  • Compute prediction !𝑦

= ℎ(𝑥

)

• Loss function for each example
  • 𝐿 𝑦, 𝑦 = −[ 1 − 𝑦 log 1 − 𝑦 + 𝑦 log 𝑦*]
• Loss function for training data
  • 𝐽 𝑊, 𝑏 =

∑

𝐿 ( 𝑦!

, 𝑦

)

Cross-entropy loss

GD for Neural Networks

• Initialization
  • For all layers ℓ
    • Initialize 𝑊

[ℓ]

[ℓ]

• Backpropagation
  • Fix learning rate 𝛼
  • Repeat
    • For all layers ℓ

GD for Neural Networks

• Initialization
  • For all layers ℓ
    • Set 𝑊

[ℓ]

[ℓ]

at random

• Backpropagation
  • Fix learning rate 𝛼
  • Repeat
    • For all layers ℓ (starting backwards)
    • 𝑊

[ℓ]

[ℓ]

$%&

' ()(

,

!

,,

!

)

(/

ℓ

• 𝑏

[ℓ]

[ℓ]

$%&

' ()(

,

!

,,

!

)

( 0

ℓ

This is

expensive!

Stochastic Gradient Descent

• Initialization
  • For all layers ℓ
    • Set 𝑊

[ℓ]

[ℓ]

at random

• Backpropagation
  • Fix learning rate 𝛼
  • Repeat
    • For all layers ℓ (starting backwards)
      • For all training examples 𝑥

$

$

[ℓ]

[ℓ]

$

$

ℓ

[ℓ]

[ℓ]

$

$

ℓ

Incremental

version of GD

Gradient Descent Variants

Review of a Perceptron

1

Backpropagation Intuition

𝑧

$

[$]

→ 𝑎

$

[$]

𝛿

$

[$]

𝑧

'

[$]

→ 𝑎

'

[$]

𝛿

'

[$]

𝑧

$

[']

→ 𝑎

$

[']

𝛿

$

[']

𝑧

'

[']

→ 𝑎

'

[']

𝛿

'

(')

𝑧

$

[(]

→ 𝑎

$

[(]

𝛿

$

(()

&

[ 6 ]

&

[ 6 ]

7&

[ 6 ]

7

[ 7 ]

&

[ 6 ]

W

7&

[ 6 ]

𝛿

$

[(]

𝛿

'

[']

Backpropagation Intuition

𝑧

$

[$]

→ 𝑎

$

[$]

𝛿

$

[$]

𝑧

'

[$]

→ 𝑎

'

[$]

𝛿

'

($)

𝑧

$

[']

→ 𝑎

$

[']

𝛿

$

[']

𝑧

'

[']

→ 𝑎

'

[']

𝛿

'

(')

𝑧

$

[(]

→ 𝑎

$

[(]

𝛿

$

[()

7

[&]

7&

[ 7 ]

&

[ 7 ]

77

[ 7 ]

7

[ 7 ]

7&

[ 6 ]

𝛿

$

[(]

𝛿

'

[']

77

[ 7 ]

7&

[ 7 ]

𝛿

'

[$]