Machine Learning Algorithms, Study notes of Machine Learning

Download Machine Learning Algorithms and more Study notes Machine Learning in PDF only on Docsity!

CS260: Machine Learning Algorithms

Lecture 10: Neural Networks

Cho-Jui Hsieh

UCLA

Feb 20, 2019

Neural Networks

Another way to introduce nonlinearity

How to generate this nonlinear hypothesis?

Combining multiple linear hyperplanes to construct nonlinear hypothesis

Neural Network

Input layer: d neurons (input features)

Neurons from layer 1 to L: Linear combination of previous layers +

activation function

θ(w

T

x), θ : activation function

Final layer: one neuron ⇒ prediction by sign(h(x))

Formal Definitions

w

(l)

ij

1 ≤ l ≤ L : layers

0 ≤ i ≤ d

(l−1) : inputs

1 ≤ j ≤ d

(l) : outputs

Formal Definitions

w

(l)

ij

1 ≤ l ≤ L : layers

0 ≤ i ≤ d

(l−1) : inputs

1 ≤ j ≤ d

(l) : outputs

j-th neuron in the l-the layer:

x

(l)

j

= θ(s

(l)

j

) = θ(

d

(l−1)

∑

i=

w

(l)

ij

x

(l−1)

i

Forward propagation

Forward propagation

Forward propagation

Forward propagation

Forward propagation

Forward propagation

Forward propagation

Stochastic Gradient Descent

All the weights W = {W 1

, · · · , W

L

} determine h(x)