Neural Network 1 , Lecture Notes - Computer Science, Study notes of Artificial Intelligence

Download Neural Network 1 , Lecture Notes - Computer Science and more Study notes Artificial Intelligence in PDF only on Docsity!

CS181 Lecture 5:

Neural Networks I:

Perceptrons

Prof. David C. Parkes

Artificial Neural Network

• A computational structure formed from a network of individual processing units that send messages to each other
• Individual unit = neuron

Neuron in the Brain

(Russell and Norvig)

Key Characteristics

• Connectivity structure
• Nature and strength of reactions at synapses
• Some synapse connections are “excitatory” others are “inhibitory”

Limitations of Analogy

• ANNs do not model architecture of brain
• ANNs typically feed forward and not „deep‟, although this is changing

Computational Properties of Brain

• Massive parallelism
  • 1 kHz versus 10 GHz clock speed
  • but 10^11 neurons; 10^14 synapses ¼ 1017 operations/sec
  • storage roughly 10^11 “RAM” and 10^13 “disk”
• Graceful degradation
• Plasticity (= “long-term learning”)
  • both in strength of connections and in the structure of network

Aside: Gradient Descent

Aside: Gradient Descent

Aside: Gradient Descentll

Aside: Gradient Descent

Aside: Gradient Descent

Follows direction of steepest descent

Reaches local minimum 17

Aside: Gradient Descent

Graph of error function direction of steepest descent

initial weights

new weights 18

Perceptron learning rule

(Russell and Norvig)

stochastic update (a) separable data, (b) non-separable, (c) non-separable + weight decay