Distributed Representations - Artificial Neural Network - Lecture Slides, Slides of Computer Networks

Download Distributed Representations - Artificial Neural Network - Lecture Slides and more Slides Computer Networks in PDF only on Docsity!

Introduction to Neural Networks and Machine

Learning

Lecture 5: Distributed Representations

Localist representations

• The simplest way to represent things with neural networks is to dedicate one neuron to each thing. - Easy to understand. - Easy to code by hand - Often used to represent inputs to a net - Easy to learn - This is what mixture models do. - Each cluster corresponds to one neuron - Easy to associate with other representations or responses.
• But localist models are very inefficient whenever the data has componential structure.

Using simultaneity to bind things together

Represent conjunctions by activating all the constituents at the same time.

• This doesn’t require connections between the constituents.
• But what if we want to represent yellow triangle and blue circle at the same time?

Maybe this explains the serial nature of consciousness.

• And maybe it doesn’t!

color neurons

shape neurons

Using space to bind things together

• Conventional computers can bind things together

by putting them into neighboring memory

locations.

• This works nicely in vision. Surfaces are generally

opaque, so we only get to see one thing at each

location in the visual field.

• If we use topographic maps for different properties, we can assume that properties at the same location belong to the same thing.

Coarse coding

• Using one neuron per entity is inefficient.

• An efficient code would have each neuron active

half the time.

• This might be inefficient for other purposes (like associating responses with representations).

• Can we get accurate representations by using

lots of inaccurate neurons?

• If we can it would be very robust against hardware

failure.

Coarse coding

Use three overlapping arrays of large cells to get an array of fine cells

• If a point falls in a fine cell, code it by activating 3 coarse cells.
• This is more efficient than using a neuron for each fine cell.
• It loses by needing 3 arrays
• It wins by a factor of 3x3 per array
• Overall it wins by a factor of 3

Coarse regions and fine regions use the same surface

1

1 1

−

− −

k

k k

r

R

c

C

N

n

total boundary cnr CNR

• Each binary neuron defines a boundary between k-dimensional points that activate it and points that don’t. - To get lots of small regions we need a lot of boundary.

fine coarse

constant

saving in neurons without loss of accuracy

ratio of radii of fine and coarse fields

Limitations of coarse coding

• It achieves accuracy at the cost of resolution
  • Accuracy is defined by how much a point must be moved before the representation changes.
  • Resolution is defined by how close points can be and still be distinguished in the represention. - Representations can overlap and still be decoded if we allow integer activities of more than 1.
• It makes it difficult to associate very different responses with similar points, because their representations overlap - This is useful for generalization.
• The boundary effects dominate when the fields are very big.

The Effects of Brain Damage

• Performance deteriorates in some unexpected ways when the brain is damaged. This can tell us a lot about how information is processed. - Damage to the right hemisphere can cause neglect of the left half of visual space and a lack of a sense of ownership of body parts. - Damage to parts of the infero-temporal cortex can prevent face recognition. - Damage to other areas can destroy the perception of color or of motion.
• Before brain scans, the performance deficits caused by physical damage were the main way to localize functions in the human brain - recording from human brain cells is not usually allowed (but it can give surprising results!).

Acquired dyslexia

• Occasionally, damage to the brain of an adult

causes bizarre reading deficits

• Surface dyslexics can read regular nonsense

words like “mave” but mispronounce irregular

words like “yacht”.

• Deep dyslexics cannot deal with nonsense words

at all. They can read “yacht” correctly sometimes

but sometimes misread “yacht” as “boat”. They

are also much better at concrete nouns than at

abstract nouns (like “peace”) or verbs.

The dual route theory of reading

• Marshall and Newcombe proposed that there are two routes that can be separately damaged. - Deep dyslexics have lost the phonological route and may also have damage to the semantic route.
• But there are consistent peculiarities that are hard to explain this way.

PEACH

The meaning of the word

The sound of the word

Speech production

visual features of the word

An advantage of neural network models

• Until quite recently, nearly all the models of information processing that psychologists used were inadequate for explaining the effects of damage. - Either they were symbol processing models that had no direct relationship to hardware - Or they were just vague descriptions that could not actually do the information processing.
• There is no easy way to make detailed predictions of how hardware damage will affect performance in models of this type.
• Neural net models have several advantages:
  • They actually do the required information processing rather than just describing it.
  • They can be physically damaged and the effects can be observed.

The equivalence between layered, feedforward

nets and recurrent nets

w 1 w 2

w 3 w 4

w (^1) w 3 w4 w 2

w (^1) w 3 w4 w 2

w (^1) w 3 w4 w 2

time=

time=

time=

time=

Assume that there is a time delay of 1 in using each connection. The recurrent net is just a layered net that keeps reusing the same weights.

What the network learns

• We used recurrent back-propagation for six time steps with the sememe vector as the desired output for the last 3 time steps. - The network creates semantic attractors. - Each word meaning is a point in semantic space and has its own basin of attraction. - Damage to the sememe or clean-up units can change the boundaries of the attractors. - This explains semantic errors. Meanings fall into a neighboring attractor. - Damage to the bottom-up input can change the initial conditions for the attractors. - This explains why early damage can cause semantic errors.