Audio Visual and Audio Tactile Interactions - Cross Modal Cognition - Lecture Notes, Study notes for Brain and Cognitive Science. Maulana Azad National Institute of Technology
dharmadaas
dharmadaas1 January 2013

Audio Visual and Audio Tactile Interactions - Cross Modal Cognition - Lecture Notes, Study notes for Brain and Cognitive Science. Maulana Azad National Institute of Technology

PDF (104 KB)
5 pages
1000+Number of visits
Description
Audio Visual and Audio Tactile Interactions, Auditory Tonotopic Maps, Crossmodal Maps, Superior Colliculus, Merging of The Senses, Visual Cortex, Early Visual Cortex, Giard and Peronnet, Introducing Massaro Baldi, Behavi...
20points
Download points needed to download
this document
Download the document
Preview3 pages / 5
This is only a preview
3 shown on 5 pages
Download the document
This is only a preview
3 shown on 5 pages
Download the document
This is only a preview
3 shown on 5 pages
Download the document
This is only a preview
3 shown on 5 pages
Download the document
Microsoft Word - CMCognition_Notes_Lecture4.doc

Lecture 4. Audio-visual and Audio-tactile Interactions (Dr. Jason Chan)

Auditory Tonotopic Maps • Each cell in MSO is tuned to particular delay  thus there’s an

auditory spatial map in MSO • This map is relayed to inferior and superior colliculus

Crossmodal maps • Located in the deep layers of the Superior Colliculus

– Spatial map of vision is eye centered – Spatial map of audition is head centered – Spatial map of touch is body centered

Superior Colliculus • SC has the spatial maps • The SC is crucial for crossmodal integration from the

various modalities. • Most single-cell studies focus on the SC

– (Stein & Meredith The merging of the senses.)

• 1/3 of the nerve cells the same sound produced a different rate of firing depending on where the eyes were pointing.

• Some neurons in inferior colliculus follow eye position (i.e. not head centred) Groh et al. Neuron, 2001

AV interactions in early visual cortex • Giard & Peronnet (1999) found visual effects of auditory

processing in the parieto-occipital scalp, 40ms after stimulus onset.

Giard & Peronnet. Jnl of Cog. Neuro (1999)

Docsity.com

AV interactions in early visual cortex • Giard & Peronnet (1999) found visual effects of auditory

processing in the parieto-occipital scalp, 40ms after stimulus onset.

• Molholm et al. (2002) replicated the findings of Giard & Peronnet. Participants responded to all auditory and visual stimuli.

Auditory interactions (parieto-occipital lobe) occurs early in processing

Behavioural implications

• Participants responded faster with simultaneous audio-visual stimuli, compared to auditory or visual alone.

Molholm et al., 2002

Early multisensory interactions Auditory effects of the visual system

• Amedi et al. (2003) found recruitment of area V1 for verbal processing in blind people.

• This was not found in sighted people

What is Lip reading? • Lip reading involves the extraction of speech from the visual clues

of the spoken message: the movements of the lips, the tongue, the lower jaw, the eyes, the eyebrows and the facial expression and gesture.

• For those who have a slight or moderate loss, lip reading can help considerably in 'receiving' the message. lip reading classes help communication and reduces feelings of isolation.

• For severely deaf or totally deaf people lipreading may be the only

Docsity.com

possible means of everyday communication.

Lip reading Viewing someone’s lips move during talking

without sound activates the brain areas involved in hearing speech

Calvert et al. (1997)

McGurk Effect • McGurk & MacDonald (1978) • Vision can influence what sound you perceive

– If the auditory stimulus is saying “ba” and the visual stimulus is saying “da”, people will hear “ga” or “da”

– An example of sensory fusion, not sensory dominance • If you are having difficulty hearing, watch their lips

McGurk Effect Introducing Massaro’s Baldi

Ventriloquist Effect An auditory stimulus is mislocalised towards the direction of an accompanying visual stimulus (Howard and Templeton, 1966)

Illusory Auditory Flash illusion • Shams, Kamitani & Shimojo (2000) • Present one visual flash, along with multiple auditory

beeps • Participants asked to determine how many visual flashes

– Participants reported see the same number of flashes as they

Docsity.com

heard beeps. • Illusion decays if auditory beeps separated by over 70 ms

http://neuro.caltech.edu/~kamitani/audiovisualRabbit/index.html

Audition also affects touch The Ventriloquist effect is not limited to audio- visual interactions Caclin et al. Perception & Psychophysics (2002)

Auditory capture of touch Bresciani et al. Exp. Brain Res. (2005)

Auditory-somatosensory processing

Foxe et al. (2002) used fMRI to investigate audio-somatosensory interactions.

• Found superadditive activation in brain area STG. STG receives feedforward projections from the auditory and somatosensory systems

Top-down interactions • Sekular, Sekular & Lau (1997) ‘bouncing ball’ illusion • Without the sound, participants perceived the two balls as either passing each

other or bouncing – i.e. no bias in the illusion. • When presenting a tone at the moment of ‘impact’, participants consistently

perceived the balls as bouncing off each other – i.e. a bias was produced

• This also works in the tactile-visual modality

Changing perception of skin through sound?

Parchment skin illusion

Docsity.com

Jousmäki V, Hari R. (1998) Current biology

What you hear affects what you touch Zampini & Spence. Journal of Sensory Studies (2004)

Recap and conclusions • Integration occurs early in the brain. • Visual areas can be recruited for auditory processing in the

blind. • Reliability of the sensory signal leads to sensory

dominance • Reliable spatial cues: Vision and touch

– Vision affects the perception of sound • lip reading • Ventriloquist effect • McGurk Effect

Recap and conclusions • Reliable temporal cues: Audition

– Behaviourally, sound affects the perception of vision and touch • Bouncing ball illusion • Shams illusion

• Early somato-auditory integration – Behaviourally, sound affects the perception of touch (texture

discrimination) – Modulating phase in A1

• Higher associative area STG involved in somato-auditory integration

Docsity.com

comments (0)
no comments were posted
be the one to write the first!
This is only a preview
3 shown on 5 pages
Download the document