NEURO2020 WEEK 7

PERCEPTION

• Determine what is 'out there' 

• Decide on best/most appropriate behaviour 

• Approach food / avoid danger / navigate path / interact with mater

Survival depends on speed and accuracy with which an organism can evaluate external events and properly react to them.  

MULTISENSORY PERCEPTION

Objects and events in the world generate information in multiple ways, enhancing our ability to perceive and understand our environment, enabling us to interact better with our surroundings.  

• Light  

• Sound  

• Mechanical 

• Chemical 

MULTISENSORY BENEFITS/WORKINGS

• Different modalities can substitute when individually compromised 

• Different fields of operation (touch/smell/taste for close up, vision/hearing for distance) 

• Each modality tends to be best suited to a particular task (visual – spatial, auditory – temporal) 

• Resolves ambiguities (boost signal to noise) 

• Combine information from multiple cues to improve stimulus detection and discrimination 

REDUNDANT TARGET EFFECTS

• Speeded response to either audio (A), visual (V), or audio-visual (AV) target.  

• Auditory and visual reaction time on its own, was longer than audio-visual together.  

• Responses of redundant signals are too fast to be explained as the faster of two responses to individual signals (statistical facilitation) 

• Assume that signals jointly contribute to process of producing response (MSI – neural coactivation) 

REDUNDANT TARGET EFFECTS - STATISTICAL FACILITATION (independent processing)

• Both elements of the auditory/visual stimulus are processed along independent channels 

• One that reaches output stage first, triggers the response 

REDUNDANT TARGET EFFECTS - NEURAL COACTIVATION (integrated signals)

• Both components of a redundant signal influence response on a single trial 

• Activation from different channels combine in satisfying a single criterion for response initiation – activation builds up over time until some criterion is reached 

• Activation builds faster when provided by two sources versus one 

FACTORS INFLUENCING MULTISENSORY INTEGRATION

Ability to dissociate between stimuli from different sources and single source 

• Temporal coincidence (bottom-up) 

• Spatial coincidence (bottom-up) 

• Temporal patterning (bottom-up) 

• Cross-modal correspondence (objective) 

• Stored knowledge (subjective) 

• Recent experience/context/expectation/attention (subjective) 

MULTIMODAL BRAIN AREAS - COLLICULUS

Superior Colliculus 

• Integrating auditory and visual info

• orienting to stimuli 

• Produces motor actions guided by sensory stimuli 

• Programming eye movement 

• Inputs = retina, cortex, inferior colliculus, spinal cord 

• Outputs = motor control of eyes, ears, head 

• Superficial layers are visual, deeper layers are multi-sensory 

Inferior Colliculus  

• Organising auditory information 

MULTIMODAL BRAIN AREAS - SUPERIOR TEMPORAL SULCUS

• Perception of language 

• Integrate visual and auditory information 

• Weights information on quality of each 

• Audio-visual speech perception (McGurk effect) 

MULTISENSORY ENHANCEMENT

• Multisensory neurons response to appropriate multisensory stimuli exceeds the response to individual uni-sensory inputs

• Get suppression of inappropriate stimuli  

MULTISENSORY ENHANCEMENT RULES

1. Spatial rule  

   • MS stimuli must occur at the same region of space 

   • Two stimuli in same receptive field of MS cell 

2. Temporal rule  

   • MS stimuli must reach the MS cell at the same time 

   • Maximal response when no time delay  

3. Principle of inverse effectiveness 

   • Enhancement is greater for weak stimuli than for strong stimuli 

VESTIBULAR SYSTEM

• Cochlea  

• 3 semicircular canals 

• Ampullae  

• Utricle & saccule (otoliths) 

RESPONSE OF NERVE FROM SEMICIRCULAR CANAL

• Hair cells have baseline firing rate at rest 

• When head rotates in one direction, activity spikes, activity decreases/stops in other direction.  

• Bidirectional deviation from baseline encodes direction of rotation 

• Fluid eventually catches up with rotation and signal is transient  

• Reflects acceleration not constant motion 

RESPONSE OF OTOLITHS

• Sensitive to linear acceleration & orientation relative to gravity  

• Tilting chin up causes spike in activity, tilting down causes decrease 

• Allows brain to infer head orientation 

OTOLITH LIMITATIONS

• Same otolith signal can be produced by forward acceleration and backward tilt  

• Signal ambiguous without visual input  

VESTIBULO-OCCULAR REFLEX (VOR)

• Wiring between vestibular system and eye muscles that produce compensatory eye movement every time the head moves 

Reflex arc (left head turn) 

1. Left horizontal semicircular canal detects rotation 

2. Sends excitatory signal to lateral rectus of right eye (to pull it right) 

3. Complementary signal sent to medial rectus of left eye 

4. The two eyes move together to the right, compensating for the leftward head turn 

MULTISENSORY CONFLICT

When two modalities provide differing spatial/temporal information, vision will override the other.

SPATIAL VENTRILOQUISM EFFECT

• Visual + auditory info conflict spatially but match temporally and semantically

• The brain assumes vision is correct and shifts the perceived location of the sound

• small disparity of audio and visual target leads to almost complete overlap in perceived location

COLAVITA VISUAL DOMINANCE EFFECT

• When presented with audio-visual stimuli, they underreport the auditory stimuli in the audio-visual target

• frequently only respond to visual modality when presented with both

• vision and audio can be characterised as competing for attention, and the brain will favour vision

MCGURK EFFECT

Conflicting visual and audio information produces a different sound from original audio stimulus

OUTCOMES OF MCGURK EFFECT (FOR AUDIO BA, VISUAL DA)

1. FUSION - vision + sound fuse into new percept (ga)

2. COMBINATION - vision + sound combined but seperate (bada, daba)

3. VISUAL DOMINANCE - visual stimulus dominates percept (da)

4. AUDITORY DOMINANCE - audio stimulus dominates percept (ba)

MORE MCGURK EFFECT Fusion

• spatial + temporal coincidence of lips + sound source = stronger fusion effect

• fusion due to long-term effects of observing speech from mouths

• both information is relevant, so fusion makes the best match

RELIABILITY DOMINANCE

When two multisensory signals conflict, the brain will favour the more accurate/reliable one.

• therefore, vision dominates because it is reliable and degrading the audio stimulus (using ambiguous consanants) makes it more reliable

IT IS NOT ABOUT VISUAL DOMINANCE BUT RELIABILITY DOMINANCE

SOUND INDUCED FLASH ILLUSION

When visual flashes paired with beep, the number of beeps will alter the perceived number of visual flashes.

• presence of 2+ beeps creates illusion of multiple visual flashes, but not linearly

• shows there is a limit

TEMPORAL VS SPATIAL

Vision has better spatial/object detail acuity, while audition has better timing acuity.

eyes to visual cortex = 70ms

ears to auditory cortex = 30ms

TEMPORAL VENTRILOQUISM EFFECT

Auditory stimuli can shift the perception of a visual stimulus closer to it in time.

REACTION TIMES ARE FASTER WHEN TWO SENSORY SIGNALS ARE PRESENTED TOGETHER INSTEAD OF ONE:

Redundant target effects