20 - Multisensory Integration

What is multisensory association/integration?

neural process by which the brain combines information from 2+ sensory modalities into a single, unified perceptual experience

Cocktail party effect as an example of multisensory integration

integration of sight and sound → in a loud room, may be hard to isolate a single voice. Looking at the speaker’s mouth provides a visual timeline of when sounds should occur, allowing your brain to filter out background noise

benefits of multisensory integration?

faster/more accurate stimuli detection and responses

noise reduction

enhance perception of stimuli/experience

core principles of multisensory integration that generally explain rules to determine if inputs belong to the same event

spatial principle

temporal principle

inverse effectiveness principle

spatial principle

integration is stronger when the different sensory stimuli occur in the same physical location in space

temporal principle

integration is stronger when the different sensory stimuli occur at or very close to the same time or within a narrow temporal window (‘temporal binding window’)

inverse effectiveness principle

integration is most effective when the individual sensory stimuli are weak or ambiguous on their own (ex, relying more on combining sight and sound to understand someone in a noisy room than in a quiet one)

what are multisensory neurons

specialized neurons found throughout the CNS that respond to inputs from mutliple senses

multisensory neurons combine inputs so that their response is…

statistically different - typically enhanced, sometimes depressed - from the sum of unisensory responses (as opposed to simply adding up inputs from neurons conveying signals from individual sensory pathways).

spatiotemporally concordant stimuli

stimuli from multiple modalities that occur in the same place at the same time → will have an enhanced response in neurons compared to just the sum of the stimuli

how is it proposed that the enhanced computation of responses is performed (total > sum of parts)?

Bayesian Causal Interference model

what is the Maximum Likelihood Estimation (MLE)

A neuron must first implicitly infer whether 2 signals share a common cause. If they do optimal integration follows reliability-weighted averaging → more reliable (lower-variance) modality gets more weight

multisensory neurons calculate the optimal integrated estimate how? equation?

Sva = WvSv + WaSa → Sva is optimal integrated estimate calculated by weighting different stimuli relative to their reliability (low-variance). equation example is for visual Sv and auditory Sa signals weighted by relative reliabilities Wv and Wa. Weights are inversely proportional to variance (chi²) of each sense

causal interference model in simple terms

computes a probabilistic estimate of ‘same cause’ vs. ‘different causes’ and blends full integration with segregation accordingly

multisensory receptive field

area of space where stimuli can trigger action potentials in a single neuron via multiple distinct sensory systems (ex, visuo-auditory neurons in superior colliculus)

key brain areas of multisensory integration

superior colliculus

association cortices

primary sensory cortices (?)

how is superior colliculus key for multisensory integration?

combines visual, auditory, and somatosensory (tactile) signals to direct attention and eye movements, and orientation towards/away from stimuli

how are association cortices key for multisensory integration?

high-level cortical areas (ex, superior temporal sulcus STS or parietal cortex) blend complex sensory streams

how are primary sensory cortices key for multisensory integration?

modern research shows that early ‘unisensory/unimodal’ areas (like primary visual cortex) receive feedback and cross-modal inputs from other senses → in this way, can they be considered invovled in multisensory integration?

how is sensory informationprocessed along several parallel pathways?

primary sensory cortices & unimodal association cortex → multimodal association cortex (in posterior parietal and temporal cortices)

where does sensory info representing different modalities converge?

upon areas of the cortex that integrate that information (unimodal and multimodal association cortices) in a multi-sensory event

what are the dark blue, pale blue, and white areas?

white: primary cortices

pale blue: unimodal association

dark blue: multimodal association / higher-order association cortices

the ____ areas that process sensory information are highly interconnected with the ___ areas responsible for planning motor actions

posterior association; frontal association

hierarchical model information processing info flow

primary sensory areas → unimodal sensory areas → multimodal sensory areas

what do unimodal areas do

integrate information from a single sensory modality (ex, shape/size/texture of an object are processed in unimodal somatosensory cortical areas)

what do multimodal areas do

integrate information form more than one sensory system (texture, color, shape, motion, taste, etc)

area 1 (somatosensory and visual cortices)

unimodal → processes texture

area 2 (somatosensory and visual cortices)

unimodal → processes shape and size

area 7 (somatosensory and visual cortices)

multimodal → integrates somatosensory information with other modalities (motion, vision, etc)

area 3a somatosensory

processes proprioceptive input

area 3b primary somatosensory cortex

receives most of the discriminative-touch input

V2 (somatosensory and visual cortices)

definition of contours, spatial orientation of contours, binocular disparity, object/background distinction

V3 (somatosensory and visual cortices)

global/coherent motion in the visual field

MT (somatosensory and visual cortices)

strong direction sensitivity

V4 (somatosensory and visual cortices)

responds to objects of intermediate complexity (simple geometric shapes, colors)

IT (somatosensory and visual cortices)

responds to complex objects, ie face perception (compare to V4)

unimodal sensory inputs converge on multimodal association areas in the …. cortices

prefrontal, parietal, temporal, and limbic

posterior association area → type of convergence, what info it deals with?

parieto-temporal convergence

concerned with extra-personal space (defining spatial relationships with the world around us) and binding individual elements of a visual scene into a coherent scene (visuo-spatial integration)

limbic association area info dealt with?

emotional expression and memory formation; these two functions are closely related (emotional impact of an event influences whether or not it will be remembered)

anterior/frontal association area info dealt with?

executive functions (judgement, planning for the future, organizing series of events from memory to prospective action)

how does sensory convergence in multimodal association areas effect multisensory integration

• improves detection/discrimination beyond what either modality could achieve alone

  • faster/more accurate stimulus detection

• noise reduction

• enhance perception of stimuli/experience

what is meant by sensory dominance

different sensory inputs converge in multimodal association areas → question of if there’s a hierarchy of senses, such that information from some senses (ie vision) are more accessible to consciousness than others, and/or dominate perception from other senses?

what is synesthesia

sensory cross-talk

changed perception may be seen in…

non-neurotypical or healthy aging brains

McGurk effect (in relation to sensory dominance)

input from one sensory pathway (vision) influences the perception of the input from another sensory pathway (audition)

when is sensory cross talk elicited

stimulation of one sensory/cognitive pathway leads to automatic/involuntary activation in a second sensory pathway, which elicits a different type of sensation

what is synesthesia v. sensory dominance

sensory dominance is when one sense overrides/suppresses another; synesthesia is a neurological condition where stimulating one sensory pathway leads to involuntary, automatic experiences in a second, unrelated sensory pathway

most common types of synesthesia

most common: grapheme-color synesthesia

also: chromesthesia, spatial sequence synesthesia (SSS)

what is grapheme-color synesthesia

letters and numbers are perceived as inherently colors

what is chromesthesia

sound involuntarily evokes an experience of color

what is spatial sequence synesthesia (SSS)

tendency to see numerical sequences as points in space (ex, number 1 farther away and number 2 closer)

in what ways are types of synesthesia commonly classified?

developmental or acquired

developmental synesthesia

miswiring due to abnormal formation or pruning of connections, mostly in association areas

predisposed to changes in sensory processing and cognitive functions

acquired synesthesia

induced by drugs, neurological disorders, thalamic lesions, sensory deprivation, etc

how does perception change in healthy aging, generally?

sensory dominance generally shifts towards an increased reliance on visual information (visual dominance) to compensate for other sensory losses

pros of changed perception in healthy aging brains

helps to cope with weaker inputs from individual senses, following the principles in multisensory integration

cons of changed perception in healthy aging brains

inaccuracy → much wider ‘temporal window’ in older brains, meaning visual and auditory events may be fused together even if they’re slightly mismatched in time

task-irrelevent visual information → as vision becomes highly dominant, older adults are often more sensitive to visual interference; easier for older adults to ignore a background sound while focusing on a visual task than it is to ignore a background visual distraction while trying to listen to something