cognitive psychology - perception + multisensory

what is transduction

• the process of which senses get their message to the brain

• transforms information from the sense organ into electrical impulses that the brain can understand

what is multisensory integration

• in order for our brain to perceive the external environment it uses multiple sources of sensory information

• research looking into how the brain coordinates sensory information provided by different sensory streams - Calvert & Thesen 2004

• different sources of information must be efficiently merged to form varied multi sensory experiences

evidence of multisensory integration - ventriloquism

• the ventriloquism effect suggests that the perceived location of an auditory signal can be captured by a nearby visual signal

  • Howard & Templeton 1966

evidence of multisensory integration - mcgurk

• the mcgurk effect suggests semantically conflicting visual and auditory speech cues are bound together to form a novel perception

  • McGurk & MacDonald 1976

evidence of multisensory integration - rubber hand

• the rubber hand illusions - seeing a rubber hand, illusory shifts where you feel your hand to be

  • Hay et al 1965, Pick et al 1969

evidence of multisensory integration - welch & warren

• the nervous system binds multisensory cues presented in close spatial and temporal proximity to form a coherent perceptual gestalt

  • Welch & Warren 1986, Stein & Meredith 1993

what is the spatial rule

• multisensory integration is more likely or stronger when the unisensory stimuli arise from approximately the same location

what did stein et al 1996 find

• that an auditory stimulus enhances perceived visual intensities, but when both visual and auditory events were presented at peripheral locations, spatial proximity was essential for multisensory integration to occur

• enhancements strongest at the lowest visual stimulus intensities

• no spacial proximity was needed for enhancements to occur at the centre of fixation

  • results indicate that spacial constrains for multisensory integration only hold for peripheral visual events and not for central visual event - Lippert et al 2007, Noesselt et al 2008

what is the temporal rule

• multisensory integration is more likely or stronger when the unisensory stimuli arise at approximately the same time

• a strong multisensory integration effect happens when the time between the onset of auditory and visual events is less than 100ms

what did meredith et al 1987 find about the temporal rule

• measured cells of the superior colliculus in the cats brain, found a decline in integration when the time windows became larger than 100ms

• further increase in temporal disparity between an auditory and visual event could even cause cells to become inhibited

• this narrow 100ms time window is a distinct feature of multisensory integration

what is the unity assumption

• the unity assumption is an observers assumption, or belief, that two or more unisensory cues belong together

  • Welch & Warren 1980 + 1986, Spence 2007, Chen & Vroomen 2013

• one of the key mechanisms by which the human brain solves the crossmodal binding problem

what did vatakis find about unity assumption

• first to report the unity effect

• audiovisual speech stimuli - humans voices and moving lips - were presented in one of two conditions

  • gender matches

  • mismatched

• when the voice and face were gender congruent more mulitsensory binding took place, leading to a unity effect

what is semantic congruence

• multisensory stimuli in natural environments share contextual or semantic information features, not only spatially and temporally congruous but also semantically congruent

  • Laurienti et al 2004

semantic congruence - cross modal

• response time was significantly faster on cross modal trials than either the unimodal visual or auditory conditions

  • enhanced responses elicited in the presence of congruent cross modal stimuli

• responses to congruent cross modal stimuli were faster than responses to either of the unimodal stimuli

• information carried by cross modal stimuli plays an important role in determining the behaviours elicited by these stimuli

• people exhibit consistent crossmodal correspondence between many stimulus features in different sensory modalities - Fairhurst et al 2015

semantic congruence - dual visual

• no significant difference between the dual visual and the unimodal circle or unimodal word conditions