Lecture 11: Hearing, Multisensory Interactions

Hearing Summary

• Pinna & auditory canal collect & amplify sound

• Eardrum vibrates in response to sound

• The ossicles transmit & amplify these vibrations

• This produces movement on the oval window & vibrates the basilar membrane inside the cochlea

• The organ of corti sits on top of the basilar membrane & contains hair cells that transduce the vibrations to neural signals

• Auditory nerve carries neural signals

The Cochlea As an Acoustic Prism

• Basilar Membrane inside the cochlea

• Different parts of the cochlea respond to different frequencies

• A prism refracts light, a prism also refracts light by different amounts for different wavelengths

• Different wavelengths get refracted by different amounts as they enter a prism.

• Splits light into different frequencies

• Cochlea does the same with sound frequencies

• Different parts of the basiliar membrane vibrate to different frequencies

• Lower frequency sounds displace the membrane more towards the apex & higher frequency sounds displace the membrane more towards the base

The Cochlea As an Acoustic Prism

• Apex of cochlea, low frequency (right)

• Base of cochlea: high frequency (left)

Basilar Membrane breaks down frequency components naturally (Fourier analysis)

What is Auditory Localization? How do we localize sounds in the real world?

Auditory Localization

Our displaced ears provide 2 location cues:

• Interaural time differences (ITDs)

• Interaural level (intensity) differences (ILDs)

Auditory Localization- Interaural Time Differences

• Sound travels at 340 m/s (in air)

• Physics dictates that most sounds will arrive at each ear at slightly different times (microseconds)

Interaural Time Differences (ITDs)

• Binaural neurons sensitive to ITDs are located in the brainstem

• First place where auditory information from both ears is combined

• Interaural time differences are smallest for locations directly in front or behind

• Largest for locations that are directly to your left & to your right

What are Interaural Level Differences (ILDs)?

• Differences in the intensity of the sound hitting each ear

• The sound that hits the closer ear is always more intense (ie. louder)

Which locations have the same ITDs & ILDs?

• ITD and ILD cues are not perfect

Cones of Confusion

• Not often a concern because our heads are rarely stationary

Summary

Physiology of Hearing

• Organ of Corti sits on basilar membrane; different frequencies are coded by different locations on the membrane

Auditory localization

• Combination of multiple cues (ITDs and ILDs)

What is auditory scene analysis?

• Breaking down the summed auditory signals into its component parts. Can distinguish one thing from many

• Relies upon cues analogous to visual depth cues and Gestalt grouping rules.

• Heuristics (shortcuts) don’t just apply to vision

Heuristics for auditory scenes- What is Motion Parallax?

Motion parallax: determine distance by moving - further sounds don’t change as much as close ones

• Closer objects change more

Heuristics for auditory scenes- What is Atmospheric Interference?

Atmospheric interference: muddier sounds are further away. Higher frequencies get blocked —> changes in frequency composition.

Gestalt Grouping Cues for Auditory Stimuli

• Grouping by similarity in pitch

• Group higher frequencies together & lower frequencies together

Auditory scene analysis- Similarity

• Timbre allows us to follow a single instrument in an ensemble, even when it crosses frequency with another instrument

Auditory scene analysis- Good Continuation

• Auditory completion: restoring the sounds when it is temporarily interrupted (occluded). Easier to hear/complete sentence when gaps filled with noise than with silence

Example: Beatboxing

Importance of context in vision

Auditory Object Recognition

• Context matters

• Context also matters for audition Interpreting sounds, speech, phrases depends on surrounding information

• Where at the silences between words?

• Gaps between words don’t give away to segmentation

  • E.g. Foreign languages

• Context helps interpret sentence meaning. Ambiguous cases arise when lacking context Sentences: must assign or infer object, subject, verb In vision, must assign or infer figure, ground, motion, etc.

Translations and ambiguous words or phrases

• In a Bangkok dry cleaner's store

• "Drop your trousers here for best results.”

• In a Cocktail lounge, Norway

• “Ladies are Requested Not to have Children in the Bar” In an Acapulco hotel

• “The manager has personally passed all the water served here.

Context

• Visual context can influence interpretation of auditory information

• McGurk effect

Sound /ba/

Lips /da/

Hear /da/

What is the McGurk Effect?

• Sometimes vision dominates, sometimes combination of vision + audition

• McGurk Effect: Often combine auditory and visual information into new interpretation /gag/ + /bab/ = /dad/

• Visual cues can support auditory perception

Summary: Auditory Scenes & Objects

• Many of the concepts we learned in vision apply in analogous fashion to hearing

• Distance cues, grouping principles

• We use context to segment words and to interpret meaning

• Rely on combinations of multiple cues (e.g., hearing + vision)

Resolution limits for different senses (2:16:08)

• Position - vision best, audition & touch worse

  • Where is it located?

• Time - audition best, vision second, touch worst

  • When did it happen?

• Intensity - vision, audition, touch all relatively good How loud, bright, painful was it?

Implications of resolution limits:

• If you lose vision, you lose spatial resolution

  • Rely on vision most for locating objects

• Rely on audition but only if there’s noise around or you have echolocation (bars, dolphins)

• Rely on touch (e.g, Braille) but touch has poor spatial resolution

Why does Braille use dots rather than letters?

What is Cue Integration?

• Brain processes information from many senses simultaneously to interpret the world around us

• Much of the information is not modality (sense) specific

• E.g., Person on TV has a visual image and an audible sound. How does the brain bind or integrate visual and auditory cues? What if our brains couldn’t bind the information?

• Cues would conflict.

Investigating cue integration- What happens when cues conflict?

• Depends on reliability of cue & sensitivity of the system

What is Ventriloquist Illusion?

• When watching TV & Movies, the sound doesn’t come from the person or character talking. But it seems to… Why?

• Brain weighs the more sensitive system and more reliable cues more heavily when interpreting scenes. Vision is reliable for detecting positions. So visually perceived position of the person or character captures sound.

• Vision dominates.

• Why would the brain want vision to capture audition? (Is the ventriloquist illusion beneficial?)

• Image and voice processed independently. Brain needs to bind/integrate Spongebob’s face with his voice. This would seem to require precise timing; is this possible?

• Solution: brain picks one cue (the most likely--vision in this case) over the other

Auditory Capture of Vision

• Audition more sensitive than vision to temporal structure (timing differences)

• When faced with ambiguous timing of stimuli, the brain takes auditory information more seriously.

• Audition trumps vision.

What is Synesthesia?

• Synesthesia: a different kind of sensory crosstalk

• Sensory input in one modality produces automatic experiences in a different modality

How do you test for Synesthesia?

• Cause of synesthesia? Many theories, none conclusive

Plasticity: The Brain’s Ability to Change & Adapt

• Audition & blindness

• Early blindness can produce remapping of auditory information in occipital lobe.

• Cortex normally reserved for vision is recruited by auditory system to improve spatial resolution (vision is normally most spatially precise modality).

Audition & Blindness

• Echolocation in humans

• Vision still better spatial resolution

• Partial compensation by audition

Touch & Blindness

• Recruitment of visual cortex for other uses

• Touch uses visual cortex in the blind, more so in early blind than late.

• In sighted individuals, activity of the visual cortex goes down in a tactile task

• Visual system can also be recruited by touch (somatosensation) in blind individuals.

Summary

Cue integration:

• Sensory modalities have different resolution limits (spatial precision, temporal precision)

• Integration of cues: integration is based on cue reliability (which sense gives me the most precise information?)

• Synesthesia: Another form of sensory crosstalk

• Plasticity, remapping:

  • The brain can reorganize itself to cope with changes in sensory input (V1 for touch, hearing)

  • The brain is most flexible early in life, but can still cope with changes in sensory input later on in life

Vision

General Principles

• Adaptation or Autocalibration

  • Our brain quickly adapts to the changing state of the world

• Heuristics/Rules of Thumb

  • The brain uses ‘shortcuts’ to solve computational complexities

  • When these shortcuts fail, its called an illusion (and its rare)

• Bet on the familiar or non-accidental

  • Go with most likely interpretation