lecture 15 +16 - higher auditory processing + introduction to multisensory integration:

How is the auditory cortex organised, and how is frequency represented?

• Primary auditory cortex receives input from the auditory thalamus and is surrounded by non-primary areas (the core and belt in primates).

• Auditory information follows: auditory nerve → cochlear nucleus → inferior colliculus → auditory thalamus → primary auditory cortex → secondary auditory cortex.

• Primary auditory cortex is tonotopically organised, meaning different cortical areas represent different sound frequencies, creating a map of frequency space.

What are spectrograms and what are the main features of complex sounds?

• A spectrogram shows time on the x-axis, frequency on the y-axis, and intensity as colour/brightness.

• A pure tone contains one frequency, whereas complex sounds contain multiple frequencies.

• Many natural sounds are harmonic, meaning their frequencies are integer multiples of a fundamental frequency.

• Harmonic frequencies can be fused by the brain into a single sound source.

How does the brain distinguish sounds with the same pitch, and what is timbre?

• Different instruments can have the same fundamental frequency/pitch but different:

  • Distribution of energy across harmonics

  • Temporal envelope (how the sound changes over time, e.g. onset and decay)

• These properties determine timbre(anything that isn’t loudness/pitch that allows us to say there sounds aren’t similar), allowing us to identify different instruments or voices.

• In speech, the vocal tract filters the sound and creates formants; the first and second formants are particularly important for distinguishing vowels.

• Pitch differences also help separate simultaneous speakers.

What is the missing fundamental phenomenon, and how is pitch represented?

• A sound can have its fundamental frequency removed while retaining its harmonics, yet we still perceive the pitch of the missing fundamental.

• This is the missing fundamental phenomenon.

• Pitch is therefore not simply read directly from the cochlea; it is related to the periodicity/repetition rate of the sound over time.

• There is evidence for pitch-sensitive neurons in auditory cortex, although whether pitch is processed by specialised regions or more widely distributed remains debated.

• A sound can be missing its lowest frequency (fundamental frequency), but we can still hear the same pitch as if that frequency were present.

• This is called the missing fundamental phenomenon.

• This shows that pitch is not determined only by which frequencies are present in the cochlea.

• Instead, the brain also uses the repeating pattern (periodicity) of the sound over time to work out its pitch.

What evidence shows that auditory cortical areas have specialised functions, and why is multisensory integration useful?

• Different auditory pathways support different functions: the posterior auditory field is important for sound localisation, while the anterior auditory field supports temporal/pattern discrimination. This was shown using reversible cooling of cat auditory cortex, producing a double dissociation. (This was shown in cats: temporarily silencing the posterior auditory field impaired sound localisation, whereas silencing the anterior auditory field impaired pattern discrimination, demonstrating a double dissociation.)

• Human lesions similarly show specialisation: Broca's area is important for speech production, whereas Wernicke's area is important for speech comprehension.

• Multisensory integration combines information from different senses to:

  • improve signal-to-noise ratio

  • provide complementary information

  • resolve uncertainty/noisy sensory signals

  • improve perception when one sense cannot provide enough information alone.

What are the main sensory modalities and how do they detect information?

• Vision: light enters the eye → lens focuses it onto the retina → photoreceptors use signalling cascades → action potentials.

• Hearing: sound vibrations move the basilar membrane → bend stereocilia → mechanically open ion channels → action potentials.

• Touch: cutaneous receptors are mainly mechanically gated.

• Smell/olfaction: chemicals activate olfactory receptor neurons, producing patterns of action potentials that represent different smells.

How do vision, hearing, touch and smell represent sensory information?

• Vision: the 3D world is mapped onto a 2D image on the retina, which is also represented in the brain.

• Hearing: the cochlea separates sounds into different frequencies, creating a frequency map that is preserved up to auditory cortex.

• Touch: information is organised somatotopically, with different body parts represented in different brain areas.

• Smell: uses patterns of activity across olfactory receptors and has a qualitatively different organisation from the other senses.

Why do sensory systems have different spatial reference frames?

Each sense represents space differently:

• Vision: retinotopic — based on where something falls on the retina.

• Touch: somatotopic — based on where the body is touched.

• Hearing: sound location is head-centred and must be calculated from auditory localisation cues because there is no spatial map in the cochlea.
  Therefore, the brain has to construct a common spatial representation across the senses. Moving your head can also change the relationship between visual and auditory spatial information.

What is the superior colliculus and what is its main function?

The superior colliculus is a highly evolutionarily conserved structure in the brainstem, above the inferior colliculus. It acts as a hub for rapid sensory-motor integration and helps orient your eyes and head towards unexpected sensory events, such as a loud bang or a bird flying past.
It receives:

• Visual input directly from the retina.

• Auditory input from the inferior colliculus.

• Somatosensory input from the trigeminal nucleus.

• More highly processed visual, auditory and multisensory information from cortex.
  Its major outputs include the frontal eye fields, which help control eye and head movements.

How is information organised within the superior colliculus?

The superior colliculus has layered, overlapping maps:

• Superficial layers: visual neurons.

• Middle layers: auditory and other sensory neurons.

• Deep layers: multisensory neurons and motor output.

Each layer contains a 2D map of space, with corresponding locations represented across the layers.

The auditory spatial map is unusual because the superior colliculus is the main brain structure with an organised map of auditory space.

Experiments with prism glasses showed that changing the visual map causes the auditory map to remap, suggesting that the visual map helps organise auditory spatial information.

What are multisensory neurons, and how do they respond to different sensory inputs?

A multisensory neuron receives information from more than one sensory modality, such as vision and hearing.

• A visual stimulus alone may produce a small response, and an auditory stimulus alone may also produce a small response.

• When both are presented together from the same location, the neuron can produce a much stronger response than expected from simply adding the two individual responses. This is called multisensory enhancement or facilitation.

• If the visual and auditory stimuli come from different locations, one stimulus can instead suppress the response to the other (multisensory depression).

• The strength of integration therefore depends on where and when the sensory signals occur.

• Multisensory enhancement is particularly strong when the individual sensory stimuli are weak — called inverse effectiveness.

• This shows that multisensory neurons do not simply add sensory information; they can enhance or suppress responses depending on the relationship between the sensory signals.

Why do sensory signals have different timing, and how does the brain deal with this?

Different senses process information at different speeds.

Hearing is faster than vision, so sound can reach the brain around 30–50 ms before corresponding visual information.

However, the brain can still treat signals as belonging to the same event if they occur within a temporal window of roughly 250 ms.

This is important because sensory signals from the same event do not necessarily arrive at the brain at exactly the same time.

What are the three main principles of multisensory integration, and why is combining senses useful?

The three principles are:

• Spatial congruence: signals from the same location are more likely to be integrated.

• Temporal congruence: signals occurring close together in time are more likely to be integrated.

• Inverse effectiveness: multisensory enhancement is strongest when the individual sensory signals are weak.

Combining senses can improve the signal-to-noise ratio, provide complementary information, and help resolve uncertain or conflicting sensory information.

What did recordings show about multisensory neurons?

• Multisensory neurons occur throughout the brain, not just the superior colliculus

• Even primary auditory cortex contains visual, auditory and audiovisual neurons

• Some neurons respond to:

  • Sound only

  • Visual stimulus only

  • Both sound and vision

  • Weak/no individual response but a strong audiovisual response

• Integration depends on spatial and temporal alignment of stimuli

What do the main multisensory illusions show?

(sensory illusions allow us to understand multisensory integration)

• Ventriloquist illusion: vision dominates spatial location (if a visual and auditory stimulus happens at the same time,

  but from different places, the visual stimulus location will capture the sound location) e.g. puppet + puppeteer or cinemas

Flash-beep illusion: audition influences visual timing/number (if I give you one flash, presented on its own ,you would correctly report as one flash, but with two very rapidly presented sounds,You'll perceive there to be two flashes when there is actually 1 flash.)

• McGurk illusion: visual mouth movements alter perceived speech sounds/speech perception

• Shows perception involves active integration, not separate sensory processing

What is Bayesian integration?

• Sensory information is weighted according to reliability

• Vision: more spatially precise → dominates spatial judgements

• Audition: better temporal precision → dominates timing judgements

• Similar reliability → information is combined/fused

• Explains why the dominant sense can change depending on the task

How does multisensory processing help speech perception?

• Auditory scene analysis: brain separates mixed sounds into individual sources

• Seeing the target speaker improves speech perception

• Seeing the wrong speaker can impair performance

• Even without lip-reading, matching visual movement can help track the target voice

• Face–voice integration occurs in some higher visual face areas, but not all

• Different brain regions therefore perform different types of multisensory processing

• Superior colliculus was the key model for discovering multisensory principles

• These principles also apply to cortical sensory areas and human behaviour

• Combining weak/subthreshold stimuli can produce particularly strong behavioural enhancement

• Multisensory processing becomes increasingly specialised and complex across brain areas

• Synaesthesia may represent an extension of normal multisensory integration

• Differences in multisensory integration may be relevant to dyslexia and autism

What is synaesthesia and what are its key features?

• A stimulus in one sensory modality automatically triggers a perception/association in another modality.

• Inducer = stimulus that triggers the experience.

• Concurrent = additional perception that is evoked.

• Synaesthetic experiences are automatic, consistent and relatively stimulus-independent.

• Usually present from childhood and experienced as an enhancement, not a replacement, of normal perception.

• Examples: letter/number → colour, sound → colour, smell → taste, time → spatial/colour representations.

What brain differences are associated with synaesthesia?

• Stronger functional connectivity between brain regions involved in the coupled senses.

• Example: word-colour synaesthesia → stronger connectivity between visual word and colour areas.

• Synaesthetes also show stronger connectivity between parietal and lower-level visual cortex, even at rest.

• EEG/visual evoked potentials show differences in early visual processing.

• Anatomically: increased grey matter and stronger white-matter connectivity between relevant sensory areas.

• Differences can extend to subcortical and emotion-processing regions.

What are the two main theories explaining synaesthesia?

• Disinhibited feedback model:

  • Normal brain has extensive feedforward and feedback connections.

  • Reduced inhibition of feedback allows activity from higher areas to reach another sensory area.

  • Predicts a delay between inducer and concurrent perception.

  • Supported by evidence that synaesthesia can be trained or drug-induced.

• Cross-activation/developmental model:

  • Infants have more widespread connections which are normally pruned during development.

  • Synaesthesia may result from retained cross-sensory connections.

  • Supported by structural differences and heritability, but direct cross-sensory pathways are not found in everyone.

What does synaesthesia tell us about sensory processing?

• Synaesthesia may demonstrate that cross-talk between sensory systems is a normal feature of brain processing.

• It is considered a trait rather than a disorder and may be associated with enhanced memory.

• Stronger sensory associations can develop through genetic predisposition + experience/cultural exposure.

• More generally, sensory perception is influenced by connections between modalities, rather than each sense operating independently.

How do the demonstrations show multisensory integration and top-down processing?

• Sound can alter visual perception: synchronised sound and visual events can make ambiguous motion appear as a collision/bounce.

• Auditory timing can make a visual feature “pop out”, reducing the need for serial visual search.

• Sine-wave speech is acoustically very different from normal speech but becomes understandable once you know the words.

• This demonstrates top-down processing: expectations, context and previous experience can transform an initially uninterpretable sensory signal into meaningful information.

• Once the brain has the correct interpretation, perception can become difficult to “unsee” or unhear.