Module 2 - ch and textbook notes

Speech perception diagram

Acoustic signal →

nuero auditory represent tion (nuerogram) →

Nuero phonological representation   (phonological sketch) - figure out what you are hearing   Sound sometimes overlap - bait vs bet

Lexical item  understand words, morphemes

Cocktail party problem

how can you understand people - one word or sound when you are surrounded by noise 

Signal differences

Sound aplitude (dB) - Speech enverlope - captures, vowels, melodies, a periodic noise

Frequency - time(s) 

• Can find S concentrated in high frequencies - at the top 

• Low frequencies - at the bottom 

• Vocal cords hitting each other osscillatory periodic noise 

Light and dark - maps amplitude - amplitude frequency time

Signal example/ frequency spectrum

Signal - hearing like vision

• Every photon gives information 
  
  

• Low frequency words on the bottom 

• S on the top - higher frequency and fricative 

• T - gap between the words/shading - voice onset time 

  • T sound - Silence - total closure 

  • 4 times tongue touches top of mouth 

Cochlea

Hammer - connected to lever → hits another membrane → connected to vacuum sealed lever in membrane 

Cochlea configuration  

• Sound/frequencies travels through cochlea  

• Base of cochlea - narrow - tuned for high frequencies 

• Apex of cochela - wider - lower pitch frequencies

Frequencies and place in cochlea

High frequency: 8000 Hz

In the front/base: 5mm 

• High frequencies resonate with top hairs and die off 

 

Low frequency: 250 Hz

In the bottom/back: 25mm 

• Low frequency activate at bottom/back - end

Tonotopy - linear gradient we get for free bc auditory cortex

Parts in primary auditory cortex respond to base or apex of cochlea
Get sounds from auditory cortex & lay out in line 

• High pitch → low pitch 

Linear mapping 

• Direct 

• High pitch to low pitch 

• Connected in a linear way 

Gradient 

• connected to nerve fibers in cochlea 

• blanketed into auditory cortex 

Top down processing Ex: cats are soft and furry

once you have expectation about what you are going to hear  → actually affects what you hear 

Ex: vocoded voice 

• Once you hear something you can’t unhear it

phonemes are feature bundles ex

phoneme /k/

• [+consonantal] (it is a consonant)

• [-sonorant] (it is an obstruent, not a vowel, glide, or nasal)

• [-voiced] (vocal cords do not vibrate)

• [+velar] or [+high, +back] (articulated at the soft palate/velum)

• [+stop] (airflow is completely blocked momentarily)

Cats are soft and fury ex - tracking

Study tracking 2 things/streams 

• Speech envelope 

• Phonemes 

Speech envelope 

• Know that its your turn to say something 

Rhythm - low level auditory tracking of speech envelope - can tell you where you are in a conversation 

Speech envelope

• Outline of speech 

• Rythym, prosody 

  • Dont have high level information 

Cochlear implants

More channels = more information and better hearing  More like the performance of a real cochlea  Ex: 20 channels - speech becomes able to understand  Music -  you still might not be able to know what kind of music it is

Nuerogram 

• Tonotopy - responding to pitch 

  • Shape 

• Periodotopy - linear spatial arrangement of speech envelope 

  • Right hemisphere tracking speech - you know where you are at a discourse level - of speech

Phonotopy

not currently agreed on whether there is a phonological mapping of features in the brain - phonotopy 

Barton et al - textbook part

How is temporal info from speech envelope encoded in auditory cortex? Barton, Brener, et al.(2012)

• An experiment mapping how different parts of auditory cortex responded to diff frequencies

  Experiment: When mapping how different parts of auditory cortex respond to sounds at different frequencies → tonopic patterns - shared

  
  
  

  Periodic stimuli → diff parts of auditory cortex responds to diff kinds of periodic stimuli → in a gradient

    ↑ new spatial code called periodotopy 

  
  
  
  

  (Neurons responding to 2Hz stimuli adjacent to those responding to 4Hz next to 8Hz) 

  • Arrangement of this response - nueeonds responding to periodotopy 

  • Changes 90° offset from those responses to frequency changes

phonemic receptive fields - mesgarani et al (2014)

While patients listened to the continuous natural speech - spoken sentences, researchers recorded real-time, high-resolution neural activity from the electrodes covering the STG—a major high-order auditory processing hub in the brain. [1]

Instead of looking for electrodes that just responded to specific words, they analyzed how individual electrodes responded to the microscopic building blocks of those words: phonemes (distinct units of sound) and their acoustic properties

• areas of neurons in the auditory cortex that are tuned to select, process, and differentiate distinct speech sounds (phonemes).

Different groups of phonemes (dipthongs + vowels, nasal, plosives) have different activations of neurons - phonemic receptive fields 

• Was found that electrodes at different sites respond to different phonemes 

  • Nemo Mesogarani- shows that populations of neurons in STG have phonemic receptive fields 

    • 1 electrode may be sensitive to class of stop consonants, not specific phones
      
      

Analysis by synthesis ex - phonemic receptive fields 

• Takes info in a neurological way and maps it to cochlea 

Generally goes forward 

•  take information about sensory event and you build upwards - what do you think is true about phonology of word - can build upwards in this 
  
  

Top down thingy 

• Info filters backwards and can influence what you hear/know 
  
  

Breaking something down to understand it 

• Break it down by building it up 

• Figure out what words are by figuring out what parts are made of - figure out what parts are made of by figuring out what words are 

multi voxel pattern analysis

How to get from continuous acoustic spectra to categorical representation

• phone to phoneme

Continuous acoustic spectra 

• populations of features that respond to things like classes 

Chang study

Natural speech - in form of like audio books - see if you can correlate patches to different phonemens 

• See if they respond selectively to different phonemess

• Neural populations that respond to individual phonemes 

• chang study 

Kinds of discrete representations we see in brain and what to look for 

Categorical representation 

Activate different features - abstract features of speech sounds  

• Manner 

• Place - in IPA - based on where your articulators - lips - come into contact 

• Voice - is it voiced or not 

• Sound 

Dedicated areas that are respondent to and tracking speech sounds that are not really reducable to just physics 

Ganong effect

Voice onset time experiment

Take a word thats an actual word and then a word thats not Tuft (T - burst of air) vs duft  What feature do they differ in  Tongue placement and amount of air  Voice onset time  d sound - Vocal chords start vibrating immediately Close to 0 - start voicing right away  t sound - air released then start vibrating Gap or puff of air  Timing thing is continuous - can start the voicing whenever you want after a T  5ms intervals between t and d  Ask people in study if they heard a t or a d  Result  People biased if it sounds like a real world vs if it doesn’t  Tuft - duft  Tash - dash  Even if speech sound in beginning not d but t - people will hear d if hearing the word dash  Start with something analog and continuous and map it to different features

Speech sampling rate ex

Speech sampling rate ex: 

Audio file,  chopped into segments, with each segment reversed.

• Listen to file and try to decipher what it says 

200 ms 

• 4 per sample 

• wrong order 

100 ms 

• 2 samples per chunk

• wrong order 

50 ms 

• Sampling once every chunk 

• Doesnt matter that much - its reversed 

20 ms 

• So small that its below speech sampling rate 

0 ms 

The rate at which we sample speech is pretty low 

• Meaning fewer data points transmitted per second 

Temporal windows of integration

• How fast phonemes come in 

• Speech envelope thing happening 

Short window  25-40 ms Phonemes Features

Long window  200-300 ms Syllables Speech envelope Audiovisual integration

2 windows of integraton + hemispheres

2 windows -  dissociate in hemispheres - different patterns of resolution 

• Right hemisphere - long window 

• Left hemisphere  - short window 

2 levels of categorical perception

• top down

1. Acoustic auditory encoding - perception

• perception - continuous clues - what you hear 

2. Acoustic space carved into categories - cognition

• cognition - categories 

The big claim: The auditory percept is changed or warped by the category labels. Whatever you hear changes based on the category you put on it

Mismatch negativity

phonological features

• A way to organize speech sounds: 

  • Phonological features 

ex: t

• + coronal

• - voiced

• +obs

underspecification

Featurally Underspecified Lexicon (FUL) uses three values.

Explains why sometimes certain sounds change in their environment, but the change is not symmetric

• Sound

• Manner 

• Voice 

In certain environment things get underspecified - t to d

Walk Vs  Walked   [wak←t]  T can change to d

Greet Vs greeted     [griea←d]  D can’t change to t

In certain environment things get underspecified - t to z - plurals

Plurals - only turn into s in certain environment (is it not this environment?)  Cat → catz

dog → dogz

The environment the phoneme/word was found in changes it 

Eulitz & Lahiri (2004)

standard stimuli create a central sound representation that is more abstract than the sum

• Linking theory 

  • Fill in the feature values as you hear something over & over again 

• 1st time you hear something - represented the way it is 

• Hear over & over - brain gets lazy do you get representation thats underspecified 

Underspecfied sounds in language   - Eulitz & Lahiri (2004)

1st time you hear it - deviant 

Deviant - features in acoustic signal

5th time you hear it - standard 

Standard - features in the memory rep. 

• Only one (one more time) should lead to mismatch 

• Clash in negativity 

Types of sounds 

• Coronal  - tip of tongue 

• Labial - lips rounded

• Dorsal - back of tongue 

O -standard 

O crossed out - deviant 

• Matters because of linking theory 

Underspecfied sounds in language   - Eulitz & Lahiri (2004) MMN

Non conflict cases - MMN amp/latency does not vary.

• mismatch negativity

conflict cases - MMN significantly earlier and larger when there is

conflict [o vs ø] compared to non conflict [ø vs o].

Hestvik & Durvasula (2016)

Did experiment to compare t vs ds

• Take all information & lay out lineralty → represent linearly in brain responses
  
  

• Phonotopy - is there an abstract phonological layer 

• Yes - underspecified features are more specivied than phonemes 

Standard vs deviant compare t and ds - Hestvik & Durvasula (2016)

Compare d’s- big difference

Compare t’s - nothing

Sound is gradient but perception is categorical.

Auditory perception + linguistic theory 

Using EEG we can see 

• Gradient acoustic encoding.

• Categorical symbolic representations.

• Decomposing features of symbolic representations.
  
  

Dual stream model 

2 pathways sound goes down

2 routes we take when doing the phonological processing of words 

Dorsal pathway  Sound to articulation Where things are located in space  Hear something - maps it to part of brain that does motor gestures  Affects people’s perception because you map your system to motor system when you hear it  We use motor system when linking

Ventral pathway  Sound to meaning Want to know what sound is   Map to meaning so you know what something is when you hear it

Temporal windows of integration (again) 

Short window  25-40 ms Phonemes Features

Long window  200-300 ms Syllables Speech envelope Audiovisual integration

IFG - Broca’s area is part of the dorsal steam that maps speech sounds to articulatory representations. 

How does this fit withe what we know about Broca’s aphasia? 

Fits with what we know about Broca’s area because broca’s area is used for production, said to be where language is localized for comprehension so it would make sense that it has to do with articulatory representations and actually articulating parts of language

Speech waveform/envelope

shows slow changes in loudness following speach intensity envelope 

• Shows syllabic information 

  • Grey line 

Spectral information -

fine structure detail that shows phoneme features.

• rapid change from [f] to [t]

• Shows more air coming our vs not 

How is sound represented in auditory cortex? 

Look at:

1. tonotopy - cortical spatial code for sound

• different parts of auditory cortex respond selectively to different frequencies 

• In spatial code created in cochela perseved in auditory cortex 

• Could also say neurons in primary auditory cortex have different frequencies

2. receptive field of neuron - connection between a certain stimulus and the response of a given neuron

Recognizing speech

(frequency information not always important for understanding) 

1. Speech envelope - outline of the peaks (loud moments) and troughs (soft moments) in waveform - gives speech patterns of syllables & stress 

 → neurons with similar tuning are adjacent to each other 

2. Fine structure - rapid sound vibrations making up speech energy at different frequencies

- has narrow bands of energy-distinguish things like vowels

- broad band- sibilants, fricatives

 →  speech comprehension good even when little fine structure & only speech envelope provided

→ testing fine structure

noise vocoding- speech envelope extracted from an utterance, fine stricture replaced with white noise - speech stull understandable 

Brain needs to represent 

• spectral signature of speech - frequency info

• Changes in loudness over time - temporal info

  • tracks it speech envelope in much detail

entrainment

match between brain activity & perceptual input.

⬇

Locks onto things you focus attention on 

- speech we comprehend only a fraction of what we hear

link found to fluctuation of brain activity & temporal lobe

Cocktail party effect

2 ways to navigate chaos of real life acoustic inputs 

1 - hear all of it at early stages (including overlapping acoustic input from multiple sources)  → rely on higher level inguistic processing to sort out words to attend to

2 - Early perceptual processing being more selective → we only hear what acoustic input we consider important

Data more consistent with #2

Toscano et al (2010)

Toscano - would think things on one side of threshold would have a cutoff and one thing would have something else 

• Tells us 

• VOT - looks like a kind of tonotopy 

• Linear distribution of a certain kidn of feature 

• EEG - amplitude of the signal 

• Getting a linear spread of the feature 

• Feature not present in tonotopy - all of the pitch information coming out of the cochlea - continuous  

• Voice onset time - temporal measure between tow events 

• The amount fo time that happens between he burst and the voicign 

• Tonotpy does not get this - to get info you have extract it over time 

• Already this is more abstract than tonotpy - but not phonotoy

Kuhl & Miller (1978)

Categorical perception is task dependent 

meaning that

a person showing categorical perception—perceiving continuous physical changes as distinct categories rather than a smooth gradient—depends on how you test or ask them to process the stimuli

tested categorical perception in chinchillas and found that it was just like humans

If you tell people to sort out sounds into categories they can do it 

• Chinchillas can also do it 

Eulitz & Lahiri (2004) test

used an auditory "oddball" paradigm + Event-Related Potential (ERP) recordings to measure - Mismatch Negativity (MMN)—to understand how the brain processes standard versus deviant phonological features

1. The Stimuli: Participants were presented with pairs of spoken vowels (such as [e] and [o]). [1]

2. The Reversal Technique: The researchers flipped which vowel served as the standard (the repetitive background sound) and which served as the deviant (the rare, unexpected sound) across different blocks. [1]

3. Brain Tracking: While participants passively listened to the sounds, their automatic, pre-attentive brain activity was monitored using EEG/MEG to track the MMN response—an electrical brain wave triggered when the brain automatically notices a change in an auditory patter

Eulitz & Lahiri (2004) predictions + results

• test to see designed to see if the brain relies on fully detailed representations or underspecified representations (where some default linguistic features are omitted from long-term memory)

• The Prediction: If the mental lexicon tracks every feature symmetrically, reversing the standard and deviant would yield the exact same MMN response. However, if the lexicon is underspecified, the MMN would be asymmetric. [1, 2]

• The Result: They discovered a clear asymmetry. When the standard vowel lacked a specified feature (like a specific "place of articulation") in the listener's mental lexicon, the incoming deviant sound did not trigger a strong mismatch. When the roles were reversed and the standard was fully specified, the deviant triggered a significantly larger and faster MMN. [1, 2, 3]

Mismatch Negativity (MMN)

—to understand how the brain processes standard versus deviant phonological features

Hestvik & Durvasula (2016) tested

underspecification theory,

• numarked phonemes are stored sparsely in long-term memory without certain features (e.g., /d/ is underspecified for voicing),

• marked phonemes are fully specified (e.g., /t/ is specified as [spread glottis] or voiceless)

Mismatch Negativity (MMN) auditory event-related potential (ERP) to test if presenting an underspecified phoneme as a rare "deviant" among specified "standards" causes a different brain reaction than the inverse

Phonemic vs. Phonetic Processing: They introduced a new experimental modification to see if changing the paradigm to encourage short-term phonetic acoustic memory traces (rather than abstract phonemic lexical memory traces) would alter this effect.

Hestvik & Durvasula (2016) results

• Underspecification Asymmetry: They successfully demonstrated an underspecification-driven asymmetry in the brain's automatic response to contrasts in English. The brain reacted differently depending on the direction of the phoneme change, confirming that some speech sounds are represented more sparsely in long-term memory than others.

• Context-Dependent Disappearance: Crucially, they found that the asymmetry disappears when the MMN paradigm is modified to force the brain to form acoustic/phonetic memory traces instead of relying on underlying phonemic traces. [1, 2]

• Neurobiological Proof: This provided direct neurobiological evidence that long-term phonological representations in the mental lexicon are highly abstract and sparsely coded compared to the fully specified phonetic processing of immediate acoustic inputs

Ding & Simon (2013)

• participants tisten to speech from audiobook mixed with noise 

  • Moderate levels - participants could understand the speech even though strongly distorted 

Temporal processing in auditory cortex

• found that we can turn up volung selectively tune in out of diff speech sounds from other acoustic inputs

Sound is characterized by 2 patterns

1. spectral pattens- sound energy at different frequencies 

2. temporal patterns - loudness varying over time

2 pieces of evidence that speech perception can’t be understood using just continuous info 

• mainly be speech signals vary greatly & variation handled with ease

Evidence 1 - categorical perception

Categorical perception - phenomenon stating that if you have speech sounds that have been artificially altered & start with 1 syllable, then 

change incrementally to another, listeners don't perciere a smooth change

Ganong effect - If hearing ambiguous sound between [s] vs [ʃ] (sh sound) they'll percive whichever one makes more sense in a word

- more likely to hear [s] for a male voice : “said”

-more likely to hear [ʃ] for a female voice : “shed”

Ganong effect different from McGurk 

• Shows that continuous acoustic info not enough to explain brain recognition of phonemes & words 

• Was found that electrodes at different sites respond to different phonemes 

  • Nemo Mesogarani- shows that populations of neurons in STG have phonemic receptive fields 

    • 1 electrode may be sensitive to class of stop consonants, not specific phones

2 pieces of evidence that speech perception can’t be understood using just continuous info 

• mainly be speech signals vary greatly & variation handled with ease

Evidence 2 - all or nothing

Evidence 2 : Best evidence that brain responses categorical

• electrodes that show classic non- linear response associated with categorical perception

• Specitic electrode responds in an "all or nothing" way when a consonant is voiced

• one responds in "all or nothing" when consonant unvoiced

• one responds to stops regardless if voiced or not

• Even carry responses show categorical, not continuous features 

• Categories can’t fully be understood as summaries of acoustic differences 

How does the brain transform a continuous neurogram to a categorical representation of speech sounds?

Input:

•  neurogram 

Transformation:

- occurs rapidly (100-200 ms after speech starts)

- in superior temporal lobe, in vicinity ot primary auditory cortex

Output: need

- Phoremic features

Speech reversal thing

Temporal window of integration

• Brain has sampling rate for speech sounds - takes a "snapshot" of acoustic info at fixed windows, everything during snapshot is in 1 chunk of info → info used to pick out phonemes

           ⬇

Speech reversal - 

• participants couldn’t understand reversed speech spanning large segments (grains) - 200 - 300 ms 

• Could hear some - less than 200 ms 

• 100% accuracy - 30-50 ms 

McGurk effect

 auditory & visual information combined and don’t match 

• alters what participants hear

Top down spech processing 

• We make predictions about what person will say next 

  • internally synthesize upcoming speech → helps in analysis of speech signal when it comes 

• The phonological sketch is refined by

• a series of feedback loops between acoustic input and linguistic knowledge called analysis by synthesis. 

• The outcome of these interlocking processes is a representation suitable for identifying words.

Point of speech perception

• point depends on listener's task (what they were asked to do) 
  
  
  

1 type of task - finding meaning

• In speech perception - identify words, phrases, sentences to understand speaker's message

•  repeat/ shadow something you're just heard

2 - phoneme Monitoring task

- lab setting, push button if you hear certain thing

Dual stream model

Ventral stream - understanding 

Place: temporal & frontal lohe Role: transfom from acoustics → phonemes → lexial items → meanins of sentences

Dorsal stream - processing 

Place: temporal → superion regions of frontal lobe Role: Map acoustic input to representations of articulation

Turkish vowel study

Researchers tested if there was a “nuetral vowel space” 

• Particiopants asked to press button when hearing a cerain consonant among sounds 

Evoked brain response - M100 response - auditiory cortex peaks about 100 ms after vowel begins 

• Researchers reasoned - if there was a spatial code for phonemes - like there are for neurograms - the location of M100 peak would differ systematically for diff Turkish vowels 

Results - 

source locations fall along systematic axes: 

• Front vowels - inferior - superior plane 

• Back vowels - anterior to posterior plane 

Results - there is a vowel map in the STG 

• Distinctive features map to separate axes in space

Phoneme confusability experiment

• Adults listened to a series of syllables that begun with different consonstants 

  • Measured similarity in terms of phoneme confusability 

  • Multi VOXEl pattern analysis 

    • Used to understand the nural representations that matter to a particular set of voxels in the brain 

Results 

• Stimuli lead to distinct patters of activation 

  • Voxels discriminate between these kind of stimuli 

  • Nueral population in these voxels have receptive fields for whatever feature(s) separate the 2 stimulus types 

Superior temporal cortex represents phonological detail in a spatial code (phonemic receptive fields) 

• Code is not organized in a phonotopic gradient with similar phonoemes activating adjacent nueronal populations

Mismatch response (MMR)

• Used to understand whether different stiumuli match or not 

  • If 2 ppl with different sounding voices make the same speech sound - does it activate the same or differnt mental representations in a listener?

Oddball design 

In odd ball experiment there are

• Standards - Certain stimuli are presented frequently - 80% of time 

• Deviants - some stimuli presented rarely - 20% 

MMR - a signal that emerges around 200 ms after hearing a deviant item - relative to hearing the same stimulus when it is prenesnted as a standard 

• Sensitive to phonemic difference, not just acoustic differences

• Expected since brain has rapid sensitivity - less than 200 ms for phonemic features 
  
  

Underspecified

- If phonemes include defaults then you don’t need to store every featural detail separately 

• Mental representation only needs to track when phoneme differs from the default 

Feauture that is underspecified 

• CORONAL feature  - captures the articulation of consonants with the tongue tip 

• Ex [t], [d], [s]

Contrasts with DORSAL feature - [k] 

[ta] is standard 

• There is no explicite expectation for CORONAL (its the default) 

• Then [ka] deviant will not show as strong a mismatch (rememebr that the acoustics are different so there will be at least some mismatch) 

IMportant point about MMR 

• Feature understpefication predicts that MMR is asymmetric

  • [ta] will bne a large mismatc against standard [ka], but [ka] will be a smaller mismatch against a standard [ta] 

Pullvermuller et all 

• Participants engaged in speech tasks while fMRI scans 

1. First task - produce [ta] and [pa] syllables 

• Observed separate patterns of activation along motor cortex when participants produce a bilabial [pa] syllable (closure of lips) compared to alveolar (tongue tip) [ta] 

2. Participants listened to those 2 types of sounds 
   
   

Results - 

• activation in auditory cortex and motor cortex 

  • Spatially separated according to articulators 

  • Tongue region activated postivielt for [ta] but not [pa] 

  • Lip redion activated for [pa] but not [ta]

Evidence that motor cortex activation → successful speech perception 

Mottonen Watkins (2009) 

Thought that 

• If motor cortex is actively involved in speech perception - discrimination between phonemens that differ in one feature will be harder if activation is inhibited 

• Inhibt activation of motor cortex using rTMS technique 

Results 

• When rTMS inhibition targeted lip region patients found it harder to distinguish bilabial [pa] or [ba] segments from alveolar [ta] or [da] segments

• Specific to lip motor cortex

• rTMS did not affect ability to discriminate between phonemic pairs not involving lip articlators 

Brain regions involved in controlling articulation may also play active role when simply listening to speech 

Evidence that 

• Speech perception may use motor system but doesn’t depend on it 

Classical characterization of brocas area 

Experiment - Rogalsky & Hickok (2011)

• 5 case studies 

• Successful speech production doesnt depend on speech comprehension 

• Al patients sufferend from left hemisphere lesions that impacted speech fluency 

• All performed wll on speech comprehension tasks 

Critique - brain may compensate for url trauma 

Wada Procedure 

• Anethetia injected inoto major artery for one brain hemisphere 

• - puts one side of prain to sleep 

• Patient not able to produce speech - affected hemisphere on that side is dominant for language 

• Patient usually can comprehend speech just fine though 

Data indicate - motor systems involved in speech production not necessary for successful comprehension 

• Motor Systems engaged during speech perception but not necessary for success 

summary kinda

Comprehension relies on different neural systems that those for non speech sound perception - pure word deafness  Systems may include motor/articualatory syrtems - for speech production  Inhibit motor cortex with rTMS  Spcial speech production systems not necessary for comprehension  Wada procediere  Auditory processing tuning to speech sounds is instatintation of general sensor systems that operate over fixed windows of time