Models of language learning

ways we can observe how we learn language

experimental investigation and computaitonal simulation

experimental investigation

We control the stimuli perceived by the participants, then measure the response in behavior to see if there is a systematic difference/effect 

Different from attempting to model the cognitive process and testing it by simulating the stimuli and see what behaviors it generates  

computational simulation + methodological advantage

Computational models required detailed specification of the input properties and the processing mechanism 

Methodological advantage:

• explicitly assumptions - all bias or constraint on the characteristics of the input data and learning mechanism are specified

• controlled input - researcher has full control over the input that the model receives in its life time

• observable behavior - impact of every factor in the input or the learning process can be directly studied in the output

• testable predictions - novel situations or combinations of data

why do we need a way of evaluating a model we build?

as we build a model based on a theory, we make assumptions about how the brain works and formally build a model using said assumptions

what is the evaluation scheme of Cognitive models of language

• What humans know about language can only be estimated/evaluated through how they use it 

  • (comparing)Language processing and understanding 

  • (comparing) Language production 

e.g. models of reading we can measure how fast people read and track eye movements  

what does language comprehension in children reveal

• Comprehension experiments reveal biases and preferences 

  • Knowledge sources that children exploit 

  • Their biases towards linguistic cues 

  • e.g. 'Tim and Kim are blicking' vs 'Tim is blicking Kim' 

Language production in children

Analysis of child production data yields valuable clues on developmental patterns: utterance length, sentence complexity; error detection overgeneralization, subject dropping 

Resources of naturalistic child-adult interaction data 

• Transcriptions of dialogues 

• Annotated videos 

why is performing better not always good

• if a model outperforms humans in certain ways, it suggests that the model uses something (computational power) that humans do not 

cognitive plausibility

Models of human language must be cognitively plausible 

• Realistic input data 

  • Make realistic assumptions about the actual properties of the data available to children, e.g. noisy input 

• Memory and processing limitations 

  • Avoid unrealistically computation-heavy algorithms, e.g. remembering every past experience 

• Incrementality 

  • Process every piece of data when received 

aspects that play in language acquisition and speech processing

word segmentation - extract words form the speech stream

word mean - map each word form to the concept it represents in the outer world

syntax - combine words and construct well-formed sentences

semantics - interpret perceptual meaning of a phrase or sentence

• Difference between recognizng phonetic units and identifying word boundaries 

• In English, stress on words in speech is a good way to notice word boundaries  

how are word boundaries identified

through:

• isolated words: 

  • About 9% of utterances directed at English-learning infants 

  • Isolated words might be used to bootstrap word segmentation 

• Utterance boundaries 

  • we can always be sure about the start and end of a sentence and it can help guide word segmentation 

• Phonological cues 

  • Phonotactics, allophonic variation, prosodic cues, etc 

• Statistical cues 

  • Regularities in syllable sequences found in speech 

What is Motherese and does it help infants learn language

AKA infant-directed speech (IDS)

• Repetitive 

• Simplified grammar 

• Exaggerated intonation 

opposing results of IDS on cognitive an emotional development

what is transitional probabilities

the probability of a word following the given word

• Experimental findings suggest that children use transitional probabilities between words and syllables 

  • Big ripe apple 

  • Bi gripe apple 

• Word level: P(apple|ripe) > P(apple|gripe) 

• Syllable level: P(rip|big)> P(grip|bi) 

unsupervised word segmentation

• Transition between linguistic units within words are more predictable than transitions across word boundaries  

• We can test an utterance by identifying the # of successors of each phonetic subsequence and figure out how many words there are 

 

connectionist models

• Neural networks have been used to segment representations of speech using distributional cues 

• Inputs: 

  • Artificial corpora (shows sequences) 

  • Phonological transcriptions  

  • e.g. input -> an artificial sequence of letters 

    • B -> ba 

    • D -> dii 

    • G -> guuu 

  • Representation of letters: vectors of phonological features  

• Along with the representation input units, theres context units that pass through to hidden units and generates output units where the network is trained to predict the next letter/phoneme as output; then the hidden and context layers adjust based on accuracy of predictions 

• Consonants are easiest to predict because when we see consonants, we know what should come next, because we know what vowel follows each consonant. At the end of each word, predictability falls because anything could come after the current word.