Assessing Pronunciation: Exhaustive Study Guide on Theories, Research, and Challenges

Introduction to Pronunciation Assessment and Perceptual Salience

• Salience of Accents: Accents are among the most perceptually salient aspects of spoken language.
• Listener Identification Abilities: Research by Munro, Derwing, & Burgess (2010) demonstrates that linguistically untrained listeners can distinguish between native and non-native speakers even under nonoptimal conditions, such as when speech is played backwards.
• Cross-Language Perception: Listeners can identify foreign accents in languages they do not even understand (Major, 2007).
• Historical Context: The Shibboleth Test: One of the earliest documented language tests is the biblical Shibboleth test (Book of Judges). Members of warring tribes were identified by their pronunciation of the word shibboleth (meaning 'sheave of wheat').
  • Those who used an $/s/$ sound instead of an $/ʃ/$ (sh) sound at the syllable onset faced fatal consequences.
• Modern High-Stakes Applications: Speech analysis is used by "experts" to determine the legitimacy of asylum seekers' claims based on perceived group identity (Fraser, 2009).
• Fairness and Stereotyping: Identity tests are not foolproof and raise concerns regarding fairness. It is often unclear if unfavorable responses are triggered by the speech signal itself or by listener expectations resulting from linguistic stereotyping (Kang & Rubin, 2009).

The Shift Away from Native-Speaker Norms in L2 Instruction

• The Gold Standard Myth: While the native speaker has long been viewed as the "gold standard" of language knowledge (Levis, 2005), applied linguists now view the eradication of foreign accents as an unsuitable goal for adult L2 learners for several reasons:
  1. Unrealistic and Undesirable Goals: Native-like phonology is unrealistic for the majority of adults; additionally, accent and identity are deeply intertwined (Gatbonton & Trofimovich, 2008).
  2. Functional Adequacy: L2 speakers do not require native-like accents to integrate into society or succeed in academic and professional tasks (Derwing & Munro, 2009).
  3. Global English/ELF: The emergence of English as an international lingua franca makes native norms inappropriate in many English as a Foreign Language (EFL) settings (Jenkins, 2002).
  4. Native Diversity: Many native speakers do not use prestige varieties like Received Pronunciation (RP) or General American English.
• Current Consensus: Effective oral communication focuses on being understandable to interlocutors and successfully conveying messages rather than accent reduction (Jenkins, 2002).

Historical Foundations: Robert Lado and Discrete-Point Testing

• The Association with "Neglect": Pronunciation has historically been de-emphasized or neglected in L2 communicative teaching based on the belief that it is extraneous to communicative competence (Celce-Murcia et al., 2010).
• Counter-Arguments: Morley (1991) argued that "intelligible pronunciation is an essential component of communicative competence" and that ignoring it is an "abrogation of professional responsibility" due to potential social and professional disadvantages.
• Robert Lado’s Language Testing (1961): This seminal work remains the most comprehensive treatment of the subject. Lado provided guidelines for testing:
  • Perception and production of individual sounds (segments).
  • Stress and intonation (suprasegmentals).
• Outdated Theoretical Premise: Lado operated under the idea that "language is a system of habits." He believed that any difference between a learner's first language ($L1$) and target language ($L2$) would result in a problem that must be tested.
• Nuanced Modern Findings: Accurate perception/production is mediated by how different a sound is from existing $L1$ categories. If no difference is perceived, the learner will substitute the $L1$ sound. Factors like phonetic environment and lexical frequency also influence performance (Flege, Schirru, & MacKay, 2003).

Challenges of Objective Written Pronunciation Tests

• Paper-and-Pencil Alternatives: Lado proposed written multiple-choice tests to avoid the subjective scoring and logistics of recording/marking speech samples.
• Japan’s National Centre Test Example:
  • Segmental items: Selecting a word with a different underlined sound (e.g., boot, goose, proof, and wool; the vowel in wool $/ʆ/$ differs from the $/u/$ in others).
  • Word stress items: Selecting a word with the same primary stress pattern as a prompt (e.g., both fortunately and elevator have primary stress on the first syllable).
• Validity Issues: Buck (1989) examined these items and found them ineffective:
  • Internal consistency coefficients ($KR-20$) were unacceptably low, ranging from $-.89$ to $.54$.
  • Correlations between written scores and actual oral production were low ($.25$ to $.50$).
  • Correlations with read-aloud and extemporaneous ratings were even lower ($.18$ to $.43$).
• Recommendation: Written tests for oral skills lack empirical evidence of reliability or validity and should be discontinued for high-stakes purposes.

Theoretical Conceptualizations of Communicative Competence

• Bachman’s Framework (1990): In the communicative language ability framework, "phonology/graphology" (handwriting legibility) is included but poorly defined.
• The "Channel" Argument: Bachman and Palmer (1982) initially omitted phonology because they viewed it as a channel rather than a component, arguing it only becomes relevant at a critical level where communication breaks down.
• Competing Ideologies (Levis, 2005):
  1. The Nativeness Principle: Aim is to achieve native-like pronunciation by reducing $L1$ traces. This aligns with the construct of "accentedness."
  2. The Intelligibility Principle: Aim is to be understandable. This is widely endorsed by researchers as the key to assessment (Levis, 2006).

Defining Intelligibility, Comprehensibility, and Accentedness

• Multifarious Definitions: Variations in definitions make cross-study comparisons difficult (Isaacs, 2008).
• Broad vs. Narrow Definitions: In the broad sense, intelligibility is synonymous with comprehensibility. In the narrow sense (Derwing & Munro, 1997):
  • Intelligibility: The actual amount of speech understood by listeners, typically measured by the proportion of an utterance correctly transcribed orthographically.
  • Comprehensibility: The subjective ease with which a listener understands speech, measured using a rating scale (analogous to a thermometer measuring temperature).
  • Accentedness: Listener perceptions of how different an L2 utterance sounds from native-speaker norms.
• Construct Independence: Research shows that comprehensibility and accentedness are partially independent; a speaker can be perfectly understandable (high comprehensibility) despite a heavy accent (Derwing & Munro, 2009).

Shortcomings in Current Rating Scales

• Omission and Inconsistency: The Common European Framework of Reference (CEFR) omitted pronunciation descriptors due to high misfit values (North, 2000). The ACTFL guidelines mention pronunciation in some levels but completely omit it in level 2 (novice mid).
• Vague Descriptors:
  • IELTS Band 4: "mispronunciations are frequent and cause some difficulty for the listener."
  • TOEFL iBT Level 2: "may require significant listener effort."
• Terminology Confusion: In IELTS, "pronunciation" may include both segments and suprasegmentals. In TOEFL, the pairing of "pronunciation" and "intonation" suggests the former refers only to segments.
• Relativistic Terms: Morley’s (1991) Speech Intelligibility Index uses terms like "basically unintelligible" and "reasonably intelligible," which offer little concrete guidance to raters.

Research on Linguistic Influences and Teacher Priorities

• International Teaching Assistants (ITAs): Pronunciation is often a scapegoat for broader communication barriers, such as acculturation issues or listener bias. Targeted training should focus on the most consequential features for intelligibility.
• The "Lingua Franca Core" (Jenkins, 2002): A proposed set of pronunciation features essential for global English, though critics argue it is based on a limited data set.
• Prosodic vs. Segmental Effects:
  • Prosody (Suprasegmentals): Aspects like stress and timing have a direct effect on intelligibility (Hahn, 2004).
  • Segments: Some contrasts (e.g., $/s/$ vs. $/ʃ/$) are more critical than others (e.g., $/θ/$ vs. $/f/$) based on the functional load principle (Munro & Derwing, 2006).
• Deconstructing Comprehensibility (Isaacs & Trofimovich, 2012): A study of 40 Francophone learners identified linguistic domains that distinguish levels:
  • Low levels: Lexical richness and fluency differentiate learners.
  • High levels: Grammatical and discourse-level measures differentiate learners.
  • All levels: Word stress is a significant differentiator.

The Influence of Rater Characteristics

• Cognitive Variables: Isaacs & Trofimovich (2010, 2011) tested phonological memory, attention control, and musical ability.
  • No significant bias was found for phonological memory or attention control.
  • Musical Ability: Musical raters were generally more severe in their judgments of comprehensibility and accentedness, likely due to heightened pitch sensitivity.
• Rater Familiarity: Studies on the effect of rater familiarity with a specific accent show inconsistent results, with some suggesting it improves scores and others showing no effect.
• Listener Attitudes: Native speakers' perceptions of understanding are often mediated by their attitudes toward the speaker’s $L1$ (Lindemann, 2002).

Automated Scoring and Technological Innovations

• Speech Recognition Algorithms: Systems like Pearson’s Versant English Test (formerly Phonepass) show high correlations with human ratings.
• Advantages: Machine scoring averages out individual rater idiosyncrasies.
• Validity Concerns: Machines may not attend to the same speech properties as humans. For example, humans perceive stressed syllables as higher in pitch than spectral analysis confirms (Crystal, 2008).
• Construct Narrowing: Automated systems are better at constrained tasks (sentence repetition) than spontaneous communication, which may narrow the assessment of speaking ability.
• Human as Gold Standard: Human interlocutors remain the ultimate arbiter of successful communication, and automated systems should conform to human-mediated standards.

Future Directions and Summary of Challenges

• Construct Definition: Researchers must filter out "accentedness" from proficiency scales and focus on "comprehensibility" descriptors that do not rely on native-speaker standards.
• Interactive Tasks: Research should move away from monologic tasks (speaking into a microphone) toward collaborative, dyadic tasks that reflect real-world interaction.
• The Post-Lado Era: There is an urgent need to reinvigorate the conversation on pronunciation within language assessment circles to move beyond the "neglect" of previous decades.
• Diagnostic Tools: Prioritizing empirical studies that isolate features (segmental or suprasegmental) to develop better diagnostic information for L2 teachers regarding communication breakdowns.