Visual Speech Perception and Manual Communication

Visual Speech Perception and Speechreading

• Terminology and Historical Context

  • Lip Reading: An older, traditional term that specifically focused on obtaining linguistic information solely from the visual movement of the lips and mouth.
  • Speechreading: The preferred modern term in research and clinical literature. It reflects the understanding that visual speech perception involves multiple non-verbal cues beyond the mouth, including facial expressions, eye movements, head motion, and body gesture context.

• Speech Articulation and Visemes

  • Production Mechanism: Speech sounds (phonemes) are generated by moving specific articulators within the vocal tract in characteristic dynamic patterns.
  • Visemes: Defined as groups of speech sounds (phonemes) that share identical or nearly identical visual appearances on the lips and face. A single viseme can map to multiple distinct phonemes (e.g., the bilabial viseme includes the phonemes $/m/$, $/p/$, and $/b/$).
  • Visual Invisibility: Many articulators responsible for speech production are internal to the vocal tract and cannot be observed visually. Movements of the tongue body/root, pharynx, and larynx (vocal folds) are largely hidden from view.

• Inventory of English Viseme Examples

  • $16$: YU (as in cute)
  • $17$: AX (as in about)
  • $18$: IX (as in kisses)
  • $19$: IR (as in killer)
  • $20$: ER (as in bird)
  • $21$: AR (as in butter)
  • $22$: OR (as in calor)
  • $23$: UR (as in churn)
  • $24$: W (as in wet)
  • $25$: Y (as in yet)
  • $26$: R (as in red)
  • $27$: LL (as in let)
  • $28$: HX (as in head)
  • $29$: RX (post-vocalic $r$)
  • $30$: LX (post-vocalic $l$)
  • $31$: M (as in met)

• Perceptual Phenomena and Visual Interference

• The McGurk Effect: A perceptual phenomenon demonstrating multisensory integration where visual speech signals conflict with auditory speech signals, altering the auditory perception (e.g., hearing "ba" while seeing lips form "ga" yields the perception of "da").

• Inherent Ambiguity: Because many phonemes map to the same visual viseme, visual speech alone is inherently ambiguous, leading to humorous or misapplied misinterpretations (e.g., pop-culture demonstrations like Bad Lip Reading).

  • Auditory-Visual Integration Performance

• Sommers et al. (2005) Findings: Comparison of percent correct identification of speech units across Visual-only ($V$), Auditory-only ($A$), and Auditory-Visual ($AV$) modalities in younger vs. older adults:

  • Consonant Recognition: Visual-only ($V$) performance ranges from $10\%$ to $20\%$, Auditory-only ($A$) yields $60\%$ to $70\%$, while Auditory-Visual ($AV$) achieves $80\%$ to $90\%$.

  • Word Recognition: Visual-only ($V$) yields $10\%$ to $20\%$, Auditory-only ($A$) yields $50\%$ to $70\%$, and Auditory-Visual ($AV$) yields $80\%$ to $90\%$.

  • Sentence Recognition: Visual-only ($V$) drops below $10\%$, Auditory-only ($A$) achieves $60\%$ to $80\%$, whereas Auditory-Visual ($AV$) reaches $90\%$ to $100\%$.

  • Speechreading Enhancement and Sensory Cues

• Speechreading Enhancement: Defined as the quantitative improvement in speech recognition achieved when an auditory signal (or portion thereof) is combined with the visual speech signal.

• Low-Frequency Auditory Cues: Even extremely degraded or partial auditory input—such as isolated fundamental frequency ($F_0$) acoustic information—yields statistically significant improvements in speech comprehension compared to visual-only modes.

• Tactile Integration: Tactile stimulation devices (e.g., Tactaid) providing fundamental frequency vibrations through the skin offer supplemental benefit to visual speech perception.

  • Performance Variability and Developmental Dynamics

• Individual Performance Range: Visual-only speechreading ability in adults exhibits extreme inter-subject variability, ranging from under $10\%$ correct to over $70\%$ correct depending on assessment materials (Summerfield, 1992).

• Developmental Trajectory (Hnath-Chisholm et al., 1998): Speechreading ability improves systematically across childhood from ages $4$ to $12$ years.

  • Assessed using the Tactile Identification of Features Test (THRIFT) under Combined, Hearing, and Speech Reading conditions.

  • Performance increases monotonically with age and follows an exponential growth curve.

  • Talker, Environmental, and Contextual Factors

• Talker Variability Factors: Talker clarity varies widely based on articulation clarity, rate of speech, regional accents/dialects, presence of facial hair, and talker movement.

• Environmental Factors: Visual-only and auditory-visual accuracy depend heavily on distance from the talker, lighting quality/angles, and room acoustics.

• Contextual Guessing: Skilled speechreaders rely extensively on top-down contextual constraints, pragmatic cues, and linguistic redundancy to resolve ambiguous visual visemes.

  • Clinical Recommendations for Speechreading Optimization

• Maximize hearing health technology usage by wearing fitted hearing aids (HA) or cochlear implants (CI) to ensure optimal acoustic audibility.

• Maintain direct visual focus on the talker's facial features and articulators.

• Optimize environmental ergonomics: minimize distance, face light sources directly, and choose quiet acoustic environments.

• Implement active communication strategies: request clarification, rephrase messages, and repeat back confirmed portions of spoken statements.

  • Speechreading Performance: Hearing vs. Deaf/Hard of Hearing

• Owens and Blazek (1985): Compared normal hearing ($NH$) adults with sensorineural hearing loss ($SNHL$) adults using Vowel-Consonant-Vowel ($VCV$) syllables. Visual-only identification averaged $40\%$ for $NH$ and $41\%$ for $SNHL$, showing no statistically significant overall difference and similar error pattern distributions.

• Early-Onset Deafness Advantage (Auer and Bernstein, 2007): Individuals with early-onset severe-to-profound hearing loss show a distinct visual-only word speechreading advantage ($40\%$ word recognition accuracy vs. $19\%$ in normal-hearing peers).

• Efficacy of Speechreading Training
  • Walden et al. (1977) Study: Administered $14\,\text{hours}$ of intensive visual-only speechreading training to $31$ adults with bilateral sensorineural hearing loss using Consonant-Vowel ($CV$) syllables.

• Walden et al. (1977) Pre- vs. Post-Training Identification Rates:

  • Viseme $\theta, \eth$: Pre-training $90.1\%$, Post-training $99.4\%$, Chance level $10\%$
  • Viseme $f, v$: Pre-training $92.2\%$, Post-training $97.5\%$, Chance level $10\%$
  • Viseme $\int, 3$: Pre-training $71.9\%$, Post-training $94.5\%$, Chance level $10\%$
  • Viseme $s, z$: Pre-training $34.9\%$, Post-training $79.2\%$, Chance level $10\%$
  • Viseme $t, d, n, k, g, j$: Pre-training $90.6\%$, Post-training $99.3\%$, Chance level $15\%$
  • Viseme $p, b, m$: Pre-training $57.3\%$, Post-training $97.1\%$, Chance level $30\%$
  • Viseme $w$: Pre-training $74.8\%$, Post-training $92.9\%$, Chance level $5\%$
  • Viseme $r$: Pre-training $36.1\%$, Post-training $88.6\%$, Chance level $5\%$
  • Viseme $l$: Pre-training $72.6\%$, Post-training $93.4\%$, Chance level $5\%$

• Training Takeaway: Training significantly increases recognition of visemes as broad categories (notably for $s, z$ and $r$), but speechreaders still struggle to distinguish individual phonemes contained within the same viseme category.

  • Applications: Commercial, Forensic, and Artificial Intelligence

• Commercial Programs: Self-paced computer-assisted software programs (e.g., Read Our Lips) allow home practice, though clinical efficacy across diverse systems remains unverified.

• Forensic Speechreading: Expert Deaf lip-readers have been employed by law enforcement and intelligence services to translate silent video footage. However, lack of standardized certification and susceptibility to error have led legal courts to question its admissibility alongside other contested forensic disciplines.

• Automatic Speechreading: Artificial intelligence and machine learning models (e.g., Google DeepMind) extend facial recognition algorithms to automatically decode visemes from video streams.

  • Speechreading and Literacy in Deaf Education

• Harris and Moreno (2006) Study: Investigated Deaf children with severe-to-profound hearing loss who achieved reading ages within $10\,\text{months}$ of their chronological age.

• Good deaf readers demonstrate high visual-only speechreading skills.

• Speechreading proficiency correlates significantly with spelling achievement.

• Spelling errors made by proficient speechreaders are overwhelmingly phonetic (e.g., spelling BATTERY as BATRY).

• Speechreading scores demonstrate a strong negative correlation with reading delay.

• No significant differences in spoken language intelligibility, general IQ, or degree of unaided hearing loss exist between good readers and poor readers ($\ge 15\,\text{months}$ reading delay).

Manual Communication Systems and Educational Approaches

• Speech Sound Inaudibility and Visual Limitations

  • Acoustic limitations of hearing technology leave high-frequency speech sounds (e.g., the fricative phoneme $/s/$) difficult to perceive acoustic-only.
  • Visual speechreading cannot completely resolve this because many articulators are non-visible and distinct phonemes form identical visemes.
  • Supplemental manual communication systems provide phonemic clarity required to foster phonological awareness.

• Cued Speech

• Definition: A visual communication system that uses handshapes at specific face locations near the mouth in synchrony with spoken language.
• Structure: Handshapes represent consonant groups, while hand positions around the face represent vowel and diphthong groups.
• Historical Origin: Developed by R. Orin Cornett at Gallaudet University in the 1960s.
• Rationale: Designed to supplement speechreading by eliminating homophenous ambiguity, offering an un-ambiguous visual representation of English phonemes to support phonological awareness and literacy.

• American English Cued Speech Chart Specifications:
  • Consonant Handshapes:
    • Handshape 1: $/d, p, zh/$
    • Handshape 2: $/k, v, z, th/$ (voiced/unvoiced)
    • Handshape 3: $/h, r, s/$
    • Handshape 4: $/b, n, wh/$
    • Handshape 5: $/f, m, t/$ (or vowel alone)
    • Handshape 6: $/l, sh, w/$
    • Handshape 7: $/g, j, th/$
    • Handshape 8: $/ch, ng, y/$
  • Vowel Face Locations:
    • Mouth/Side of cheek: $/ee, ur/$
    • Chin: $/aw, e, ue/$
    • Throat: $/a, i, oo/$
    • Consonant alone position: Side of neck
    • $1/2" - 3/4"$ down (Chin to throat transition): $/uh/$
    • $1"$ forward (Chin forward movement): $/ah, oe/$
    • Dipthong movements: Side-to-chin or chin-to-throat for $/lay, oi/$ and $/ie, ou/$

• Educational Outcomes: Used across Auditory-Oral, Total Communication, and Bilingual-Bicultural programs. Early exposure to Cued Speech significantly improves reading ability, spelling accuracy, and phonological decoding in deaf children (Wandel, 1989; Leybaert, 2000).

  • Fingerspelling Systems

• American Sign Language (ASL) Fingerspelling: A one-handed manual alphabet system corresponding to the 26 letters of the English alphabet.

• British Sign Language (BSL) Fingerspelling: A two-handed manual alphabet system distinct from one-handed systems.

• Role in Natural Sign Language: Serves as a fundamental sub-component in ASL for proper nouns, technical vocabulary, brand names, and loan words from English.

  • Manually Coded English (MCE) Systems

• Definition: Artificial sign systems developed in the early 1970s to represent the grammar, syntax, and morphological structure of spoken English manually.

• Signing Exact English (SEE): Includes SEE-I (established 1971) and SEE-II (established 1972, active in educational settings). Synthesizes ASL signs, modified signs, and unique inventions to map English syntactically.

• Contact Sign / Pidgin Signed English (PSE): Non-standardized blend using ASL signs in English word order without full grammatical affixing.

• SEE-II Feature - Initialisms: Incorporates the fingerspelled first letter of an English word into the handshape of a sign (e.g., signing CAR using the 'C' handshape pulled apart rather than the natural ASL steering-wheel gesture).

• SEE-II Feature - Affixes: Adds unique signs for English prefixes, suffixes, and verb tenses (e.g., -s, -es, -ing, -ed, -tion, -ly) that do not exist as independent signs in natural ASL.

• Efficacy: Simpler syntax for hearing adults to learn than ASL; supports English phonic mastery and reading performance in deaf students (Luetke-Stahlman and Nielsen, 2003).

  • American Sign Language (ASL) vs. Spoken English

• Linguistic Contrast Matrix:

  • Grammar & Word Order: ASL utilizes Topic-Comment word order (Subject or Object elevated first); Spoken English utilizes Subject-Verb-Object (SVO).

  • Temporal Tense: ASL utilizes time indicator signs at the beginning of a sentence frame; Spoken English uses inflectional verb conjugations and auxiliary verbs.

  • Symbolic Form: ASL exhibits higher iconicity (signs physically resembling their referents); Spoken English is largely abstract and arbitrary.

  • Emotional Inflection: ASL conveys emotional tone through facial expressions and spatial intensity; Spoken English relies on acoustic intonation, pitch variations, and vocal stress.

  • Spatial Modifiers: ASL establishes fixed three-dimensional space in front of the signer to show distance, size, and relational syntax.

  • Mouthing: ASL incorporates specific mouth shapes/adverbials to function as linguistic intensifiers.

  • Sociolinguistics and Language Acquisition of ASL

• Milestone Parallelism: Deaf infants of Deaf signing parents acquire ASL native fluency on the exact same developmental trajectory and timeline as hearing children acquiring spoken languages (Newport & Meier, 1985).

• Demographics: Approximately $500,000$ native ASL signers and upwards of $2,000,000$ total users in the United States. Many fluent hearing signers are Children of Deaf Adults (CODAs).

• Cultural Arts: ASL features rich storytelling, theatrical traditions, and poetry leveraging spatial dynamics and visual art forms.

  • Educational Approaches for Deaf and Hard of Hearing Children

• Auditory-Oral Approach: Focuses exclusively on spoken language development, maximum listening output, and speechreading; strictly excludes manual signing.

• Total Communication (TC): Incorporates any and all communication modes tailored to child needs, including spoken English, hearing aid/CI amplification, Cued Speech, MCE, and fingerspelling simultaneously.

• Bilingual-Bicultural (Bi-Bi): Establishes ASL as the primary/native language ($L1$) for communication and instruction, teaching written English literacy and spoken English as a second language ($L2$).

  • The Auditory-Visual Continuum and Planning Framework

• Auditory-Visual Continuum Breakdown:
  • Auditory ($A$): Does not require visual support for clarification.
  • Mostly Auditory ($A_v$): Primary acoustic comprehension; needs occasional visual support for clarification.
  • Auditory-Visual ($AV$): Equal necessity for combined auditory and visual supports.
  • Mostly Visual ($V_a$): Primarily visual; understands limited spoken words or phrases.
  • Visual ($V$): Completely visual; does not understand spoken auditory phrases.

• A-to-V Instructional Matrix:
  • New Material: Delivered using balanced Auditory-Visual ($AV$) inputs to maximize comprehension.
  • Review Material: Delivered using Mostly Auditory ($A_v$) modalities to fade visual dependency.
  • Routine Material: Delivered using Auditory-only ($A$) modalities to strengthen acoustic auditory processing.