Multimodal Incidental Acquisition of Foreign Language Vocabulary Notes

Introduction to Multimodal Incidental Vocabulary Acquisition

• Learning another language can lead to:

  • Increased awareness and appreciation of other cultures.

  • Metalinguistic and cognitive benefits (e.g., Bialystok, 2008; Sanz, 2013; Yelland, Pollard, & Mercuri, 1993).

  • Improved employment opportunities in a global economy (Graddol, 2006).

• Incidental learning is a method of vocabulary acquisition through exposure to foreign language (FL) input during various tasks, followed by a surprise vocabulary test (Horst, 2005; Hulstijn, 2001; Laufer, 2001).

• Most research focuses on incidental FL vocabulary acquisition through:

  • Reading (e.g., Brown, Waring, & Donkaewbua, 2008; Horst, 2005; Hulstijn, Hollander, & Greidanus, 1996; Kweon & Kim, 2008; Pellicer-Sánchez & Schmitt, 2010; Pigada & Schmitt, 2006; Rott, 1999; Vidal, 2011; Waring & Takaki, 2003).

  • Listening (Brown et al., 2008; Vidal, 2011).

  • Reading-while-listening (Brown et al., 2008; Horst, Cobb, & Meara, 1998; Webb, Newton, & Chang, 2013).

• Incidental learning from spoken input is more difficult for learners (Brown et al., 2008; Vidal, 2011).

• Providing orthographic forms facilitates learning by:

  • Allowing learners to segment auditory word forms into manageable chunks.

• Meaning acquisition can be facilitated by:

  • Providing additional input in the learner's native language (NL).

Dual Language Input

• Lambert, Boehler, and Sidoti (1981) and Lambert and Holobow (1984) used combinations of languages in FL radio programs.

  • Students had access to both spoken and written input in the FL.

  • Other students received some of the input in their NL through the soundtrack or in writing.

• Lambert et al. (1981) found that participants better understood word meaning in context when they had received the FL in writing.

• Lambert & Holobow (1984) indicated that having the NL in the spoken input and FL in writing led to better contextual meaning comprehension.

• Lambert and Holobow (1984) concluded that the best input for learners was having the NL spoken dialogue combined with written FL script.

The Role of Pictorial Information

• Combining FL input with pictorial information facilitates meaning acquisition of new FL words.

• Pictures allow even complete beginners to access word meaning.

• Research has explored incidental acquisition of FL vocabulary through films with subtitles.

  • This includes auditory verbal information, written verbal information (subtitles), and pictorial information.

• Types of subtitles:

  • Standard Subtitling: FL in the soundtrack and NL in the subtitles.

  • Reversed Subtitles: FL in the subtitles and NL in the soundtrack.

  • Intralingual Subtitles: FL in both.

  • Intralingual subtitling in the NL (captioning) is used to make programs accessible to the hard of hearing and deaf community.

Early Studies on Films with Subtitles

• d’Ydewalle and Pavakanun (1995) investigated the effectiveness of films with subtitles in promoting incidental FL vocabulary acquisition.

  • Participants watched a 12-minute cartoon and completed a 5-alternative forced-choice (AFC) meaning recognition vocabulary test.

  • The study included all possible subtitling conditions and multiple control conditions.

  • Adult data showed vocabulary acquisition, with standard and reversed subtitles performing best.

  • Children showed no significant differences between the groups on vocabulary test scores.

• d’Ydewalle and Pavakanun (1997) concluded that considerable vocabulary acquisition occurred from watching a short subtitled video, and reversed subtitles enhanced vocabulary acquisition more than standard subtitles.

• d’Ydewalle and van de Poel (1999) focused on standard and reversed subtitles using a 10-minute still-motion movie with Danish or French as a FL.

  • For Danish as FL, there was a significant increase in written test performance in standard and reversed subtitle groups compared to a control group with NL intralingual subtitles.

  • In the auditory test, only the standard subtitles group performed better than the control group.

  • No significant French vocabulary acquisition was found.

  • The authors concluded that incidental vocabulary acquisition might be facilitated when the FL and the NL are similar.

• Koolstra and Beentjes (1999) found a small increment in vocabulary acquisition with children correctly recognizing the meaning of two more words if they had watched a FL video with NL subtitles (standard subtitles) compared to a control group.

• These studies suggest that incidental vocabulary acquisition from watching films with subtitles is possible.

Research on Multimodal Situations

• Gullberg, Roberts, and Dimroth (2012) investigated the incidental acquisition of FL words using a multimodal situation with FL auditory information and pictorial information.

  • Participants watched a short weather report in a FL and word form recognition was measured.

  • Participants were able to recognize 57.5% of the auditorily presented target words, indicating early incidental learning of word form.

• Bisson, van Heuven, Conklin, & Tunney (2013) observed incidental learning of form–meaning links in a multimodal situation.

  • Participants were presented with FL word forms (auditory and written) and line drawings.

  • The written FL word was relevant to a letter-search task.

  • Participants linked the FL word forms to meaning representations through the line drawings.

  • Participants were asked to explicitly learn FL words through a translation recognition task with feedback.

  • Higher accuracy scores were found in the explicit learning task for the words presented during the incidental learning task, both immediately and one week later.

• Bisson, van Heuven, Conklin, and Tunney (2014b) varied the number of exposures (2, 4, 6, and 8) to the FL words during the letter-search task.

  • As little as two exposures to words in the incidental learning task was sufficient for incidental vocabulary acquisition to occur.

  • The initial exposures had more of an impact on incidental learning than the subsequent exposures.

Intralingual Subtitle Situations

• In Bisson et al. (2013, 2014b), FL acquisition occurred when both the auditory and written word forms were in the FL.

  • Participants had to process the pictures to derive meaning from the FL word forms.

• Advantage: even complete beginners can benefit, as the pictures provide meaning information.

• In a multimodal situation, meaning information can be provided in one of the verbal streams (audio or written information).

Standard Subtitling

• Most subtitled films are presented with FL soundtrack and NL subtitles.

• The NL subtitles provide a translation of the dialogue, reducing the need to attend to the FL words.

• Incidental learning studies using FL films with NL subtitles concluded that learning occurred (d’Ydewalle & Pavakanun, 1995, 1997; d’Ydewalle & van de Poel, 1999; Koolstra & Beentjes, 1999), suggesting that the FL words in the soundtrack were processed.

• Saffran et al.'s (1997) "Saffran task" provided evidence for the processing of irrelevant auditory information during an incidental learning task.

  • Participants were exposed to a continuous recording of pseudowords while completing an unrelated computer task.

  • Adults and children correctly chose words from the tape with 59% accuracy in a surprise word form recognition test.

  • A further group of participants repeated the task on two consecutive days and achieved 73% (adults) and 68% (children) accuracy.

  • This study illustrated that both adults and children are able to use statistical information to extract words from a continuous speech stream.

Current Study Aims

• Assess the incidental acquisition of FL vocabulary using a situation similar to a film with standard subtitles.

  • Written NL translations of the FL auditory word forms are provided instead of FL written words.

  • Any learning that occurs can only be explained by the processing of FL auditory word forms.

• Determine if there is an added benefit of having access to pictures in addition to written translations.

• The study uses eye tracking to assess the allocation of attention to the different elements of the multimodal situation.

Predictions

• Having access to all three types of information (auditory FL word forms, written NL translations, and pictures) will benefit learning.

• Accuracy scores on both word conditions presented with meaning (written translations or both written translations and pictures) should be higher than those for the words presented with auditory FL word forms only.

• The experimental group will outperform the control group for words presented with meaning in the incidental learning phase.

• Participants will spend a considerable amount of time looking at the written word due to the letter-search task.

• The time spent looking at the pictures will predict scores on the explicit learning task and the recall and recognition test.

• The time spent looking at the NL word will predict the learning outcomes.

• The time spent looking at the pictures will decrease across the duration of the incidental learning phase.

Method

• Participants:

  • Sixty-six students from the University of Nottingham.

  • Native English speakers with no prior knowledge of Welsh.

  • Final sample included 28 participants in the experimental group (mean age = 23.1 years, 21 females) and 25 participants in the control group (mean age = 22.8 years, 18 females).

• Design:

  • Mixed design with group as a between-subject factor (control vs. experimental) and word condition as a within-subject factor (auditory FL word form only (A), auditory FL word form and written NL translation (AW), auditory FL word form and written NL translation with a picture illustrating the meaning (AWP)).

• Stimuli:

  • 78 auditory Welsh words, written English translations, and pictures.

  • Three lists of stimuli with 26 words from each condition were created to allow for counterbalancing across participants.

Procedure

• Three phases:

  • Phase 1: Incidental learning phase (experimental group only).

  • Phase 2: Explicit learning phase (both groups).

  • Phase 3: Recall and recognition phase (both groups).

• Phase 1: Incidental Learning

  • Letter-search task with eye-tracking (Eyelink 1000). Participants saw a series of hash symbols (#####) instead of the written NL translation that appeared in the other two word conditions.

  • Each stimulus presented six times: three times with a letter present and three times with a letter absent.

  • 468 trials split into six blocks of 78 trials.

  • The session started with the set-up and calibration of the eye-tracker using a 9-point calibration grid.

  • Each trial started with the presentation of the to-be-searched letter at the top of the screen for 500 ms.

  • Then the picture or a blank screen was displayed in the middle of the screen for 300 ms before the presentation of the written word (or hash symbols) at the bottom of the screen.

  • The auditory FL word onset was simultaneous with the onset of the written target string (word or hash symbols).

  • The end of each trial occurred when participants made a response, unless the auditory FL word was still playing; in those cases, the trial ended with the offset of the auditory FL word.

  • A short break was included after each block allowing for the recalibration of the eye-tracker.

• Phase 2: Explicit Learning

  • Translation recognition task.

  • Participants were presented simultaneously with a FL auditory word form and a possible NL written translation.

  • Participants indicated whether the written NL word was the correct translation of the FL auditory word form with a button press.

  • Feedback was provided (