Multimodality, Genre & Multimodal Discourse Analysis

Vocabulary flashcards covering key concepts from the lecture notes on multimodality, genre and multimodal discourse analysis.

Multimodality

Discourse studies

Linguistics multimodality

Two Definitions

1. Discourse Studies
   Texts create meaning not only through words but also through other modes like pictures, film, video, images, and sound.

2. Linguistic Multimodality
   The combination and integration of different modes (words, images, sounds, etc.) in structure, meaning, and function within social interaction.

MULTIMODAL DISCOURSE ANALYSIS

The study of how texts use multiple modes of communication (words, images, sound, video, layout, colour, etc.) together to create meaning. It examines how these semiotic tools are combined and designed in communication.

words + visuals + sounds + design

QUESTIONS MDA ASKS ① What meaning is being made in a text?

② How is meaning being made?

③ What resources have been drawn on to make the meaning in the text?

④ In what social environment is the meaning being made?

⑤ Whose interest and agency is at work in the making of the meaning?

Modes and Multimodal Texts

1. Printed- text: language(writing);Picture(still); Typography:

2. Audio-text: language(speech); Music: Noise

3. Audiovisual Text: language(speech/writing);Picture(static/moving); Music; Noise; Typography(static/dynamic)

4. Electronic text: language; typography; picture; music; noise;(navigation,Interaction)

Main Assumptions about Multimodal(-ity) Text

1. Mode Equality
   All modes (words, images, sound, etc.) can contribute to meaning — none are meaningless, even if not equally balanced.

2. Different Potentials
   Each mode has its own strengths and weaknesses, so they create different kinds of meaning.

3. Strategic Combination
   Authors/designers choose and combine modes on purpose, arranging them in space/time to work together and achieve the communicative goal.’

Multimodal texts = all modes matter + each mode has strengths + modes are combined strategically.

Modal Reach/Modal Logic

The communicative and representational potential of a mode — what it can do, where it is strong, and what limits it has in meaning-making.

Modal reach = what a mode can do (its strengths + limits) in communication.

Different Modes – Differing Reaches/Logics

4

1. Semantics

• Language: clear reference, logical structure

• Image: salience(виразність), foreground/background

2. Semiotics

• Language: linear, abstract signs + grammar

• Image: spatial layout, concrete signs

3. Cognition

• Language: slower, more detached from perception

• Image: fast processing, strong memory impact

4. Function

• Language: narrates, explains

• Image: shows, presents detail

Genre Goes Multimodal

• A genre is a staged, goal-oriented social process.

  • Social → involves interaction with others

  • Goal-oriented → used to achieve specific purposes

  • Staged → usually involves multiple steps

(Martin & Rose 2007: 8)

Multimodal Interpretation of Genre

• A multimodal artefact is an example of a genre realized through different modes (e.g., text, image, sound).

• Understanding multimodal genre means seeing how modes combine to structure stages and achieve goals.

Realization of Stages in Multimodal Text

1 Stage = 1 Mode

• Example: Print

• IMAGE: Shows actor/place

• LANGUAGE: Tells the news/event

1 Stage = 2+ Modes

• Example: Film

• LANGUAGE: Tells the news/event

• IMAGE: Tells the news/event

A Model for Describing Multimodal Genre

It’s a framework that helps you break down a text or document into layers like layout, navigation, language, content, rhetoric, and genre.

GeM Model – Primary Layers (Theory + Example)

• Layout Structure – Visual arrangement & hierarchy → Top-down structure, paragraphing, frames/boxes

• Navigation Structure – Guides document use → Magnifying tool, hyperlinks/tags, like/share, scroll

• Linguistic Structure – Verbal elements realizing layout → Name image elements, explain theme, interpret context

• Content Structure – Information content/“field” → Painter, art period, content + making + interpretation

• Rhetorical Structure – How content is argued/structured → Elaboration, background, circumstance (image as nucleus)

• Genre Structure – Stages of content delivery → Painting → Annotation/Navigation → Commentary → Description → Tags/Hyperlinks

Another Model for Multimodal Genre Analysis

• Subdivision/Delineation → Multimodal Configuration: How different modes are arranged and separated.

• Comm.-Act-Structure → Multimodal Stages: The sequence of communicative actions or steps in the text.

• Thematic Structure → Multimodal Themes: Main topics or recurring themes across modes.

• Mode Connectedness → Multimodal Coherence: How different modes (text, image, layout) connect to form a coherent whole.

• Intertextuality → Multimodal Discourse: How the text relates to other texts or cultural references.

Mode Combinations – Mode Linking

• Definition: The linking and integration of different semiotic modes (text, image, layout, sound, etc.) in a text.

• Key Concepts:

  • Interplay between modes – how modes interact (Jewitt, 2014)

  • Intermodal harmony – smooth coordination between modes (Norris & Maier, 2014)

  • Intersemiotic complementarity – modes complement each other (Royce, 1998)

  • Multimodal cohesion & coherence – how modes combine to create meaning (Bateman, 2014)

Interplay – Harmony – Coherence (of Modes)

• Interplay – How different modes interact and influence each other.

• Harmony – How modes are coordinated smoothly to work together.

• Coherence – How modes integrate to create a unified, meaningful whole.

• Key Idea: Multimodality relies on linking modes so that formal, semantic, and functional aspects work together (Stöckl, 2019).

Multimodal Text-Connectedness

Media Genre I – INNS

• Definition: INNS = Interactive Narrative, News, and Social Media.

• Core Idea: Media genres with specific conventions and formats that shape how multimodal texts are produced and understood.

• Features:

  • Combine multiple modes of communication (text, image, video, interaction).

  • Encourage user interaction and immersion.

  • Blend news reporting with narrative and social media dynamics.

• Multimodal: combines image + headline (nucleus) + caption/context (satellites).

• Conventions: News angle, values, journalistic style.

• Function: Immersive reporting, guides interpretation, reader interaction.

• Example: News photo + headline + explanatory caption.

Multimodal Text-Connectedness

Grammatical Cohesion –

How grammar links parts of the text across modes.

Lexical Cohesion

Repetition, synonyms, or related words that connect meaning.

Coherence - Relational Propositions

Logical or semantic relationships between ideas.

Coherence - Frames/Script

Shared knowledge or patterns that help interpret content.

Coherence – Rhetorical Figures

Use of rhetorical structures (e.g., contrast, cause-effect) to connect ideas.

Genre Properties of INNSs

• Core Idea: Some news genres use a central image to convey the main point (the nucleus) of a story.

• Structure:

  1. Nucleus: Central image + headline → conveys the story’s core.

  2. Satellites: Extended caption elaborates the story with additional details.

• Function: The image drives reader understanding; text supports and elaborates.

TYPICAL GENRE FEATURES (Caple 2013: 127–131)

① large, aesthetically pleasing image

② heading + caption intro – both directly relating to image

③ verbal-visual play between image, headline, caption

④ extended caption – describing image + providing context

⑤ evaluative stance – expressing editorial/ideological angles

Rhetorical Structure of INNS

1. Experiential Orientation (satellite)

   • Clarifying and sharpening our understanding of the image

   • Circumstance

     Process

     Participants

     AIMED

     towards the image

   • Example: Members of the Royal Canadian Police carry portraits of slain colleagues at a memorial.

2. Contextual Extension (Satellites)

   • Extending beyond what is needed experientially to understand the image

   • News

     -Angle

     -Values

     AIMED

     towards the 'news'

   • Example: Investigation of a suspected marijuana-growing operation; James Rosko, 46, committed suicide.

Key Idea: The central image conveys the core story, while captions provide context and elaboration.

Evaluative Orientation: tends to rely heavily on play, calling on experiences of the reader to decode(Heading(prosodic Tail)); tends to present represented participants in an eye-catching, aesthetically peasing manner (image)

Media Genre II – Stand-alone Infographic

• Definition: A form of visual communication.

• Purpose: Presents complex information clearly, succinctly(лаконічно), and effectively.

• Features:

  • Heavy use of graphical elements (charts, icons, visuals).

  • Designed for quick comprehension and audience engagement.

  • Functions independently (doesn’t rely on supporting text).

• Visual communication form.

• Uses graphics, charts, icons to simplify complex data.

• Function: Quick comprehension, engagement, independent of supporting text.

• Example: Infographic on global homicide rates with visuals + minimal text.

Function of the Generic Stages in INNSs

Multimodal Cohesion/Coherence Applied

• Grammatical Cohesion – Links through graphics, e.g., size or color of tags.

• Lexical Cohesion – Word choice creating connections, e.g., murder, to be murdered, homicide rates, countries (linked to images/graphics).

• Relational Propositions – Logical relationships, e.g., elaboration of a general question.

• Frames & Scripts – Shared knowledge or cultural schemas, e.g., Halloween (spookiness/fear), murder, statistics.

• Rhetorical Figures – Stylistic and structural devices: question–answer structure, part/whole metonymy (world/countries).