Notes on From Text Linguistics to Multimodality by Hartmut Stöckl (PAGE_BY_PAGE)

Page 1

  • Topic and scope
    • Hartmut Stöckl discusses multimodality as a field in linguistics, noting it has grown into a central area with many theoretical and methodological approaches.
    • The field is diverse: semiotics, text linguistics, discourse analysis, rhetoric, production-centered ethnographic approaches, and reception-oriented approaches.
    • There is a call for bridges across approaches to connect different regions of multimodal research, rather than treating them as separate silos.
    • The author cautions against letting the dominance of English (linguistic imperialism) unduly shape the field; encourages cross-fertilisation and overlap among approaches.
    • The present paper reflects on the state of multimodality research and foregrounds concepts and methods that derive from text linguistics and semiotics.
  • Aims and structure of the paper
    • The paper proposes to (a) outline a view of multimodality and clarify the concept of mode, (b) critically assess notions of genre, register, and style in multimodal texts, (c) develop an analytical framework for studying multimodal genres, and (d) illustrate the framework with samples from print genres and offer hypotheses about how multimodality research might develop further.
  • Core ideas introduced
    • Much of multimodal theory builds on text linguistics and semiotics, and a central claim is that multimodal phenomena are best understood through metaphorical transfers from traditional disciplines (e.g., how concepts from text linguistics map onto multimodal textures).
    • The goal is not to produce a single universal theory of multimodal texts but to understand how multimodal genres are characterized, how they differ, and how they form families or networks of genres.
  • Terminology and grounding ideas to expect
    • Terms to watch: mode, modality, multimodality, semiotics, text linguistics, discourse analysis, genre, register, style.
    • The paper promises a practical analytical framework for analyzing multimodal genre and aims to show its explanatory value in genre studies.

Page 2

  • Cognitive metaphor theory and the source-domain idea
    • Disciplines such as semiotics and text linguistics serve as the source domain from which concepts for multimodal phenomena are drawn.
    • Metaphorical mappings help understand multimodal textuality and how different modes link; however, mappings can highlight some aspects while obscuring others.
    • Multimodality research thrives on metaphorical projections when transferring concepts across modes.
  • Core claim about multimodal texts
    • Producing and understanding multimodal texts is primarily an act of multimodal genre recognition based on genre knowledge.
    • The author argues for a better understanding of what characterizes multimodal genres, how they differ, and how they form families or networks.
  • Plan for the paper (preview of sections)
    • (2) Multimodality and mode: clarifying the concept of mode.
    • (3) Notions of genre, register, and style and their applicability to multimodal texts.
    • (4) An analytical framework for text-linguistic study of multimodal genres, with illustrations from three print genres.
    • The paper closes with short hypotheses about the development of multimodality research.
  • Preview of key concepts to come
    • The discussion will set up a framework where mode is a central organizing concept, distinct from but related to resources, and where mode-coding, medium, and perception interact to shape multimodal texts.

Page 3

  • Why multimodality deserves its special status
    • Mode integration has become more intense with digital media, but is argued to be a historical given in human communication (symbolic origins of writing, from pictorial to phonographic systems; the modern typography revival of typo-pictoriality).
    • Multimodality aligns with human perception and cognition: combining multiple senses/codes tends to enhance cognitive ease and communicative effectiveness, often in playful or exploratory ways (e.g., rebus riddles, pictorial metaphor in advertising).
    • Translation/transcription between modes is fundamental: language contextualizes other modes; mutual contextualisation across modes is a foundational multimodal principle that contributes to a broader cultural semantics.
  • The conceptual challenge: defining “mode”
    • Definitions and terms differ: communicative resource, semiotic resource, mode of communication, representational mode. These terms emphasize instrumentality, materiality, or the type of signs used.
    • A useful view is to think of modes as having sets of resources that enable meaning-making; however, the terms are not always consistently understood.
    • Modes are characterized by functions tied to Halliday’s metafunctions (e.g., ideational, interpersonal, textual). This leads to two approaches to mode: empirical (usage-based, community-determined) and categorical (criteria-based delimitation).
    • The empirical view (Kress) emphasizes that a mode is what a community treats as a mode in its practices and is shaped by culture, media, and semiotic work, but it often lacks clear delimiting criteria.
    • The categorical approach seeks clear criteria to delineate modes, but may struggle with boundaries since signs can blend across modes in practice.

Page 4

  • Toward a clearer theory of mode: a three-level conception
    • The author argues for a categorical approach that ascribes to each mode three constitutive layers: coding, medium, and perception.
    • For a signing resource to claim mode status, it should be structured along meaning, form, and function.
    • Signs (mode elements) can be realized in different media (e.g., pictures as painting/photography) and mediated through different technologies (canvas, catalogue, screen).
    • Media influence the processing of signs and thus affect conventions, genres, and reading practices; not merely the message but the conduit shapes interpretation.
    • The sensory channel (e.g., visual, auditory) is the most basic layer and can differ from the medium through which signs are realized.
  • Simple typology and the role of media
    • A simple typology (Figure 3.1) links a mode to its typological components; language can be spoken, written, or signed, realized in different media.
    • The notion of a multimodal text often requires understanding how signs interlock across modalities within the same perceptual channel and how they contribute to a joint communicative design.
  • Distinctions and practical consequences
    • The idea of a Communicative Form (Kommunikationsform) captures the media- and situationally conditioned practices that realize genres.
    • Modes can blend or transmute (e.g., music becoming sound effects; writing acquiring pictorial qualities); such transmutations do not undermine the categorical approach but emphasize intermodal blending in practice.

Page 5

  • Developing a theory of multimodal texts: categorical approach details
    • The categorical approach uses three levels to describe modes in multimodal texts: coding, medium, and sensory channel.
    • For modes to claim status, signs must be internally structured along meaning, form, and function; although grammars differ (linear vs. spatial), there are shared basic properties (lexicogrammar in a broad sense).
    • Coding alone is not sufficient to characterize a mode; signs must be realized in materials/technologies and conveyed through different media.
    • Media influence reading and interpretation, shaping conventions and genres; the medium is not the message but affects how signs are produced and read.
  • The necessity of a clearer, more categorical concept of mode
    • Modes may merge when perceived through the same channel or when their material signs blend; delineating modes requires careful analysis of how signs are organized and how they function within the text.
  • Heuristic categories and the challenge of subdivision
    • A pragmatic typology helps in constructing a simple typology of multimodal texts based on whether they are print/electronic and whether they rely on audio or visual channels.
    • Three generalizations guide this typology:
      1) Complexity varies with the number of modes involved.
      2) Print and electronic texts are spatially organized; audio and audiovisual texts are temporally organized.
      3) Central modes (e.g., language and picture) may have medial variants; peripheral modes (e.g., typography) accompany central modes and depend on them.
  • Communicative forms and multimodal genres
    • The notion of Communicative Form aggregates medial, material, spatio-temporal, and situational aspects and governs how genres are instantiated in practice.
    • This leads to a categorical hierarchy: communicative form → multimodal text type → multimodal genre.
    • The relationship among form, type, and genre is central to organizing and analyzing multimodal texts.

Page 6

  • Typology implications for analysis and practice
    • The typology yields crude but useful text types akin to Werlich’s broad verbal text types (e.g., argumentative, narrative).
    • Contemporary German text linguistics uses the concept of communicative form to denote higher-level categories that regulate the realization of genres (e.g., weather forecast, report, interview, infographics within a newspaper).
  • Summary of key ideas so far
    • Modes are not just signals but structured resources with coding, medium, and perception layers.
    • Media influence how modes are realized and interpreted; signs can blend across modes within multimodal genres.
    • A clear, categorical understanding of mode and its constitutive levels helps to describe and compare multimodal texts across genres and media.

Page 7

  • Multimodal genre theory: grounding and rationale
    • The semiotic and functional paradigms helped establish text as a category applicable to multimodal artifacts; multimodal texts are now often called multimodal texts rather than traditional prose alone.
    • Four arguments for mapping text and textuality onto multimodal phenomena:
      1) Non-verbal modes (music, pictures) can themselves be texts.
      2) Verbal coherence transfers to intermodal connections (cohesion and coherence can extend across modes).
      3) A semiotic view treats any object as a text if it functions as a communicative instrument with a social purpose.
      4) The make-up and style of texts are influenced by the demands of a given situation and context, applying to multimodal texts as well.
  • Genre as central organizing concept
    • Unlike some abstract typologies, genre approaches emphasize producers’ and recipients’ knowledge and intuitions; genres are real social practices.
    • Figure 3.2 introduces three levels of genre description: pragmatic (communicative acts), cognitive (topics organized in patterns), and social (expressions/styles conditioned by social factors).
    • Genres serve as normative patterns for production and reception but exhibit variations and hybrids in contemporary use.
  • Hybridity and networks
    • Hybrids and networks are common in today’s genre landscapes, reflecting cross-genre influences and blending between genres.
    • Genre networks and spaces map how genres relate, cluster, and evolve, i.e., which genres are central, which are peripheral, and how they interconnect.

Page 8

  • Relation to systemic-functional tradition
    • Genre research hails from systemic-functional linguistics, focusing on how organizations of texts (e.g., Hasan’s Generic Structure Potential) express social purposes.
    • Register concerns the functional relationship between the situation and linguistic means; style is connected to register historically but may be treated as a broader or narrower notion depending on tradition.
    • In multimodal contexts, style can refer to subgenres or subregisters (e.g., banner ad vs. billboard; car ad vs. watch ad); multimodality tends to enlarge stylistic possibilities as more modes allow more variation.
  • Important takeaway
    • Multimodal genres expand the palette of stylistic variation and broaden the potential for design choices across modes.

Page 9

  • Why moving to multimodal genre is natural (Bateman, Delin, Henschel)
    • They argue that multimodal texts can be analyzed in terms of multimodal semiotic resources, with contributions to the overall communicative and social function of the text.
    • Three practical reasons to embrace multimodal genre analysis:
      1) Texts are agents of social processes; multimodal textuality should not be left out.
      2) Since genres organize texts, multimodal structures must consider all modes.
      3) The choice of which modes are used, and how, depends on situational aspects, linking context and form across all modes.
  • Modes-at-stages and the structure of multimodal texts
    • Two options for how modes realize stages in a text: (a) a stage realized by one mode with alternatives, or (b) a stage that itself is multimodal (multiple modes within one stage).
    • Example for (a): News genre where language reports facts and photographs authenticate the event and actors.
    • Example for (b): An advertisement where language and image combine within a single stage to construct an argument.
  • Concept of genre spaces
    • Bateman’s idea of genre spaces: a detailed overview of possible genres and networks occupying social/functional space.
    • Adamzik’s idea of Textsortennetze: functional, thematic, formal relations among genres.
  • Implication for research
    • There is a need for more empirical mapping of multimodal genre spaces to understand how genres are organized and connected in real-world practice.

Page 10

  • A model for multimodal genre analysis: overview
    • Several models exist for linguistic analysis of genres (e.g., GEM model by Bateman, Martin/Rose frameworks, etc.). Common features across models include:
    • Situation/context
    • Content/thematic units
    • Function/commu nicative acts
    • Form/markers and typical structure
    • These models largely focus on language; the challenge is to adapt or extend them to include intersemiotic (across modes) relations and the design of non-language modes.
    • The author notes that existing models often under-specify multimodality or fail to integrate mode connections comprehensively.
  • The proposed five-level analytical model (Figure 3.4)
    • Five interwoven levels for describing a multimodal text:
      1) Subdivision/Delineation: how the multimodal configuration is visually/auditorily partitioned; borders between text units.
      2) Communicative act structure: distribution of modes across stages and whether every mode can perform communicative actions.
      3) Thematic structure: topics contributed by the modes and their arrangement within the multimodal total.
      4) Mode connectedness: how modes connect to build a cohesive whole and perform rhetorical functions.
      5) Intertextuality: relation to neighboring genres in terms of form and function.

Page 11

  • The five-level model (continued) and its utility
    • The model foregrounds linking of modes and intertextuality as central to multimodal genre analysis.
    • It provides a practical framework for describing how modes work together to fulfill communicative purposes within a genre.
  • The analytical aim
    • The framework aims to help researchers describe both the internal organization of a multimodal text and its relations to other texts (genre spaces).

Page 12

  • Contrastive genre analysis: brief illustration (setup before details)
    • The author sets up a brief contrast among three multimodal print genres: news, advertisement, and infographics.
    • Each exemplar comes from printed media and could appear in a newspaper or magazine.
    • The analysis will examine the three exemplars at levels 1–3 and 5 (excluding mode connectedness at first), using a contrastive approach to reveal genre-specific patterns.
  • How the illustration will proceed
    • For each exemplar, the analysis will look at subdivision, communicative act structure, thematic structure, and intertextuality (level 4 is postponed to later discussion).
    • A table (Table 3.1) will summarize observations across the four analytical levels for each genre.

Page 13

  • Illustrative focus: the advertisement (Volkswagen Front Assist)
    • Internal subdivision: a three-part graphic structure typical of minimalist promotional communication—pictures, a line of writing (claim), and logo/slogan.
    • Comparison with the news report and infographics shows differences in the number and spatial distribution of textual parts.
    • Delineation: ads feature large white space and rough surface texture; strong frames or white space clearly delimit the ad from surrounding content; proximity and symmetry in other genres signal connections across genres.
    • Communicative act structure: three stages in the ad—pictures present, writing explains features, logo/generalization sums up.
    • Thematic structure: juxtaposition of two topics (fortune telling and safety features) creating an argumentative matrix that links seemingly unrelated topics.
    • Intertextuality: the ad has no explicit intertextual reference to neighboring genres, though it implicitly connects to technical discourse and VW advertising history.
  • Contrasts with the other exemplars (news and infographics)
    • News: one major topic introduced by headline, a photo provides a single detail, caption elaborates; intertextual links to neighboring genres are explicit.
    • Infographics: two stages (question and answer), visuals illustrate, with sources and authors stated to lend authority.
  • Table 3.1 (multimodal genre comparison on four analytical levels)
    • Level 1 Subdivision/Delimitation: ad uses pictures, claim, logo; news uses headline, body, photo, caption; infographic uses data-visuals (text, picture, labels).
    • Level 2 Communicative Act Structure: ad uses three stages; news uses four (announce, report, show actor, evaluate); infographic uses question and answer with sources.
    • Level 3 Thematic Structure: ad juxtaposes fortune-telling and safety; news uses narrative around Hollande; infographic centers on murder and country-specific data.
    • Level 4 Intertextuality: explicit network references in news; implicit references in ad; explicit data sources in infographics.

Page 14

  • Continuing the contrastive analysis (ad, news, infographic)
    • The advertisement’s thematic matrix relies on a juxtaposition that creates a comparative logic; news uses a narrative around a single topic with a visual emphasis on a key actor; infographics present a data-driven explicative matrix.
    • The advertisement’s intermodal connections (pictures and writing) create a three-stage structure with a central argument linking two seemingly unrelated topics.
    • The news report uses captions and imagery to ground a single ongoing event in time and place; infographics provide a data-driven, explanatory framework.
  • General observations from the contrastive analysis
    • Each genre has its own characteristic subdivision of text elements visible to readers upon first glance, which helps recognition even before content is parsed.
    • The way modes are distributed across stages (the communicative act structure) and how topics are organized (thematic structure) are key markers of genre identity.
    • The overall function of the text governs how these elements come together across modes.
  • Intertextuality in multimodal genres
    • Intertextual connections can be explicit (clear references to other genres or sources) or implicit (genre-internal references, branding, or technical discourse)

Page 15

  • Thematic and structural observations (summary from the example discussion)
    • In the ad: a juxtaposition of fortune-telling imagery with safety feature messaging; a semantic contrast (antithesis) ties the two topics together, though interpretation requires contextual work.
    • In the news: metaphorical language in captions (e.g., all eyes on him) literalized through imagery; intermodal coherence through descriptive/temporal cues.
    • In the infographics: the question–answer pattern structures the discourse; data labels and sources lend authority.
  • Intertextual relations across genres
    • News connects to neighboring genres through explicit cross-references (e.g., continuing coverage).
    • Infographics explicitly cite data sources and authors, reinforcing authority and linking to broader informational discourse.
    • Ads connect implicitly to broader technical and marketing discourse through branding and product-context references.

Page 16

  • Key empirical observations from the contrastive analysis
    • Graphic subdivision is a salient, genre-specific marker that correlates with the communicative act structure and thematic structure.
    • The relationship between content and mode distribution (which modes carry which topics) distinguishes genres; the overall text function constrains both mode usage and topic development.
    • Connections to neighboring texts (intertextuality) vary in explicitness and function, influencing how readers interpret the multimodal text within a wider genre network.
  • Important implication for multimodal genre analysis
    • The contrastive examination demonstrates the utility of the five-level model for describing and comparing multimodal genres across different print forms.

Page 17

  • Intersemiotic cohesion/coherence: framework introduction
    • The paper now turns to intersemiotic texture, or intersemiotic cohesion/coherence, which links signifying expressions across modes.
    • This dimension is central to multimodal genres because comprehension relies on making sense across modalities.
    • The discussion adopts a Hallidayan approach (cohesion and coherence) and adapts it to multimodal analysis, as in Liu/O’Halloran and Bateman works on intersemiotic texture.
  • Proposed levels of mode-connectedness (overview)
    • The author introduces a structure for analyzing how different semiotic modes connect to form a cohesive whole, setting up five levels (to be elaborated in the following pages).

Page 18

  • Level 1: Grammatical cohesion across modes
    • Grammatical cohesion concerns devices that point to other elements (e.g., pronouns, ellipses, parallel structures, conjunctions).
    • While primarily linguistic, signs across media can be grammatically cohesive (e.g., lines, frames, colors, and boxes used to link parts).
    • Example: In the Volkswagen ad, the relative pronoun what links to four pictures and their subjects, tying joint categories to a higher-level object class.
    • In infographics, alignment of color, size, and labels creates semantic groupings; in the news, a caption line under a photo links text and image.
  • Level 2: Lexical cohesion across modes
    • Lexical cohesion arises from repetition, paraphrase, or relational connections among coded signs across modes (e.g., co-hyponyms and hypernyms in imagery and text).
    • Advertising example: four co-hyponyms (driver, lady, courier, van man) connect to a hypernym like “what,” forming a thematic group.
    • Infographics rely on lexical fields (e.g., country names) to group information; news relies on metonymic links (e.g., cameras and images) to connect data points.

Page 19

  • Level 3: Coherence through relation propositions and scripts
    • Coherence is thought of as sense-continuity across modes, which can be captured by relational propositions (e.g., cause-effect) as in Rhetorical Structure Theory.
    • In the ad, relational propositions express a General–Specific relation: pictures illustrate and specify the product feature.
    • In the news, captions provide temporal/local background; in infographics, the topic (murder) is restated in the image and data elaborates the question (where are you most likely to be murdered).
  • Level 4: Frames and knowledge scripts
    • Coherence is also about how collaborating modes evoke frames or knowledge scripts; for example, the ad opens frames of fortune-telling versus automotive safety; the news and infographics tend to align with more stable frames (current events, data interpretation).
  • Level 5: Content relations and rhetorical patterns
    • Content expressed in modes can instantiate rhetorical patterns (metaphor, question–answer, etc.). Examples include the ad’s antithesis (fortune-telling vs. safety) and the use of a metaphor literalization in the news example.
  • Cross-genre observations
    • All three exemplars show grammatical cohesion across modes and explicit intermodal connections, with the infographic often showing the strongest formal connection due to its didactic purpose.
    • Lexical cohesion and the density of lexical items vary by genre, with different thematic relations and rhetorical patterns across modalities.

Page 20

  • Synthesis of the modal cohesion/ coherence comparison
    • Cross-genre similarities: pronouns and explicit cross-modal references are common; readability depends on consistent cross-modal cues.
    • Cross-genre differences: lexical cohesion density and the texture of content relations vary; intermodal coherence is mediated by frames/scripts and how modes align with these frames.
    • Overall, intermodal cohesion and coherence dynamics provide the most distinctive contrasts among genres and are central to multimodal texture.

Page 21

  • Critical reflections on the analytic model
    • The author argues the proposed model counters critiques that multimodality theorizes in a rigid, formalistic way without addressing content and context.
    • The model aims for a comprehensive, empirically testable approach to analyzing multimodal cohesion, thematic structure, and mode-linking across genres.
    • The approach is positioned as more detailed and potentially more useful than some existing models (e.g., Bateman, Royce) for mapping and understanding multimodal cohesion and stage-based genre structure.
  • Relevance to genre spaces and hybridity
    • The model emphasizes mapping multimodal genre spaces to understand how genres cluster and change as new media and design tools become available.
    • Hybridity and stylistic variation are seen as natural outcomes of multimedia proliferation, enabling more nuanced language–image linking and richer texture across genres.

Page 22

  • Three lines of inquiry for future research on multimodal texture
    • 1) Following communicative forms: language–image linking shares characteristics defined by medial and contextual factors; cross-genre similarities emerge from these forms.
    • 2) Large text types: global classes like argumentative, narrative, or explanatory shapes could determine texture; language–image linking would be similar across genre instances within the same type.
    • 3) Genre-specific textures: hard-wo rld genres (e.g., hard news, celebrity stories) may have distinct language–image linking patterns even if they share a medium and a general text function.
    • The author notes that real answers will require systematic empirical research and mapping of multimodal genre spaces.
  • Takeaway about the framework
    • The five-level model provides a practical scaffold for describing how modes are linked in multimodal genres and for exploring their position within larger genre spaces.

Page 23

  • References and theoretical anchors (selection)
    • The text cites a broad range of foundational works across multimodal analysis, text linguistics, semiotics, and genre studies (e.g., Halliday/Hasan, Kress, Jewitt, Bateman, Martin/Rose, van Leeuwen, Royce, Liu/O’Halloran, etc.).
    • It situates the GEM model and other frameworks within a broader landscape of multimodal discourse analysis.
  • Purpose of the reference section
    • To situate the proposed framework within established scholarly traditions and to guide readers toward foundational sources for deeper study.

Page 24

  • Continued references as a scholarly map
    • The references illustrate the interdisciplinary basis of multimodal analysis (linguistics, communication, media studies, semiotics).
    • They also show the evolving conversation about how to model, describe, and map multimodal genres and textures.
  • Illustrative figures and tables in the text
    • Figures referenced (3.1 to 3.8) and Table 3.1 are used to illuminate typologies, genre description levels, and exemplar comparisons.

Page 25

  • Closing framing and implications
    • The author reiterates that multimodality acts as a powerful catalyst in the development of genre groups and families, driven by new technologies and the expanded semiotic toolkit available for design.
    • The proliferation of multimodal styles is tied to both tool-driven creativity and the evolving demands of communication in a digital era.
    • The study closes with a forward-looking agenda: to empirically map multimodal genre spaces, test the five-level model across diverse datasets, and refine understandings of how language–image linking is conditioned by communicative forms, larger text types, and genre-specific textures.

List of Figures and Tables mentioned

  • Figure 3.1: Simple typology of multimodal texts and their corresponding modes
  • Figure 3.2: Three levels of genre description
  • Figure 3.3: Ways of mapping genre spaces and networks
  • Figure 3.4: A five-level analytical model for multimodal genre description
  • Figure 3.5: Hollande und Sarkozy in der Stichwahl (news image)
  • Figure 3.6: Volkswagen Front Assist (advertising image)
  • Figure 3.7: Murder International (infographics)
  • Figure 3.8: Levels of mode-connectedness
  • Table 3.1: Multimodal genre comparison on four analytical levels

Notes on scope and use

  • The notes above condense the key arguments, frameworks, and exemplars from the provided transcript pages. They are intended to serve as study notes that reflect the author’s positions and analytical approach for multimodal genre analysis, with an emphasis on the five-level model, cohesion/coherence across modes, and the mapping of genre spaces. If you plan to study or quote directly, please refer to the original text for exact wording and page-specific details.