Multimodal Linguistics II

What is multimodality and what does it recognize?

The use of multiple modes of communication, such as text, image, sound, and gesture, to create meaning

Recognizes that richer understanding is conveyed through the combination of different resources, and the way these modes are combined, arranged, and presented is crucial for communication

Name three examples of multimodal communication.

Childrens books (combine text with illlustrations)

Presentations (use text and spoken words)

Apps (use layout, colour and text)

What are the three key aspects of multimodality?

Multiple modes, interconnectedness and context + culture

What are the five types of modes in multimodal communication?

Linguistic (written and spoken language)

Visual (images, video)

Auditory (sound, music)

Gestural (hand gestures, body language)

Spatial (layout, arrangement of elements)

What is multimodality in relation to semiosis?

The co-presence of various signing systems (codes, sign modes, modalities) in one and the same text (as opposed to mono-modality) and their strategic integration to achieve communicative end

How can semiotic modes relate to each other in multimodal texts?

Semiotic modes may:

Reinforce each other (add to one another's meaning while saying roughly the same things)

Fulfill complementary roles (each contribute something special to the whole)

Be ordered hierarchically (one mode being dominant, others adding subordinate meanings)

What are three ways multimodality can be looked at according to the lecture?

1. As a text (communicative event/semiotic product) containing various modes that can be analyzed semiotically

2. As a general mental operation/cognitive faculty linking information from various modes to produce social meaning

3. As a skill text producers need to link and integrate various modes to make coherent and meaningful texts

One view of multimodality has it that semiotic modes are equally powerful. What does that mean?

Something that can be done verbally, can also be done with pictures or other modes

A multimodal text can make ... and is therefore ... than a mono-modal text.

meaning on various semiotic levels

sementically richer

How many semiotic modes are there?

Thats not clear

What are the three central semiotic modes identified in the lecture (excluding smell, touch, and taste)?

1. Language

2. Picture

3. Sound (which can be further split into music and noise)

What are the two large sensory groups that semiotic modes fall into?

Visual (eyes-vision) and Auditive (ears-hearing)

Why is it difficult to neatly tell modes apart?

Because modes interlock, overlap and mix

What are the three major kinds of multimodal texts based on media division?

Print

Audio

Audiovisual

What modalities are involved in Multimodal Text A (Print: Language-Image-Text)?

Language (Writing)

Picture (Static)

Typography (Paraverbal)

Typical text types include posters, print advertisements, newspaper articles, and packaging text

How does print text differ from audio text in terms of organization?

Print text is space-based, involving written language, static pictures, and typography/layout

Audio text is time-based (temporal organization), featuring spoken language with paraverbal qualities, music, and noise

What modalities are involved in Multimodal Text B (Radio: Audio-Text)?

Language (Speech)

Music (Sound)

Noise (Sound)

Typical text types include radio commercials, traffic information and podcasts

What modalities are involved in Multimodal Text C (TV: Audio-Visual Text)?

Language (Speech & Writing)

Picture (Dynamic & Static)

Music and Noise

Typography (static & dynamic)

Typical text types include TV commercials, documentaries, movies, and talk shows

What happens when visual and auditive channels combine?

The number of semiotic modes increases, thus the communicative potential and the semantic range/flexibility grow

What are the four layers (strata) in Kress and van Leeuwen's model of multimodal communication and which of them is on the content side, which on the expression side?

Content Side: Discourse and Design

Expression Side: Production and Distribution

The multi-layer (strata) model of multimodal communication sets out to demonstrate ...

... that meaning is made on a number of interconnected levels

What is "discourse" in Kress and van Leeuwen's multimodal communication model?

Discourse refers to "socially constructed and situated forms of knowledge of some aspects of reality" (e.g., environmental discourse, wine discourse, education discourse)

What is "design" in the multimodal communication model?

Design is the conceptualization (plan) of the form of semiotic products and events

What is "production" in the multimodal communication model?

The articulation in material form of semiotic products/events either in the form of a prototype or a final form

What is "distribution" in the multimodal communication model?

The technical 'recoding' of semiotic products and events for purposes of recording and/or distribution

What is the difference between "mode" and "medium" in multimodal theory?

Modes are signing resources/semiotic systems (codes) used to put thoughts/ideas/information into symbolic representation (language, picture, music, noise)

Media are tools, substances, technologies used to realize modes (stenciling, stamping, brushing, printing).

There is a reciprocal relationship: media make modes available, modes partly predetermine their media.

What are the four different ways "medium" can be conceived of?

Material/physical: sensory and material preconditions of information transfer

Technical: technologies and devices used to convey texts/messages

Social: institutions and their social fabric that design, produce, and distribute texts

Cultural: semiotic conventions of various genres and multimodal text types

Name seven generalized points about 'text'.

Are combinations and configurations of signs

Are of a certain complexity

Possess internal structure

Are relatively bounded

Are Coherent

Pursue a communicative function

Establish systematic connections to other texts in context

What is multimodality and what does it examine?

The use of multiple modes of communication, such as writing, image, sound, and gesture, to create meaning

How these modes are combined, arranged, and presented and the difference this makes for communication

What is semiosis and where is it located?

Semiosis sits at the intersection of meaning and matter, as well as between signs and sign-users (meaners)

The result of this dynamic process is texts/artefacts

What are semiotic modes and what function do they serve?

Semiotic modes are codified sign systems that supply semiotic resources - specific means for making meaning

Examples include angle or perspective in pictures, lexis and grammar in language, or tune in music

What are the three abstract semiotic levels that define/has a semiotic mode according to Bateman et al. (2017)?

A deformable perceptible materiality (canvas)

A classification of formal units and structures (paradigmatic/syntagmatic)

A level of discourse semantics for the abductive construction of contextual interpretations

Which three concepts can be used to describe multimodality?

Mediality, codality and modality

What distinguishes mediality, codality, and modality?

Mediality: perceptible materiality that can be mediated by technology

Codality: codified form with minimal semiotic units

Modality: sensory-motor modalities for perception and processing

How does Halliday (2005) describe the relationship between matter and meaning?

Humans inhabit two phenomenal realms: a world of matter and a world of meaning. Meaning relies on matter to make it accessible to a receiver; matter relies on meaning to organize it. Both are involved in all regions of experience.

Name the four communicative potentials from modes.

Picture, Language, Noise and Music

What are affordances and how do they function?

Affordances are properties of an object or environment that suggest or make possible actions an individual can perform

They emphasize the relationship between object and user - e.g., a chair affords sitting for many adults, but probably not for a toddler who cannot reach it

Name the seven key aspects of affordances.

1) Affordance = relationship between object and user

2) Includes actual possibilities for action and the perceived clues how to use it

3) Materiality

4) Provenance (historical and cultural development influences its meaning)

5) Meaning co-constructed through social use and norms

6) Relational affordances

7) Meaning potential based on past uses

What are semiotic constraints and how do they function?

Semiotic constraints are limitations on behavior and communication imposed by systems of signs and symbols or their environment

They arise from the inherent structure of the sign system itself and how it interacts with the material world

Name the five key concepts of semiotic constraints.

1) based in the system of signs

2) they create limitations

3) influences thought and behavior

4) operates on multiple levels

5) can be overcome through better understanding

What is the core concept from mediation?

Technology is not a neutral conduit but actively mediates communication

Name the three way how technology can mediate.

At the level of meaner potential

At the lebel of semiotic modes

At the level of meaning-making

How can technology mediate the meaner? Give examples.

Technology augments (erhöhen) individual sensory-motor and cognitve capacity

Glasses, hearing aids, translation apps and generative AI

How can technology mediate the semiotic mode? Give examples.

Technology reshapes communicative affordances and introduces constraints

Twitter character limits and writing systems

How can technology appear as meaning-maker? Givve Examples.

Technology functions as apparent interlocuter in communication

Generative AI and virtual humans

Name the twelve different views on text.

Text as basic unit in communication

Text as argest linguistic unit

Text as linear chain of signs

Text as written verbal entity

Text as delineated unit

Text as thematic/content unit

Text as context-determined unit

Text as more than the sum of its parts

Text as supra-linguistic sign complex

Text as a representation of knowledge

Text as functional unit

Text as textuality criteria

Describe the seven textuality criterias: Cohesion, Coherence, Intentionality, Acceptability, Informativity, Situationality and Intertextuality.

Cohesion = formal and grammatical connectedness of the text

Coherence = sense-continuity and logical-semantic connectedness of the elements of a text

Intentionality = text function as realized by certain forms

Acceptability = expectation that the text is gramatically well-formed

Informativity = texts are the more informative they more they contain unexpected and unknown information

Situationality = texts respond to the needs of the situation

Intertextuality = any text presupposes the knowledge of other, previous texts

How are text and context connected?

Context is that which surrounds a text and makes it meaningful and creates semiotic variation in text

Both text and context can be understood as analytical constructs

What are the three main modules of multimodal theory covered in this lecture?

1) Sign formation

2) Variously Codified Sign Systems (also known as Semiotic Modes)

3) Multimodal Text created through the integration of Semiotic Modes

Define human sensory-motor modalities, what does they involve and explain their interconnection.

Human sensory-motor modalities are the different ways the body senses and interacts with its environment

They involve sensory pathways that bring information to the brain and motor pathways that carry out actions

They are interconnected because the brain constantly integrates sensory information to plan and execute appropriate motor responses

What does sensory modalities include, what does motor modalities control?

Sight, sound, taste, smell, and touch (which involves pressure, temperature, pain, and proprioception)

Control voluntary and involuntary movements of skeletal and smooth muscles

What are the four sensory modalities calssified as "General Senses" which are spread throughout the body?

Touch (includes pressure and vibration)

Pain (nociception)

Temperature (thermoception)

Proprioception (The sense of the relative position of one's own parts of the body and strength of effort being employed in movement)

What are the four sensory modalities classified as "Special Senses" and what is the difference to the general senses?

Sight (vision)

Hearing (audition)

Smell (olfaction)

Taste (gustation)

These are more localized compared to the general senses

Name and describe the three types of motor modalities.

1) General Somatic Motor (GSM): Controls voluntary movements of skeletal muscles

2) General Visceral Motor (GVM): Controls involuntary movements of smooth muscles in organs, part of autonomic nervous system

3) Special Visceral Motor (SVM): Controls muscles derived from pharyngeal arches, such as facial and jaw muscles

What are the three types of interactions bewtween sensory and motor modalities?

Sensorimotor integration

Prediction and correction

Sensorimotor learning

What is sensorimotor integration and how does it work in everyday actions?

Sensorimotor integration is when the brain continuously combines sensory and motor information to control movement

For example seeing a cup requires the brain to use sensory input from the eyes to plan and execute a motor action to pick it up

How do sensorimotor learning and prediction/correction work together?

The brain predicts the sensory consequences of a movement - if there is a mismatch between the prediction and actual sensory feedback, corrective motor actions are generated

This interaction between sensory input and motor output is crucial for learning new skills like playing instruments or sports

What three factors help researchers understand animal communication better?

1) Sensorimotor potential (understanding how a species is likely to communicate based on their sensory-motor capabilities)

2) Eco-social environment consideration (understanding the context to understand the communication better)

3) Sensorimotor preference (what environment a species lives in (e.g., urban animals) indicates what communication systems they might favor)

Describe how corvids demonstrate multimodal communication.

Corvids use vocalizations (caws, rattles, clicks) to convey information about food, predators, and social status - they can mimic sounds including human speech, use context, timing, movement and specific call patterns

They enhance communication through multimodal signals by linking vocalizations with visual cues and body postures, creating multimodal texts

How did Beguš use technology to analyze whale communication and what new dimension was added?

Beguš used generative adversarial networks (GANs), a machine learning model that identifies patterns in datasets by listening and imitating - this technology makes it possible to learn about the structure and potentially the meaning of animal communication

Analyzing vowels added a completely new dimension of complexity beyond just whale clicks and inter-click timing

There is absolutely no doubt that all animals, and possibly all living beings, communicate. The more social their childhood the species and the longer the more ... - but the question is, ... ?

Complicated the communication systems

What are they communicating about

How do plants communicate with each other and their environment?

Plants communicate using chemical signals like volatile organic compounds (VOCs) through the air, underground through their root systems, and via electrical signals and networks

These signals warn neighbors about threats, attract pollinators, recruit beneficial organisms, and manage internal stress

What is the traditional assumption about plant behavior and how does modern research challenge it?

Plants are traditionally assumed to be noncognitive organisms whose behavior is completely determined either genetically or environmentally

However, a growing body of empirical research shows that many sophisticated cognitive capabilities traditionally assumed exclusive to animals are exhibited by plants too

How are communication and cognition interconnected and what does the dialectic relationship mean?

Communication and cognition are deeply interconnected through the relationship between information and behavior

Cognition enables communication by providing skills for understanding and producing messages, but communication also directly shapes cognition

This dialectic relationship means language is a fundamental part of cognitive function, influencing thought and behavior

Animal cognition is the study of the cognitive abilities of non-human animals. Define animal cognition according to Shettleworth (2001).

It includes perception, learning, memory and decision making—in short, all ways in which animals take in information about the world through the senses, process, retain and decide to act on it

What key areas are studied in animal cognition research (4) and how does researchers infer cognitive processes?

Social cognition (such as understanding others' minds)

Tool use and adaptation

Creativity and play

Philosophical questions about animal consciousness and morality

Researchers infer cognitive processes by observing animal behavior

What is multimodal cognition and how does it help humans navigate the world?

Human cognition is multimodal

To navigate through our complex world, we use prior experience to generate expectations and anticipate events - our expectations then shape the way we perceive our environment, including anticipating social behavior and responses to social actions

What does research by Friedrichs et al. (2022) reveal about the sensorimotor system? (2)

Activation of the sensorimotor system is associated with the expectation of an upcoming action from someone else in a predictable context

The sensorimotor system is predictive and highly susceptible to contextual features and prior experience, which means the entire cognitive system is as well

What is interpersonal predictive coding and what does it suggest about human brains?

Interpersonal predictive coding is the use of prior experience to generate expectations about the presence and consequent action of another person

This suggests humans possess Bayesian brains: the brain generates predictions and compares these to sensory input, which is especially advantageous in complex or noisy environments

According to Fox (2018), what crucial role do emotions play in multimodal communication and cognition?

Emotions are at the heart of how we understand the human mind and relationships within the social world - they play a crucial role in tuning us in to the most salient events, influencing decision-making, magnifying memory for important past events, and coloring perception of the future

Emotions help signal, both internally and externally, our deepest motivations and priorities

Which two key concepts are part of text-ness in context?

Multimodal Genre and Multimodal cohesion and coherence

What is a multimodal genre and what are these genres exspecially?

A multimodal genre is a type of communication that systematically combines two or more semiotic modes (such as linguistic, visual, aural, gestural, and spatial) to fulfill a particular social purpose

These genres are recurring, conventionalized forms of social action where conventions apply across all integrated modes

What is the difference between multimodal coherence and multimodal cohesion?

Multimodal coherence is the way different semiotic modes work together to create a unified and meaningful message in context - it connects the multimodal text to its context

Multimodal cohesion is the way different semiotic modes are linked together to create a cohesive and coherent multimodal text

What are the five factors that define genres?

Content: typical topics/propositions

Structure: typical speech acts and logical-semantic structure

Function: text function and social practice of producer and recipient

Style: typical uses of lexis and grammar

Form: outward shape of a text (typography, layout, size, positioning)

What role do genres play for text producers and recipients and what do they provide?

Genres are points of orientation for both text producer and text recipient

They provide cognitive patterns that help make and understand concrete texts of a certain type, serving as frameworks for how things get done when language is used to accomplish them

What does represent a significant semantic and temporal binding problem during face-to-face conversation?

Multiple layers of visual (and vocal) signals, plus their different onsets and offsets

According to Holler & Levinson (2019), how do multimodal messages compare to unimodal messages in terms of processing speed?

Multimodal messages appear to be processed faster than unimodal messages

What two mechanisms are proposed to facilitate multimodal language processing?

Multimodal gestalt recognition and multilevel prediction

What is the basis of processing mechanisms involved in multimodal language comprehension and what does this require?

The basis is hypothesized to be domain general, co-opted for communication and refined with domain-specific characteristics

This requires a situated framework that considers the multilayered, multimodal nature of language in conversational interaction

What does the concept of multimodel genres recognize?

That meaning is created through the interplay of various semiotic modes, rather than relying solely on written language

How does genre function as a multimodal gestalt principle?

Genre is a way of segmenting and binding during multimodal interaction across multiple levels with different sign systems

Generic structure functions as a multimodal gestalt principle that allows us to assemble multimodal information and interact within the temporal constraints that multimodal information imposes

How can a text be modelled in terms of cohesion and coherence?

A text can be modelled as a structure of propositions related to one another, forming a conceptual network that is cohesive and coherent - this structure consists of clause relations - logical/semantic connections between clauses (propositions)

What are the nine main types of clause relations in texts?

Causal (because), Conditional (if), Concessive (even though), Consecutive (therefore), Final (in order to), Adversative (but), Alternative (or), Explicative (which) and Instrumental (by)

What are frames and scripts in relation to coherence and how are these concepts stored?

Frames and scripts are global patterns of knowledge representation that help facilitate and organize our perception and experience of the world

Concepts are not stored separately but are put into these patterns, which help us process texts by allocating and integrating information into appropriate slots

The Global Patterns consist of a frame, slots/fillers and script. Explain these components.

Frame = a static bundle of knowledge (i.e. information)

Slots/fillers = every frame has characteristic slots which can be activated or provided with fillers (e.g. university slots: courses, lecture hall, library, students etc.)

Script = a dynamic bundle of knowledge which represents process or sequences of actions including all necessary objects and the acting subject

What is a macro-structure in text processing and what does it model?

A macro-structure is the summarized, central content of a text (the message in a nutshell)

The way we actively produce coherence

Recipients arrive at macrostructures by applying a number of cognitive procedures (macro rules). What cognitive procedures (macro-rules) do recipients use to arrive at macro-structures?

Recipients can leave out and select information, generalize and abstract from details and infer central meaning from the relation of various items

This process is part of generic structure formation

What is a superstructure according to van Dijk/Kintsch (1983)

Superstructures are schemata for conventional text forms - they are organizational patterns for specific text types

What are the components of an advertisement's superstructure?

Headline (attract attention, introduce product), Picture (attract attention, show product), Body Copy (describe/explain/evaluate/recommend product), Signature Line (state brand/product name), Slogan (sum up brand identity) and Standing Details (provide additional information)

You have the statement "The number one address in Vienna for classical music. And now also for rock." - explain frame and script in this context.

Frame = music store (Address/Vienna, Number one, Musician/Artists, Rock/Classical)

Script = concert (musician, venue, instrument, kneel, destroy instrument)

How can coherence be created even without explicit cohesive ties between different semiotic modes?

Coherence can be created through our understanding of a genre and all the elements that go into building a genre - this is what makes texts make sense in context

There is a fundamental and inherent quest for coherence