Multimodal Linguistics III

What is multimodal discourse analysis?

An approach that examines multiple modes of communication (text, colour, images) and analyzes not just how individual modes communicate, but how they interact with one another to create semiotic meaning

Name the five different approaches to multimodal text analysis mentioned in the lecture.

Mediated Discourse Analysis (MDA)

Systemic Functional Multimodal Discourse Analysis,

Social Semiotics

Multimodal (Inter)action Analysis

Multimodal Conversation Analysis

Which three multimodal analysis approaches focus on image-centric texts?

Mediated Discourse Analysis, Systemic Functional Multimodal Discourse Analysis and Social Semiotics (visual communication)

Which two multimodal analysis approaches focus on interaction and speech?

Multimodal (Inter)action Analysis and Multimodal Conversation Analysis (gesture, behaviour and speech)

What are the three main challenges that multimodal discourse analysis needs to overcome?

What are the boundaries for a multimodal text?

What are the different semiotic modes operating in a multimodal text?

How should these different semiotic modes be broken down for analysis?

What are the four stages for analyzing multimodal texts?

Stage 1 - Notice communicative situation (function/purpose/goals)

Stage 2 - Recognize visual 'gestalts' and form signs (genre conventions)

Stage 3 - Work out text content (what is depicted in which context)

Stage 4 - Integrate different semiotic modes (how they relate to each other)

What fundamental assumption does multimodal discourse analysis make about semiotic artefacts?

Every semiotic artefact operates on three levels called 'meta-functions', which are adapted from Michael Halliday's functional grammar to accommodate multimodal discourse

What is the claim of visual grammar?

There are formal configurations/patterns in images, which come equipped with certain, more or less stable social meanings

What are the three original metafunctions from Halliday's functional grammar?

Ideational function (semiotic systems represent aspects of the outside world, e.g. statements)

Interpersonal function (semiotic system construes a social relationship between producer viewer and objects represented, e.g. speech acts)

Textual function (semiotic systems form text)

What are the three image-based metafunctions for the multimodal text (communicative artefacts), corresponding to Halliday's original metafunctions?

Representation (what is depicted)

Interaction (relationship between image and viewer)

Composition (how elements are arranged)

What are the four elements that contribute to REPRESENTATION?

Narrative or Conceptual --> Setting and Look of Participants

When is a image narrative?

If it depicts actions, processes or events

What does conceptual structures do and what does they not contain?

They represent participants (visual elements) in their class, structure, properties and meaning

Do not contain vectors --> they show objects

Name the three subcategories of conceptual.

Type of - attributive - whole-part

In terms of 'setting', pictorial meaning does not only emerge from the represented participants and their relations, but also from ...

... the circumstances of the action or object representation (context/situation)

What does the element 'Look of participants' mean?

Any optical detail of the main visual elements (participants) colours in the visual meaning expressed by the configuration of actors/participants (e.g. clothes, appearance)

What is a transactional narrative image?

A unidirectional transaction where a person performs an action directed at an object

Example: an athlete shooting hoops - the vector connects the athlete/basketball to the basket

Name the five types of narrative representations in images.

Non-transactional (intransitive actions)

Reactional (formed by eye-lines or gaze)

Conversion (circular action)

Speech process (participants speaking)

Mental process (thinking/reflecting)

Type one of conceptual pictures, 'type of', has three major types of structures, name them.

Classificational structures, Analytical structures and Symbolical structures

Descripe a sample for type two of conceptual pictures, 'whole-part'.

A shoe (whole) is shown as consisting of various component parts ('exhaustive analytical process')

What are 'circumstances' in visual meaning and what two categories/elements do they fall into?

Circumstances are secondary represented participants that are not directly part of the narrative or conceptual structure but provide additional detail

Elements of the setting and Look of the participants

What is INTERACTION about?

Interaction is all about how viewers are (socially) positioned to pictorial content

What are the four elements that contribute to INTERACTION?

Action Orientation (Image Act/Gaze) - Distance - Perspective Angle - Coding Orientation

What is action orientation (image act/gaze) about and what are the two subcategories?

Pictures can offer information or demand action

Demand (directive - Kontakt zwischen dargestellten Teilnehmern und Betrachter besteht) and offer (representative - Kontakt besteht nicht)

What is distance about and what are the three subcategories?

Relationships between represented and viewer - how far is it physically away and/or how far is the social distance?

Close - intimacy/precision, medium - social relationship, long - impersonal

What is perspective about and what are the three subcategories?

How are viewing it and from which perspective?

Frontal (involvement/sympathy), high (power/dominance) and low (fear/intimidation)

What is coding orientation about and what are the four subcategories?

The relationship between content and reality

Sensory (hyperrealistische Darstellung), technological (Röntgenbilder, Modelle), naturalistic (Fotografie) and abstract (Skizze, Karte, Diagramm)

What is COMPOSITION about?

Organises and guides the attention of the viewer during visual understanding

Relates the representational and interactive meaning of the picture to each other through interrelated systems

What are the three elements that contribute to COMPOSITION?

Information value, framing and salience

What is information value about and what are the three subcategories? (examples for subcategories - week 10 page 22, 24 and 26)

The placement of elements and how they transport the specific informations through their position

Left-right structure, top-bottom structure and centre-margin strucutre

Semiotic modes ... in contexts to form multimodal texts.

co-operate

Is there a mono-modal text?

No (all multimodal)

What are the two subcategories of framing?

Connected and disconnected

Through what four subcategories can salience be expressed?

Shape/size, colour/lighting, contrast and visual effects

We experience events - what does these events do? (3)

Units demarcate what happens (Grenzen Geschehen ein), inform our actions and reactions and help us react correctly to the things happening in the world

What do we subjectively experience rather than raw sensory data?

Events

(These event units demarcate what happens, help us encode and retrieve information in memory, support inferences about what will happen next, inform our actions and reactions, and fit our understanding of causal origins and outcomes - Grenzen das Geschehen ab und helfen uns es einzuordnen, zu verarbeiten und richtig darauf zu reagieren)

What are the two types of processes involved in event representation?

Bottom-up (low-level) processes:

Reacting to prominence markers such as edge detection, contrast, boundaries, or thresholds

Top-down (higher-order) processes:

Prior experience, learning, goals, values, and information about causality

What is the difference between temporally bounded and unbounded events and which endpoints are more salient?

Bounded events have a natural, inherent endpoint and are temporally bounded

Unbounded events are unspecified about when they end and are temporally unbounded

Research shows that endpoints of bounded events are more salient than those of unbounded events

Why are event boundaries important for cognition?

Event boundaries serve as anchors for comprehending, learning, and describing events

Sharply defined boundaries carry richer information and make it easier to individuate events, compare them to other events, generalize from them, and remember them

What is the bi-directional relationship between events, boundaries, and attention?

Events are connected to both boundaries and attention

Both boundaries and attention can shape and influence whether we experience something as a cohesive unit or as a fragmentary and disconnected stream of sensory data - the relationship works in both directions

What are the three central concepts that distinguish how we process information and how do they differ?

Prominence (standing out):

Not all things that stand out get our attention or matter

Salience (attention):

Not all things that get our attention stand out or matter

Importance (mattering):

Not all things that matter necessarily stand out or get immediate attention

These three concepts are related but distinct aspects of information processing

Name two tools for the multimodal annotation of interactional data.

Elan and NOVA

How are multimodal texts formed?

Semiotic modes co-operate in contexts

What is the "division of communicative labour" in multimodal texts?

In multimodal texts, every mode has a specific function to fulfill

Different modes are assigned different communicative tasks in order to optimally achieve the text's aim in context

Each mode contributes what it does best

What are affordances and constraints of semiotic modes and what do they do?

Affordances are what a mode can do well or makes possible (its strengths and capabilities)

Constraints are what a mode cannot do well or limits (its restrictions)

These affordances and constraints determine what each mode is useful for in a text

What does it mean that mode-combination is "strategic and patterned"?

The integration of the semiotic modes can be described and analysed

How do modes create cohesive and coherent wholes?

Modes interlock on various levels to form unified texts - the integration happens at multiple levels simultaneously, and it takes different analytical approaches to fully describe what is happening in this integration process - the modes work together to create both cohesion (structural unity) and coherence (meaningful unity)

What are the three ways that language can be mediated by technology according to the lecture?

Mediating ourselves to do new things (e.g., glasses, hearing aids, bio-hacking)

Mediating the environment to support different things (e.g., social media, presentations, film/TV)

Mediating the code to do different things (e.g., manipulating, recording, editing sound, image, movement)

How do typography and sound differ as second-order semiotic systems?

Typography is inherently attached to written language (you don't get typography without typeface)

Sound is not inherently connected to speech

Technology has changed what's possible with typography and who can access it, while sound has been mediated to shift from ephemeral (vergänglich) and situated to stable and digital

Why has typography moved from the periphery to the centre of the 'semiosphere'? (2)

Technology has enhanced typographic options and made them available to everyone, not just specialists like type-setters

Our sensitivity to the social meanings of typographic design has increased with exposure

What does the text-graphic design of a piece of communication include? (4)

Font design/type-faces used

Copy-setting on the page

Layout of page and document

Choice of material and printing technique

Why is typography described as a "secondary system of meaning-making"?

Because it clings to a primary one (written language)

We don't pay much attention to type as we focus on the meaning of words, but typography cannot be avoided - just as you cannot speak without intonation, voice, and stress

According to van Leeuwen (2005), what are the two levels of meaning in printed words?

The 'word image' - the idea represented by the word itself, constructed from letters

The 'typographic image' - the holistic visual impression created by the typography

What are the four kinds of typography?

Microtypography

Mesotypography

Macrotypography

Paratypography

Describe Microtypography and what four aspects does it include?

Design of fonts and individual letter

Type face, type size, type stile and colour

Describe Mesotypography and what three aspects does it include?

Configuration of signs in lines and blocks

Line spacing, type composition and amount of print

Describe Macrotypography and what four aspects does it include?

Graphic structure of overall document

Paragraphing, emphasis, ornamentation and text-image

Describe Paratypography and what two aspects does it include?

Materials and techniques for producing graphic signs

Surface material and signing technique

Name four levels or aspects on which typography can be designed and adjusted to fit situation and communicative task.

Individual letters and lines of print must be designed using fonts

Letters must be arranged on the page

Pages from documents

All typographic decisions depend on printability

Is there a Grammar of Typography? Explain the six elements ideational, interpersonal, textual, semantics, pragmatics and syntax in this context.

Ideational: typography supports meaning expressed in language

Interpersonal: typography socially addresses the reader

Textual: typography gives texture and shape to the documents

Semantics: typography makes connotative and pictorial meanings available

Pragmatics: typography aims at aptness and decorum

Syntax: typographic elements configure to form conventional patterns

What can typography be compared to in terms of these aspects?

A sign system (semiotic mode)

Name three criteria that make typography and semiotic modes comparable.

Typographic signs can make meaning

They both have pragmatic rules

Typical graphic designs for a certain text type

In which three major ways is typographic meaning made?

Through connotation and metaphor

Through pictoriality

Through spatial division and emphasis

What are the five functions of Typography?

Optical division of space (lends the text visual structure and semantic order)

Pictoriality (converts wiriting into picture)

Semantic emphasis (emphazises content and speech act functions of text)

Connotative indexing (conveys additional meanings)

Inter-semiotic harmony (adapts to all other modes in form and function)

In which three components can sound be devided?

Music

Noise

Speech Sound

Name two reasons why the communicative potential of sound is limited.

Music can't depict or represent

Noise can only indexically point to places, actions and states

Sound as a semiotic resource can be divided into three aspects, name them and give one example.

Levels of design - tune

Choices/distinctions - rising/descending pitch

semiotic value - activating or soothing

How can music become functionally communicative? (5)

Musical genre (genres allude to times and carry prestige)

Knowledge of music (knowledge helps to contextualise and give it meaning)

General character (effects emotions)

Iconic/indexical (imitate environmental phenomena)

Foreground one aspect (one feature is foregrounded)

Name the seven Submodes of music.

Tune, Harmony, Tempo, Dynamics, Timbre, Sonic interaction and perspective

Name three reasons why music is a semiotic resource.

Tunes can ascend and descend

Volume can increase and decrease

Sounds may come from near or far

How are music and noise both utilized as semiotic resources?

Both music and noise function as semiotic resources that can create meaning in multimodal texts

In addition noise can serve functions including cohesion in multimodal compositions

Name and shortly explain the two general semiotic principles according to which music can be made sense of.

Provenance

Music carries connotations, which may be imported into new contexts

Experimental meaning potential

Some quality of the music metaphorically stands for some abstract quality

There are three principal ways in which noises can become meaningful in audio texts, name them.

Indexical (noises make meaning by communicating)

Metaphorical (noises generate sense)

Intertextual (noises make meaning by becoming stereotypical)

What creates new opportunities for creating multimodal texts?

New media