week 3 & 4

Eye Movements: Hidden Insights into Cognition, Social Interaction, and Clinical Disorders

I. Introduction to Eye Movements and Vision for Action

A. Fundamental Concepts

• Eye movements are a crucial window into human cognition, revealing how individuals acquire and process visual information.

• The fovea, a small central area in the retina, provides sharp central vision, covering only 1-2 degrees of our visual field. Because our clear vision is so limited, we constantly move our eyes.

• We move our eyes approximately three times per second.

• Fixations occur when the eye stops to focus on an object, while saccades are the rapid movements between these fixations. We are effectively blind during saccades.

• Vision is an active process; we actively move our eyes to seek out information based on our goals and tasks, rather than passively receiving stimuli. This is demonstrated by studies showing eye movements change based on the task, confirming that what we look at is shaped by our intention.

B. Eye Movements in Everyday Activities (Vision for Action)

• Research on eye movements in ordinary activities like food preparation (tea-making, sandwich-making) shows a tight spatio-temporal coupling between vision and action.

• The eyes typically lead manipulative action, reaching the next object in a sequence before any hand movement begins. In tea-making, the average lead time was 0.56 seconds, while in sandwich-making it was shorter at 0.09 seconds.

• Gaze is almost exclusively allocated to task-relevant objects; task-irrelevant objects are rarely viewed (under 5% in studies, except during waiting periods). This indicates that eye movement control is primarily 'top-down,' driven by the task script rather than the intrinsic salience of objects.

• The way the human visual system is constructed leads to very similar oculomotor techniques across competent subjects interacting with objects, despite some variations in action order.

• Information is primarily obtained from the scene "as it is needed" (just-in-time processing), rather than building up a detailed internal model of the surroundings.

C. Functions of Fixations during Object-Related Actions (ORAs)

• Object-Related Actions (ORAs) comprise all acts performed on a particular object without interruption.

• Fixations in everyday tasks can be classified into four primary functions:

  1. Locating: Establishing the positions of objects for future use, often occurring during an initial "surveying" period before manipulation begins.

  2. Directing: Establishing the target direction for the hand prior to contact or when putting an object down. The eye typically moves away just before the hand reaches, suggesting the grasp often occurs without continuous visual feedback.

  3. Guiding: Supervising the relative movements of two or more objects during manipulation, such as a kettle and its lid or knife, bread, and peanut butter. This often involves multiple, alternating fixations and continuous visual control.

  4. Checking: Establishing whether a particular condition is met before an action terminates (e.g., kettle full, water boiling, lid off). The eye dwells on the relevant region to monitor the condition.

• Vision is used economically; it disengages as soon as another sense (like touch or proprioception) can take over. For instance, hands are rarely fixated, objects are not fixated once acquired by the hands, and for pouring, the destination vessel is fixated, not the source.

• The oculomotor system needs information about the identity, location, and required monitoring of the next object to perform an ORA. Object localization can be a two-stage process, using place memory to get close and then visually guided saccades for precise fixation.

D. Eye Movements in Expertise and Driving

• Anticipation is a key marker of expertise in tasks like driving and sport.

• Driving:

  • Experienced drivers look further into bends, anticipating the curve, unlike novices who keep gaze strictly in line with the car's heading.

  • Drivers rely on a "tangent point" on the inside of a curve, fixating it 1-2 seconds before the bend.

  • Familiarity with routes can lead to a reduction in attention to details and an increase in off-road dwell time.

II. Cognitive Ethology: Bridging the Lab and the Real World

A. Critique of Traditional Lab-Based Research

• Traditional lab research aimed to create universally valid theories by simplifying the environment and maximizing experimental control to find causal relationships.

• However, it became clear that most lab findings were only true under specific laboratory conditions and became unpredictable outside them. Cognitive processes are variable and context-dependent.

• The field's responses have often been "pathological": either denying the problem or acknowledging it but continuing to conduct research based on the false assumption of invariance.

• Lab assumptions of invariance (processes are regular across situations) and control (situational variability can be reduced) lead scientists away from complex real-life situations to simple, artificial paradigms. This eliminates the need to confirm if lab processes occur in the real world.

• Lab-based effects are often fragile and not reproducible outside strict lab confines, or even with minor changes within the lab.

• General Systems Theory suggests that tight experimental control is ineffective for complex, non-linear systems like human cognition, where stable characteristics only emerge when multiple variables vary together, which labs prohibit.

• Claims about the real-world relevance of lab findings are often unscientific because they cannot be falsified; it's difficult to prove real-world data manifests the same process as in the lab due to uncontrolled variables.

B. Principles and Approach of Cognitive Ethology

• Cognitive Ethology (CE) is a novel research framework that advocates for first directly studying how people behave in their natural real-world environments before moving into the lab.

• It rejects the standard assumptions of invariance and control, assuming instead that cognitive processes are contextualized, and that variance itself can reveal key characteristics of cognitive processing.

• The initial job of the researcher is to systematically observe, describe, and measure what people do and experience in their natural environment to define the domain of inquiry.

• It proposes that the conceptual language for human cognition should initially be grounded in everyday "folk-psychological" language.

• CE views cognition as a distributed system, where important aspects emerge when embodied individuals interact with their natural environment and other people.

C. Integration with Laboratory Investigations

• CE and lab-based studies are complementary, not competing. CE guides lab research by providing rigorous descriptions of real-world behavior, against which lab findings can be validated.

• If lab behavior deviates from real-life observations, CE provides a self-correcting mechanism, indicating that the laboratory environment fails to capture what people truly do.

D. Personal and Subpersonal Levels of Explanation

• Personal Level Explanation: Focuses on the person as a whole organism interacting with their environment. It explains cognition in terms of overt behavior, interactions with others and the environment, and people's subjective experiences, goals, and beliefs.

• Subpersonal Level Explanation: Focuses on the internal organization and processes of the brain. Lab studies typically operate at this level, assuming invariant cognitive processes and neural activations.

• Shortcomings of Subpersonal Alone: Subpersonal explanations often fail to explain why cognitive performance is as it is, how it relates to other systems, or that cognition is distributed among multiple individuals and the environment.

• Complementary Nature: Personal-level explanations complement and "ground" subpersonal ones, providing the "why". For example:

  • Visual Search Asymmetry: The difficulty in finding a black-topped target against white-topped distractors (vs. vice versa) is explained at the personal level by our bias to expect overhead lighting in the real world.

  • New Object Attention: The strong attention-capturing effect of new objects is explained at the personal level by their fundamental importance for survival (e.g., identifying predator/prey) compared to mere feature changes.

  • Collaborative Attention (Eastern Airlines Flight 401 Crash): The crash due to pilots' distributed attention on a malfunctioning light, neglecting flight control, highlights that failures in collaborative attention cannot be explained solely by individual subpersonal mechanisms; a personal-level account is needed.

• Cognitive processes cannot be fully understood at the subpersonal level unless grounded in a personal-level understanding of overt cognitive behavior, experiences, beliefs, and intentions in everyday environments.

E. Role of Subjective Experience

• CE emphasizes the importance of observing and describing subjective experiences (first-person reports) in addition to objective behavior. Subjective experience is central to cognitive performance in complex natural settings.

• While historical criticisms questioned the reliability and validity of subjective reports, many of these criticisms also apply to objective measures (e.g., studies changing the factor being measured, experimenter demands).

• Utility of Subjective Reports: They provide direct access to participants' explicit goals, intentions, and behaviors in everyday situations, offer insights into individual differences, reveal cognitive strategies, and can generate experimental hypotheses (e.g., Reason's studies on everyday attention slips).

• Subjective and objective measures can be integrated complementarily, mutually constraining and supporting understanding of cognition.

III. Eye Movements in Clinical Disorders

Eye movement patterns and gaze allocation differ significantly in various clinical populations, providing valuable insights into cognitive and social processing abnormalities.

A. Schizophrenia

• Smooth pursuit eye movements (SPEM) are often abnormal in schizophrenia, characterized by:

  • Increased frequency of saccades during pursuit.

  • Increased anti-saccade error rates and latencies.

  • Increased frequency of catch-up saccades (repositioning gaze to the target) and intrusive saccades (moving away and then back).

  • Low "gain" scores, indicating difficulty matching gaze velocity to target velocity.

• These intrusive saccades are found in non-psychotic children of schizophrenic patients, suggesting eye tracking can serve as a marker of susceptibility for developing schizophrenia.

• In natural tasks, patients with schizophrenia fixated significantly shorter and more often on predefined stationary targets than controls, but the opposite was true during free gaze (longer and less frequent fixations). However, they did not show elevated looks to task-irrelevant locations in scene viewing.

B. Attention Deficit Hyperactivity Disorder (ADHD)

• Individuals with ADHD may have difficulties in oculomotor tasks requiring the suppression of reflexive saccades.

• In pro-saccade tasks (looking towards a target), ADHD participants showed longer reaction times, greater intra-subject variance, reduced peak velocities, and increased saccade durations. They also had increased anti-saccade error rates and latencies.

C. Williams Syndrome (WS)

• Characterized by "hyper-sociability" or "pro-social compulsion" and a drive toward social behavior.

• Individuals with WS exhibit a propensity to hold prolonged face gaze during interactions and interview conditions.

• In tasks involving embedded or scrambled faces, once the face was located, WS participants fixated upon it for significantly longer durations than typically developing individuals.

• This prolonged gaze is likely related to attention disengagement difficulties rather than enhanced face-finding abilities.

D. Autism Spectrum Disorder (ASD)

• Individuals with ASD often show reduced attention to social aspects of the environment.

• In tasks with hidden or scrambled faces, participants with autism took significantly longer to locate the face and spent significantly less time fixating on it once detected compared to typically developing individuals.

• There's a negative correlation between face gaze length and the severity of autism (CARS score), meaning higher-functioning individuals show increased face gaze.

• Atypical attention mechanisms may guide behavior, potentially reflecting disinterest or inattention for individuals with autism.

• Subtle abnormalities in social gaze for adults with ASD, such as being less inclined to look at an object a person in a scene is looking at during free-viewing, can lead to serious real-life difficulties. Infant viewing of social scenes is also atypical and under genetic control.

E. Body Dysmorphic Disorder (BDD)

• Characterized by preoccupation with an imagined defect in appearance and behaviors like excessive mirror gazing or avoidance, and self-comparison.

• Individuals with BDD spend more time looking at their own "unattractive" features.

• They also allocate more visual attention to the attractive features of others, perpetuating upward social comparisons.

F. Eating Disorders (ED)

• Associated with an over-evaluation of shape and weight, leading to behaviors like repeated body checking and comparison to others.

• In Bulimia Nervosa (BN), individuals spend significantly more time fixating on images of individuals with low BMI and less time on those with high BMI.

• ED participants generally spend significantly more time looking at unclothed body parts.

• Patients with Anorexia Nervosa (AN) showed more attentional disengagement to food pictures compared to controls.

G. Alcohol-Related Disorders

• These disorders are linked to an increased salience of addiction-related cues, which capture and hold attentional resources, increasing craving and fostering alcohol use.

• Key eye-tracking findings include:

  • Increased dwell times for alcohol-related stimuli.

  • Reduced inhibitory control on saccadic movements to appetitive (alcohol-related and non-alcohol-related) stimuli presented peripherally.

  • Increased pupillary reactivity to visual stimuli, regardless of their emotional content.

  • Limited visual attention to prevention messages presented on alcohol packaging.

• The attentional bias is stable in heavy drinkers but dependent on high alcohol expectancies in light drinkers. This bias is not always influenced by internal factors like consumption intention or craving.

• Acute alcohol intoxication can linearly decrease this attentional bias in heavy drinkers.

• The bias related to more controlled attentional processes disappears when ambivalence towards alcohol emerges in hazardous drinkers.

IV. Social Gaze and Interaction

A. General Principles

• The eyes are a remarkably useful source of information during social interaction.

• While laboratory studies often show a strong preference for looking at the eyes of others (70-80% of fixations) when faces are in isolation, this preference is diluted when the full body and context are visible.

• Gaze serves a dual function: acquiring visual information (e.g., from a conversational partner's face) and signaling information to others (e.g., indicating where attention is directed).

B. Gaze Patterns in Conversation

• Mutual gaze during conversation correlates with the combined agreeableness and familiarity of participants. High amounts of gaze from one participant may cause the partner to avoid returning gaze.

• Gaze plays a significant role in turn-taking and signaling during interactions:

  • Starting a turn: Talking events are often preceded by averted gaze (looking away from the other person).

  • Ending a turn: Speakers tend to end their turn with direct gaze at their partner, signaling a turn transition.

  • Monitoring: Looking back at the partner can serve a monitoring function to check for understanding.

• Individuals show a degree of stability and idiosyncratic preference in their gaze allocation during conversations, some preferring eye gaze, others mouth gaze, and others distributing to both.

• Gaze shifts can guide another person's attention, and gaze cues are rapidly used during cooperative behaviors.

• In multi-person interactions, a speaker's gaze may be distributed across partners and then directed to one specific partner at the end of a turn to offer the floor.

C. Joint Perception and Gaze Coordination

• In a joint perception scenario where participants know they are looking at the same pictures as a partner, an initial preference for negative stimuli gives way to a preference for positive stimuli, unlike when looking alone. This suggests social context influences emotional alignment.

• During live spontaneous dialogue, participants' eye movements are coupled, meaning conversers are most likely to be looking at the same thing at the same time ("common ground").

• Social referencing occurs when a person looks to a specific group member (e.g., a minority individual seen as an "expert" on prejudice) to assess a situation, especially when that individual can hear a potentially offensive remark and make an informative reaction. Eye movement studies show viewers indeed look to the bystander who may be offended.

V. Challenges and Future Directions in Eye Tracking

A. Methodological Challenges

• Control (or lack of): Conducting research in natural settings inherently involves less control than labs.

• Volume of data and manual coding: Mobile eye-tracking generates huge volumes of data, making manual coding time-consuming and labor-intensive, requiring strong research questions and potentially teams of coders.

• Calibration techniques: Ensuring high-quality, reproducible data requires robust calibration, which can be affected by factors like lighting, participant movement, and physical characteristics (e.g., eyelashes, glasses).

• Naturalness of behavior: The awareness of being tracked might influence natural behavior.

• Conceptual clarity: Terminology (e.g., fixations, gaze events) may not map perfectly from static lab studies to real-world contexts, where additional eye movements like smooth pursuit and vestibulo-ocular reflex (VOR) components are present.

B. Addressing Challenges and Future Directions

• Specific research questions: Having strong theoretical models and specific research questions helps manage the volume of data and ensures meaningful analysis.

• Technological advancements: Development of automatic gaze coding systems, use of markers and masks to identify areas of interest, and the increasing role of Virtual Reality (VR) environments for controlled naturalistic studies.

• Biomarkers: Eye movements and visual behaviors have potential as early, non-invasive predictors or biomarkers for certain clinical disorders.

• Interdisciplinary approach: A dynamic system approach to interaction, investigating gaze in actual interactive contexts and as part of multimodal communication, considering time-dependency and various timescales (microgenetic, ontogenetic, phylogenetic).