Challenges of Eye Tracking Research

What are the 3 categories that challenges in the eye movement research field can be split into?

• Theoretical

• Conceptual

• Methodological

Theoretical Challenges of Eye Tracking

The idea of ecological validity from eye tracking research

• There are arguments that lab-based eye tracking studies lack ecological validity, and therefore should be ‘ditched’ for naturalistic studies

• However, Hessell et al. (2020) argue that some research seems to have no basis for being conducted in the real world rather than the lab for any reason other than ecological validity

  • There is too much emphasis on the WHERE and not enough emphasis on the WHY

• At the core of this, what we need is strong theoretical underpinning or models for WHY eye movements are relevant in a particular setting

Think: It doesn’t matter whether you track the eye movements of someone with a particular disorder in a lab based setting using computer stimuli and a stationary eye tracking device or out in the world with a mobile eye tracker if you don’t have a theory or reason for why they might exhibit differences in the first place.

“what might these eye movements actually tell us?'“

Conceptual Challenges in Eye Tracking

Not all researchers define what they mean with the terms or concepts they use

• There is a lack of consistency across the meanings or references of terminology across the literature which makes comparisons difficult

  • E.g. where some researchers may refer to ‘fixations’ as the technical term for a specific eye movement, others use the term to describe more general gaze events

  • Therefore, two authors may be using the same term but talking about completely different things, therefore comparisons are limited and/or conceptually inaccurate

• Same with the concept of attention

  • In eye movement studies, attention is kind of conceptualised and operationalised as an over-allocation of visual attention such as gazes and fixations (they may not even be thinking about it)

  • But attention is more complex than just eye movements and multifaceted - are we really conceptually capturing it?

• It is also important to consider that when we are comparing between lab and naturalistic settings, visual behaviour and eye movements are conceptually different

  • Eye movements in labs are typically made up of fixations, saccades and blinks

  • In real-life settings, it is those PLUS smooth pursuits, stabilising eye movements and vestibulo-ocular reflexes

Takeaway: Across studies and settings, are we really comparing like-with-like?

Methodological Challenges in Eye Tracking Research

• Lack of control

  • People behave differently, and you can rarely replicate the exact same environment in a naturalistic setting

• Volume of data (assessing frame by frame)

• Manual coding

  • Threatens inter-rater reliability and introduces potential rater bias if coding protocols are not robust

  • Independent evaluators and inter-rater reliability tests help with this

• Calibration techniques

  • I.e. going from darkness to bright light makes your pupil shrink dramatically and the calibration is lost

• Sensitivity to movement/conditions

  • Think: god forbid someone sneezes

• observer effects

  • Do people behave naturally when they know they are being watched?

  • Studies were discussed in live lecture! (acclimatisation is good)

Takeaway: These methodological challenges don’t automatically make findings redundant or unusable, but they must always be considered in interpretation of findings

Describe how theoretical and methodological challenges could combine in context

Methodological:

• Data volume

• Manual coding/rater reliability

Consideration: Is it even feasible to conduct the research on the scale it is needed because each rater will have to analyse SO much data, frame by frame, and if the Px sneezes or moves in a way that will disrupt calibration, the data is lost.

Theoretical

• Studying eye movements without a specific research question, theory or model in mind to be tested

Consideration: In conjunction with aforementioned methodological challenges, researchers will be sifting through several hours of data that could completely meaningless, making it even more time consuming,

What is the only way to get valid, reproducible results from eye tracking data?

Record high quality data, which involves the use of robust calibration techniques

• Maximises other methodological limitation of experimental control

What considerations are required to get high quality data to be valid and reproducible?

Need good quality control for:

• Participants

• Eye tracking technology

• Experimental set-up

• Experimenter/data collector

• Instructions

• Physical environment characteristics

Takeaway: We want to limit the influence of extraneous variables, which can present from the factors above

Calibration Techniques

Participants are to look at a number of predefined positions/targets in a stimulus space

• At each target, the eye tracker detects the characteristics of the eye movement/position and associates that eye image with the position in the space.

Holmqvist’s Challenges to Calibration

Identified 6 challenges that can negatively influence calibration of eye trackers

• Downward facing lashes

• Pupil size

• Mascara

• Contact lenses

• Glasses

• Dominance of eye being tracked

Hessels et al.’s Challenges to Callibration

• Sensitivity to movement

  • Calibration may be compromised if people move, smile, nod walk or touch the eye tracker

• Conditions

  • Changes in lighting conditions can change the pupil, resulting in calibration loss

  • OR the light reflecting off other parts of one’s eyes can be mistaken for a corneal reflection, or something else the tracker is trying to measure

How can you address methodological challenges?

The main challenge is volume of data and manual coding. You should:

• Be specific and have strong RQs in mind (think: fix the theoretical challenges before having to sift aimlessly through meaningless data)

  • Know what it is you are looking for because it will narrow down the relevant data

• Code what is necessary

• Teams of coders

• Blind double coding

• Use markers in the stimulus setting (i.e. code areas that the tracker knows is ‘speaker 1’ etc)

  • Harder to use in complex, dynamic environments so really only good for mostly still scenes

• Use robust calibration techniques

What are some problem solving techniques you can use to prevent calibration issues?

• Curling lashes upwards so they don’t obstruct camera over the eye

• Manually adjust eye tracking parameters to match the size of individuals’ pupils

• Remove contacts and mascara

What is an issue derived from almost a combination of the theoretical, methodological and conceptual challenges being faced by the eye tracking research area?

There is a growing notion that eye trackers are easy to use, cheap and ‘plug and play’, but this idea may undermine the reliability and meaning of eye tracking research, with devalues the work

• May lead to the proliferation of research that is not grounded in theory or lacks the scientific rigour of previous experiments

• It is important that there is openness on methodologies, coding rationales and theories that are being used within studies, so robust comparisons and conclusions can be made

Describe some future directions for eye tracking research

• The development of gaze coding systems to reduce the burden of manual coding

  • But the tech is not there yet!

  • I.e. the program knows what they are looking at, the fixation time, and the type of eye movement

• Incorporation of eye tracking into VR

  • Would improve experimental control in terms of controlling the environment, including manipulating it

• Visual behaviour and eye movements as biomarkers

  • Already happening, but more preliminary and research based rather than being used in diagnostics

• Applications in:

  • Ageing and associated risks and potential health problems

  • Visual attention and changes in communication

  • Well-being, stress and mental health