Visual Imagery Notes

Visual Imagery

• Visual imagery is the ability to "see" an object or scene in the absence of a visual stimulus. For example, recalling the panoramic view from the Eiffel Tower.
• Mental time travel, a characteristic of episodic memory, often involves visual imagery.

Imagery in the History of Psychology

• The history of imagery can be traced back to Wilhelm Wundt's psychology laboratory (1879).

Early Ideas About Imagery

• Wundt proposed that images were one of the three basic elements of consciousness, along with sensations and feelings.
• He suggested studying images as a way to study thinking, due to the link between them.
• This gave rise to the imageless thought debate: whether thought is impossible without an image.
• Francis Galton (1883) observed that people with difficulty forming visual images (aphantasia) were still capable of thinking and problem-solving, arguing against the necessity of imagery for thought.
• The debate ended when behaviourism overshadowed imagery in psychology.
• John Watson, the founder of behaviourism, considered images as "unproven" and "mythological" and not worthy of study (1928).
• Behaviourism dominated from the 1920s to the 1950s, pushing the study of imagery out of mainstream psychology.

Imagery and the Cognitive Revolution

• The cognitive revolution in the 1950s and 1960s revived the study of cognition.
• Cognitive psychologists developed methods to measure behaviour and infer cognitive processes.
• Alan Paivio's (1963) work on memory showed that concrete nouns (e.g., truck, tree) are easier to remember than abstract nouns (e.g., truth, justice) because they can be easily imagined.
• This relates to the picture superiority effect.
• Shepard and Metzler (1971) used mental chronometry to infer cognitive processes by measuring the time needed to carry out cognitive tasks.
• In their mental rotation experiment, participants determined if two pictures were of the same object, with varying angles between the views.
• The time to decide was directly related to the angle difference, suggesting participants mentally rotated one view to match the other.
• This experiment provided evidence for the visuospatial sketchpad in Baddeley's working memory model (Baddeley & Hitch, 1974; Baddeley, 2000a).
• It was one of the first experiments to apply quantitative methods to the study of imagery and suggested that imagery and perception may share the same mechanisms.

Imagery and Perception: Do They Share the Same Mechanisms?

• Shepard and Metzler's (1971) experiment indicated a spatial correspondence between imagery and perception.
• Stephen Kosslyn's mental scanning experiments further explored this idea.

Kosslyn's Early Mental Scanning Experiments

• Kosslyn (1973) asked participants to memorize a picture (e.g., a boat) and then create an image of it in their mind, focusing on one part.
• Participants then looked for another part of the object and pressed a button when they found it.
• Kosslyn found that participants took longer to scan between parts of the object that were farther apart, supporting the idea that visual imagery is spatial.
• In another experiment, participants memorized an island with seven locations and scanned between all possible pairs of locations.
• The reaction time increased with the distance between locations, reinforcing the idea that visual imagery is spatial in nature.
• Pylyshyn (1973) proposed an alternative explanation, leading to the imagery debate: whether imagery is based on spatial or propositional mechanisms.

The Imagery Debate: Is Imagery Spatial or Propositional?

• Kosslyn interpreted his research as supporting a spatial representation for imagery.
• Pylyshyn (1973) argued that the spatial experience of imagery is an epiphenomenon and not part of the underlying mechanism.
• Pylyshyn proposed that the mechanism underlying imagery is propositional, using abstract symbols to represent relationships.
• Spatial representation involves a spatial layout, while propositional representation uses abstract symbols (e.g., "The cat is under the table.").
• Depictive representations are like realistic pictures.
• Pylyshyn argued that scanning time increases with distance because participants base responses on their knowledge of real-world scenes.
• Pylyshyn (2003) suggested that people simulate what it would look like to see something when asked to imagine it.
• This is called the tacit knowledge explanation, where participants unconsciously use their knowledge about the world.
• Finke and Pinker (1982) countered this by using unfamiliar stimuli presented briefly.
• Participants saw a four-dot display followed by an arrow and indicated whether the arrow pointed to any of the dots.
• The results showed that participants took longer to respond when the arrow was farther from the dots, even with unfamiliar stimuli.
• Finke and Pinker argued that their experiment supported the idea that imagery is served by a spatial mechanism shared with perception.

Behavioural Experiments: Comparing Imagery and Perception

• Kosslyn (1978) examined how the size of an object in a person's visual field affects imagery.

Size in the Visual Field

• Objects appear smaller and details are harder to see when far away.
• As you move closer, objects fill more of your visual field, and details become clearer.
• Kosslyn investigated whether this relationship also holds for mental images.
• Participants were asked to imagine two animals, such as an elephant and a rabbit, and to move closer to the image until it filled their visual field.
• They were then asked if the animal had specific features (e.g., whiskers on the rabbit, spots on the elephant).
• Participants answered more quickly when they imagined the larger animal (elephant) because it filled more of their visual field.
• Kosslyn concluded that images are spatial, like perception.
• Kosslyn (1978) also used a mental walk task, where participants imagined walking toward a mental image of an animal and estimated how far away they were when the image overflowed their visual field.
• Participants had to move closer to small animals (e.g., a mouse) than to larger animals (e.g., an elephant) before overflow occurred, as they would with real animals.
• This further supported the idea that images are spatial, like perception.

Luminance, Contrast, Motion, and Orientation

• People respond differently to bright vs. faint stimuli or fast vs. slow-moving objects.
• Bright stimuli are detected more rapidly than faint stimuli, and fast-moving objects are detected more rapidly than slow-moving objects.
• Broggin, Savazzi, and Marzi (2012) measured the effect of variations in visual characteristics on both perception and imagery using a reaction time experiment.
• They reasoned that if perception and imagery rely on overlapping representations, reaction times should be affected similarly for real and imagined stimuli.
• They investigated the RT effects of luminance, contrast, spatial frequency, speed of motion, and orientation.
• In the imagery task, participants visualized stimuli and responded to the mental image as quickly as possible after a warning tone.
• Then, participants performed a similar task with real visual stimuli (perceptual task).
• Except for spatial frequency, RT patterns for perception and imagery looked very similar.
• RTs were slower for stimuli with low vs. high luminance, low vs. high contrast, low vs. high motion speed, and oblique vs. vertical orientation, regardless of whether they were real or imagined.
• RTs were generally slower for real stimuli.
• Broggin et al. (2012) suggested that there is an overlap between the structural representations of perception and imagery.

Interactions of Imagery and Perception

• Demonstrating that imagery affects perception, or vice versa, indicates that they both have access to the same mechanisms.
• Perky's (1910) experiment involved participants projecting visual images of objects onto a screen while a dim image of the same object was projected unbeknownst to them.
• Participants' descriptions of their images matched the projected images, even though they didn't report seeing them.
• For example, when asked to create an image of a banana, participants described their image as being oriented vertically, matching the projected image.
• Participants mistook the actual picture for their own mental image.
• Farah (1985) instructed participants to imagine either the letter H or the letter T on a screen.
• After forming a clear image, they pressed a button that caused two squares to flash, one containing a target letter (H or T).
• Participants were more accurate at detecting the target letter when it matched the letter they were imagining.
• For example, they were more accurate at detecting the letter H when they had been imagining the letter H.
• This demonstrated an interaction between imagery and perception.

How Can the Imagery Debate Be Resolved?

• Anderson (1978) warned that we can't rule out the propositional explanation, and Farah (1988) pointed out that it's difficult to rule out Pylyshyn's tacit knowledge explanation based on behavioural experiments alone.
• Participants might unknowingly simulate perceptual responses in imagery experiments based on prior perceptual experiences.
• For example, in mental walk experiments, participants could use their knowledge of animal sizes to conclude that they would need to be closer to a mouse than an elephant before the animal filled their field of view.
• Investigating how the brain responds to visual imagery could provide a way out of this problem.
• By the 1980s, evidence about the physiology of imagery was becoming available from unit recordings and neuropsychology, and later from brain imaging research.

Imagery and the Brain

• Physiological experiments, including brain imaging techniques like MRI, have provided evidence supporting a close connection between imagery and perception, although the overlap is not perfect (Dijkstra, Bosch, & van Gerwen, 2019).
• A perfect overlap is not expected, as it would be difficult to distinguish between actual perception and, for example, a dream or hallucination.
• Perception relies on both bottom-up (sensory input) and top-down processes (stored information), while imagery lacks bottom-up input from the eyes.
• The lack of visual input in imagery must impact the extent and temporal order in which some brain areas are activated.

Imagery Neurons in the Brain

• Kreiman, Koch, and Fried (2000) recorded activity from single neurons in epilepsy patients who had electrodes implanted in their medial temporal lobe.
• They found neurons that responded to some objects but not others.
• For example, a neuron might respond to a picture of a baseball but not a face.
• Importantly, the same neuron also fired when the person closed their eyes and imagined a baseball (vigorous firing) or a face (no firing).
• Kreiman et al. (2000) called these neurons imagery neurons.

MRI Research

• Le Bihan et al. (1993) demonstrated that both perception and imagery activated the visual cortex using MRI.
• Activity in the primary visual cortex increased both when a person observed visual stimuli and when they were imagining the stimulus.
• Goldenberg et al. (1989) found that the visual cortex was more activated when participants answered questions that required imagery (e.g., "Is the green of the trees darker than the green of the grass?") compared to questions that did not (e.g., "Is the intensity of electrical current measured in amperes?").
• Kosslyn (1995) focused on the topographic organization of the visual cortex to specify the role of the visual cortex in imagery.
• The topographic organization of the visual cortex refers to the fact that specific locations on a visual stimulus cause activity at specific locations in the visual cortex, and that points next to each other on the stimulus cause activity at locations next to each other on the cortex.
• Looking at a small object causes activity in the very back of the visual cortex, while looking at larger objects causes activity to spread toward the front of the visual cortex.
• Kosslyn instructed participants to create small, medium, and large visual images while in a brain scanner.
• The result was that when participants created small visual images, activity was centered near the back of the brain, but as the size of the mental image increased, activation moved toward the front of the visual cortex, similar to what was found for perception.
• Kosslyn concluded that both imagery and perception result in topographically organized brain activation.
• However, some researchers were not convinced by Kosslyn's results, while others failed to replicate them (e.g., Roland & Gulyas, 1994; Mellet et al., 1996).
• Kosslyn and Thompson (2003) performed a meta-analysis of over 50 brain imaging studies to explain the inconsistent results.
• They identified three critical methodological factors for finding evidence of activation of topographically organized visual areas as a result of visual imagery:
  1. The task has to require high-resolution images of the details of the shape.
  2. The task should not require spatial images.
  3. A sensitive MRI technique needs to be used (Kosslyn & Thompson, 2003).
• The early visual cortex can be activated reliably by visual imagery, but whether this happens or not depends on the exact task circumstances.
• Ganis, Thompson, and Kosslyn (2004) used MRI to measure and compare activation during perception and imagery.
• For the perception condition, participants observed a faint drawing of an object they had studied before.
• For the imagery condition, participants were told to imagine a picture they had studied before when they heard a tone.
• Both conditions had participants answer a question such as "Is the object wider than it is tall?"
• Experiments have demonstrated an overlap between areas activated by perception and by imagery but have also found differences.

Transcranial Magnetic Stimulation

• Showing that an area of the brain is activated by imagery does not prove that this activity causes imagery.
• Brain activity in response to imagery may be an epiphenomenon.
• Kosslyn et al. (1999) used transcranial magnetic stimulation (TMS) to temporarily stimulate or inhibit certain brain regions.
• They found that applying TMS to the visual cortex slowed reaction time to imagery tasks, suggesting the visual cortex is causally involved in imagery.
• This provided more direct evidence for the link between brain activity and imagery.
• For example, TMS was applied to the occipital lobe during visual imagery tasks, and it was observed that it impaired performance compared to a control condition, supporting that the visual cortex causally involved in imagery.

Neuropsychological Case Studies

• Studies of people with brain damage can help us understand imagery in two ways:
  1. Determine how brain damage affects imagery.
  2. Examine whether brain damage affects imagery and perception in the same manner.

Removing Part of the Visual Cortex Decreases Image Size

• Patient M.G.S. had part of her right occipital lobe removed to treat epilepsy.
• Before the operation, she performed a mental walk task and felt she was about 15 feet (approximately 4.5 metres) from an imaginary horse before its image overflowed.
• After the operation, the distance increased to 35 feet (approximately 10.5 metres).
• This occurred because removing part of the visual cortex reduced the size of her field of view.
• This result supported the idea that the visual cortex is important for imagery.

Perceptual Problems Are Accompanied by Problems with Imagery

• People who have lost the ability to see colour due to brain damage also report being unable to create colours through imagery (DeRenzi & Spinner, 1967; DeVreese, 1991).
• Patients with unilateral neglect (damage to the parietal lobes) ignore objects in one half of their visual field.
• Bisiach and Luzzatti (1978) tested the imagery of a patient with unilateral neglect by asking him to describe things he saw when imagining himself standing at one end of the Piazza del Duomo in Milan.
• Without being prompted, the patient only described the buildings on the right side of the piazza (his intact visual field) and omitted the buildings on the left side.
• When asked to imagine standing at the opposite end of the piazza, the patient described the buildings previously omitted and omitted the buildings previously described.
• This showed that the patient also neglected one half of his mental image, mirroring his perceptual deficit.

Dissociations Between Imagery and Perception

• Cases of dissociations between imagery and perception have been reported.
• Guariglia, Padovani, Pantano, and Pizzamiglio (1993) reported about a patient whose brain damage had little effect on his ability to perceive but did cause neglect in his mental images.
• Patient R.M. had damage to his occipital and parietal lobes (Farah, Levine, & Calvanio, 1988).
• R.M. was able to recognize objects and draw accurate pictures of objects placed before him but could not draw objects from memory or answer questions that depend on imagery.
• Patient C.K. suffered from visual agnosia and could not visually recognize objects but was able to correctly draw objects from memory (Behrmann, Moscovitch, and Winocur, 1994).
• When shown his own drawings later, he was unable to identify the objects he had drawn himself.

Making Sense of the Neuropsychological Results

• The neuropsychological cases present a paradox: some cases show close parallels between perceptual deficits and deficits in imagery, while others show dissociations.
• The presence of a double dissociation is usually interpreted to mean that the two functions (perception and imagery) are served by different mechanisms.
• However, this conclusion seems to contradict the other evidence we have presented that shows that imagery and perception share mechanisms.
• The existing model suggests perception involves bottom-up and top-down processing while imagery heavily uses top-down processing.

The Role of Gaze in Mental Imagery

• It takes time to generate comprehensive visual images, and a serial search process is needed to inspect all individual items (Laeng, Bloem, D'Ascenzo, & Tommasi, 2014).
• Gaze patterns during the recollection of a particular scene look remarkably similar to those measured during actual perception of that same scene (e.g., Johansson, Holsanova, & Holmqvist, 2006; Martarelli & Mast, 2013; Wynn, Shen, & Ryan, 2019).

Eye Movements During Recollection

• The fact that scan patterns are very similar for retrieved and perceived visual information provides supportive evidence for the idea that visual imagery is spatial in nature and that it shares mechanisms with perception.
• It raises the question of why there are eye movements during the recollection of visual scenes when there is nothing to see.
• These eye movements likely reflect the content of the original scene.
• Laeng and Teodorescu (2002) found that when participants viewed a grid pattern containing black cells, those who were allowed to freely move their eyes during memorization showed similar scan paths during visualization.
• Those who maintained fixation during encoding showed hardly any eye movements during visualization.
• The more similar the imagery and perception scan paths were, the better the participants performed in a subsequent spatial memory task.
• Laeng and Teodorescu (2002) concluded that eye movements during visualization are used as some kind of spatial reference in the process of image regeneration.
• When such eye movements are prevented, the process of image reconstruction is disrupted, leading to a reduced ability to gain information from the mental image.
• Johansson and Johansson (2014) controlled for cognitive load and differentiated the influence of eye movements on remembering intrinsic object features from their influence on remembering spatial relationships between objects.

Eye Movement Hotspots

• For real-life objects, there are typically more eye fixations at those parts of the objects that are the most informative.
• Lang et al. (2014) wondered whether such eye-movement "hotspots" are also present in imagery after an object picture had disappeared from the screen.
• If so, this would be evidence that eye movements not only relate to the spatial aspects of a memorized object (where) but also to its contents (what).
• They hypothesized that if these eye movements accompany some kind of rehearsal process of the original picture, then the greater the resemblance between fixation patterns during and following perception, the better the subsequent memory performance would be.