Lecture 8 - Imagery

Define imagery.

A mental representation of a perceptual experience.

Is imagery solely visual?

No, imagery comes in other modes too.

Explain what a symbolic representation is.

An arbitrary mental representation of a concept.

Why are symbolic representations arbitrary?

It is arbitrary because the symbol itself does not represent the object itself, but it denotes its meaning.

ie. the word ‘dog’ is a symbolic representation because it makes us think of a physical dog even though the word ‘dog’ does not resemble a dog in any way.

What are the characteristics of a symbolic representation?

1. It is often stored as language

2. It can be schematic

3. It can be an abstract concept of knowledge

Is imagery the same as a symbolic representation?

No, imagery is not the same as a symbolic representation.

What theory describes symbolic representation?

Paivio’s dual code theory.

Explain Paivio’s dual code theory.

It is the idea that our thoughts can be represented in 2 ways:

1. As words (symbolic code)

2. As images (analogue code)

But that everything can be represented verbally, but not everything can be represented as an image. Hence, some items have a dual code, and some have just one code.

Explain an example of an object with a dual code, and another object with just one code.

Dual Code:

A cat can be represented verbally, or through an image. However, typically an image is opted for because its easier to just picture it.

Single Code:

We can describe freedom verbally, but it is not easy to picture it.

What is the imagery debate about? Explain the imagery debate.

Imagery debate is about how knowledge is represented in our minds. There are 2 debated views on this:

1. Functional equivalence hypothesis

2. Propositional Theory

Explain the functional equivalence hypothesis.

Kosslyn proposed knowledge is stored as an image (analogue code) that is a mental copy of the object we see in reality. He also felt that the images were represented as spatial representations. Meaning, the mental image in our mind has the correct proportions to the object in real life.

ie. if you see a dog in real life, it takes up space and likewise, the image in your mind will take up space that is proportional to what you see in reality. So the distance between the dog’s head and tail in real life will match your mental image of it.

Explain the propositional theory.

Pylyshyn proposed that knowledge is stored as propositions, and that images are by-products of the proposition.

This means that images are not represented as an analogue code, but propositions generate images in our minds.

What is a proposition?

It is the smallest idea that someone can say is true or false. Hence, it is a purely mental thing.

ie. “This is a cup” That’s a proposition because one can either say yes that is a cup, or no that is not a cup.

or

“This cup” That is not a proposition because this cup what?? People cannot say that idea is true or false, so it is not a proposition.

How do people express propositions?

Through verbal code (language).

Is a proposition the same as a verbal code? Why?

No, it is not the same as a verbal code. This is because you must first think of a proposition and then you use a verbal code (language) to express that proposition.

So verbal code is merely the thing you use to express the proposition, but it is not the proposition itself.

How does Pylyshyn’s Propositional Theory about images connect with the idea that propositions are not a verbal code?

Similar to how a person must think of a proposition, and then use a verbal code to express it, in propositional theory, someone must first think of the proposition and then they can use imagery to express it.

How are propositions represented?

In a propositional network.

What type of research has been conducted to address the imagery debate? Explain how certain results either support or refute Kosslyn or Pyshylyn’s hypotheses.

Many experiments have been conducted to test functional equivalence by testing to see if images are the fundamental unit of thought. Because if they are, then the images should be analogous to perception (ie. people’s behaviour should be the same when they perceive things (hear, see, touch, smell), and when they just thought about the object).

If people’s behaviour are found to be the same through perception and just thinking about the images, then that supports Kosslyn’s functional equivalence hypothesis.

However, if people behave differently to when they perceive things vs just think about it, it supports Pyshylyn’s propositional theory.

What were the experiments conducted to investigate the imagery debate?

1. mental rotation

2. image scanning

3. image scaling

4. perceptual interference

5. ambiguous figures

Explain the mental rotation experiment and the results.

In this experiment, two images are shown on a screen side by side. They are 2D images, but they are meant to represent 3D shapes.

The two images are rotated at different angles. Participants are then asked to figure out if those two shapes are the same. The time it takes them to figure this out is measured (ie. the dependent variable).

The results show that when the shapes are the same, the time it takes people to figure out if they’re the same or not gradually increases as the angle of rotation increases.

However, when the shapes are not the same, the time is takes people to figure out if they’re the same or not is much slower than when the shapes are the same, and the reaction time is the same regardless of the angle of rotation.

Whose theory does the mental rotation experiment support? Why?

This experiment supports the functional equivalence hypothesis.

This is because, when participants saw both shapes, they mentally rotated the shapes until they could find a match between the objects. When the shapes were different, they had to mentally rotate the objects longer than when the shapes were different to try and find a match, which made their reaction times slower (and vice versa).

This is the same thing that we would do if those objects were given to us in real life. The more you have to rotate the object, the longer it takes to figure out if the object is slowing down our reaction times. Whereas, the less we have to rotate the objects, the faster we can figure out if they’re the same or not, decreasing our reaction times.

Hence, the pattern of our behaviour when perceiving the object (ie. touching and seeing), were the same as when we just thought about the object.

Explain the image scanning experiment.

Kosslyn showed people a map, and asked people to memorise the map. Once people could accurately memorise the map from memory, he asked them questions about the map:

1. he asked them to draw the map

2. then he asked one group to imagine they’re on the beach in the map and then mentally scan the map until you’re at the windmill. When they reach the windmill, press a button.

3. he then asked another group to do the same as the other group but scan till the lighthouse.

Kosslyn measured participants response times to see how far people perceived the distance from the beach to the windmill or lighthouse in their minds.

This is because, if imagery is a functional unit of thought, then the distance people perceive from the beach to the windmill should be shorter than the lighthouse because even when you just draw a line from the beach to the windmill or lighthouse, it will take you longer to draw a line to the lighthouse than the windmill.

Hence, if people’s behaviour matched real life when just thinking about moving from the beach to the location, then it would support functional equivalence hypothesis. But if the behaviour did not match, then it would support propositional theory.

What were the results from image scanning? Whose hypothesis did it support and why?

Kosslyn found that the farther people had to travel mentally on the map, the longer it took them to get to the location. Thus, it supports functional equivalence hypothesis.

This is because it mirrors behaviour in the real world because in real life, the walk from the beach to the windmill would be shorter than to the lighthouse. Also, just drawing a line from the beach to the windmill would be shorter than to the lighthouse. And people’s behaviour did change according to the distance shown in the map, even when they were only mentally travelling.

Explain why the image scanning experiment’s results does not match with propositional theory.

Propositional theory would suggest that people just have a list of the relationships between the structures on the map (ie. the windmill is on the northeast part of the map), but it’s a description, and not an actual spatial representation of the map and its landmarks.

Explain the criticism in the image scanning experiment?

Kosslyn told participants to ‘mentally scan the map.’ Which implies that people should be taking more time to travel greater distances, because it is not necessarily a natural response to mentally scan.

Hence, some argue that perhaps Kosslyn’s instructions altered participants natural response.

How was the image scanning experiment altered to account for the criticism in the initial experiment? What were the results?

Instead of scan the map, participants were told that there are lights at each landmark and that right after the light turns on at the beach, it turns on at the windmill (or lighthouse). Then, participants response times were measured again.

The results showed that response times were the same for both conditions, so distance on the map no longer made a difference.

How does the altered image scanning experiment change the conclusion?

1. It could refute functional equivalence hypothesis because the results did not show people taking more time to travel further distances (which is what would occur IRL)

2. However, based on the instructions given about lights going off, maybe participants weren’t travelling at all because if you were just looking at lights flashing on a map IRL, you would not see a difference in response times either.

Therefore, this experiment does not have a conclusive conclusion, but generally researchers find support for functional equivalence.

Explain the basis for image scaling experiments.

Kosslyn was interested if people’s perception of objects in reality were proportional to the objects in the real (like do the sizes of the objects in real life match the sizes we have displayed in our minds?).

To investigate this, he conducted image scaling experiments.

Explain image scaling experiments.

He got people to imagine a rabbit next to an elephant.

Then, he asked participants questions regarding details about the animals (ie. do they have eyelashes?). He measured people’s response times to those questions.

If a rabbit and elephant were present in real life, it would be easier to spot eyelashes on the elephant compared to the rabbit because it is much bigger. Hence, you would need to travel closer to the rabbit to inspect it’s eyes and see if it has eyelashes. This takes time.

Therefore, if images are functionally equivalent to perception, then you would have to mentally zoom into the rabbit to answer that question, but you wouldn’t have to zoom in for the elephant.

What were the results for the image scaling experiment? Whose hypothesis do the results support?

Functional equivalence hypothesis.

This is because Kosslyn found that participants responded faster to questions about visual details about elephants, compared to rabbits.

What is a criticism about the image scaling experiment?

Kosslyn tested Americans. Rabbits are very common in America compared to elephants. Therefore, it might just be that people payed more attention to the elephant than the rabbit because if was more interesting (and attention facilitates processing so, makes sense).

What did Kosslyn have people do as a result of criticism to his initial image scaling experiment? What were the results? Whose hypothesis do the results support?

Kosslyn had people image a rabbit and a fly. Then, the asked them about whether they have whiskers.

It was found that response times were faster for he larger animal (the rabbit compared to the fly). Hence, supporting the functional equivalence hypothesis.

What was the criticism about Kosslyn’s image scaling experiment about the fly and the rabbit?

People argued that perhaps the fly is particularly boring compared to a rabbit. Or maybe people know a lot elephants, less about rabbits and nothing about flies.

Therefore, perhaps their response times mirror their knowledge rather than anything about imagery.

How did Kosslyn reconcile the criticism about knowledge being measured rather than imagery processing? What were the results? What hypothesis did the results support?

He got participants to imagine an elephant sized rabbit, and a fly sized elephant. Then he asked participants the detailed questions.

What he found was that people’s response times were faster for the elephant sized rabbit, compared to the fly sized elephant.

Therefore, supporting the functional equivalence hypothesis because it suggest that participants had to mentally zoom in to the smaller object because it was taking up less mental space (hence, maintaining those spatial relations).

What is the basis for perceptual interference experiments?

It was the idea that on top of our behaviours being the same via perception and imagery, researchers thought that the same mechanisms are used when we process perception and imagery.

Mechanisms is a broad term because this idea was initially proposed in the 1970s. However, what researchers meant was the type of cognitive processing should be the same for perception and imagery.

Now, researchers extend this thought to be that we expect the same brain mechanisms during perception and imagery.

Explain the rationale for the perception and imagery experiment.

If perception and imagery are analogous, and our cognitive resources have a limited capacity, if we are using resources for imagery, we should have less capacity for perception. Therefore, we should see some interference between perception and imagery.

Explain what happened in the perception and imagery experiment. What were the results?

Participants were told to imagine a tree, and another group of participants were asked to imagine the sound of a piano.

Then, all participants engaged in a detection task where they had to look out for a physical stimulus and when it appears, press a button. The physical stimulus was either an arrow on the screen, or a telephone ring. The stimuli were very low intensity.

Participants who imagined a physical stimulus (the tree), detected the telephone ring (auditory stimulus) faster than the arrow (physical stimulus).

However, participants who imagined an auditory stimulus (sound of piano) detected the arrow (physical stimulus) faster than the telephone ring (auditory stimulus).

What do the results from the perception imagery experiment suggest?

The results suggest interference occurred because when people were imagining a physical stimulus, they were using up their visual processing abilities, making them slower when detecting the real physical stimulus.

On the other hand, when they were imagining something auditory, that used up auditory processing resources, meaning fewer were available resources to detect the actual auditory stimulus. Thus, slowing down detection.

Therefore, supporting the idea that imagery and perception use similar cognitive processes, which also supports functional equivalence hypothesis.

Explain another study about perception and imagery regarding FFA and PPA brain regions. What were the results? What was the conclusion?

We know that FFA is the brain region that responds to faces, and PPA is the brain region that responds to houses.

Therefore, a researcher showed a series of faces to participants, and a series of houses to participants and measured the activity of their FFA and PPA brain regions.

Then, the researcher asked participants to think about the faces that they were just shown and the houses that they were just shown. Again, the activity in their FFA and PPA brain regions were measured. This was process was repeated twice.

The results indicate that there was activity in the FFA region when participant’s viewed faces, and very little activity when they viewed houses. Conversely, there was activity in the PPA region when participant’s viewed houses, and very little activity when they viewed faces.

The same activity pattern in FFA and PPA regions were displayed when participants merely imagined the faces and houses, respectively. Therefore, supporting the perception and imagery use same brain regions.

***However, there is less activity during imagery compared to perception. Hence, the processes during both are not exactly the same. This is not surprising because people should be able to differenciate reality from their imagination.

Is mind reading possible by observing activity in people’s FFA and PPA brain regions? How?

Yes, by observing the activity in people’s FFA and PPA regions, researchers in the field can tell what people are thinking.

Explain the study about ambiguous figures.

Reed showed participants a shape and asked them to memorise the shape. Afterwards, they were shown two shapes, and asked if either of them were in the original shape they saw.

Despite the first shape being correct, only 14% of participants correctly identified that it was in the original shape.

Therefore, suggesting that people’s ability to deconstruct images into parts from memory is very difficult and many oftentimes get it wrong.

Whose hypothesis does the ambiguous figures study support? Why?

It refutes the functional equivalence hypothesis. This is because when people saw the image they were not storing the image, but a proposition (ie. star, 2 overlapping triangles etc…). However, none of the propositions are of a parallelogram. Therefore, when people retrieved their proposition, people were unable to answer about a parallelogram because it’s not part of people’s proposition.

However, in real life, if someone was shown the shape in real life, they would have no problem finding the shape because they can just look and compare.

Therefore, perception and imagery did not yield the same results when it came to mentally deconstructing images.

Which experiments support which hypotheses?

What hypothesis do most researchers adhere to regarding the imagery debate? What do they focus on?

Researchers typically take both into account and think about what we do with imagery or propositions?

Define picture superiority effect.

The idea that if we form an image of something it can improve our LTM of the thing.

Explain why picture superiority effect occurs.

Paivio’s dual code theory suggests that if you have a dual code (verbal and visual code) of something, you will remember the item better than if you only have a singular code of the item (ie. the default verbal code).

How does the interactiveness of an item influence our memory? Why?

The more interactive an item is, the more likely we are to remember it later.

This is because the bizarreness effect suggests that if we have an interactive image, it will be very distinct/unusual. Thus, standing out in our minds and enhancing our memories.

What is the basis for embodied cognition?

Because both functional equivalence hypothesis and propositional theory are based on classical theories where thoughts are mental representations of the item in the real world.

This is an issue due to the symbol grounding problem where if you only stick with the mental representation, there is no way to properly derive meaning of what that representation is, unless you physically interact with it. Otherwise, it is just a symbol that you must use other symbols to explain (ie. if you have an apple, you can describe it as a fruit, but then what is a fruit…so the only way to understand what an apple is is to interact with it in the real world).

However, humans don’t have an issue with this as we physically interact with items all the time in the world. Hence, the problem with functional equivalence hypothesis and propositional theory are not the grounding problem, but the fact that the theories don’t consider the fact that humans do interact with the physical world.

Therefore, the embodied view arose.

Define embodied cognition.

When the brain, body and environment interact in meaningfully ways to form intelligent behaviour.

*must be meaningful in the sense that our interactions with the environment change our perception/behaviour.

Explain an example about a meaningful interaction between our body and our environment (ie. embodied cognition).

Participants were asked to judge the slope of an incline from the bottom of a ramp. One group of participants were carrying an empty backpack, and another group was carrying a heavy backpack.

Those carrying the heavier backpack judged the incline to be steeper than those with the empty backpack.

This is a meaningful interaction between our body and our environment because people with the heavier backpack actually thought that it would be harder to walk up the incline, making them think it looked steeper compared to those who thought walking up would be easier.

What evidence used to support functional equivalence hypothesis can be used to support embodied cognition?

1. Mental Rotation

One could argue that the reason why mental rotation is analogous to physical rotation (in the mental rotation experiment) is because what we are doing is recreating what is happening in real life in our brain. Hence, reliving the experience.

2. Perceptual Interference

For example, if you see a chair certain neuron’s in your brain will be active, and when you remember the same chair those same neuron’s will also be active. Hence, imagery is analogous to physical perception (which also supports embodied cognition).

Define semantic dementia.

When people progressively lose their knowledge of concepts due to damage to the anterior temporal lobe.

Where did cognitive neuroscientists think that semantic knowledge was stored?

Due to people with semantic dementia all having damage to their anterior temporal lobe, researchers thought that semantic knowledge was stored there.

Why does it not make sense that semantic knowledge is stored in the anterior temporal lobe?

This is because some people with semantic dementia lose some categories of semantic knowledge and not others.

For example, damage to someone’s visual areas is frequently associated with a loss of knowledge of living things. However, damage to motor areas is typically associated with the loss of knowledge of non-living things.

However, neither of those two areas are connected to the anterior temporal lobe, so semantic knowledge cannot just be stored there because knowledge deficits occur even though there is no damage to the anterior temporal lobe.

Moreover, imaging studies in healthy participants suggest that semantic knowledge is found in various regions of the brain.

Explain what study that reconciles the differences between cognitive research and embodied cognition.

Hauk et al. (2004)

Looked at brain activity of participants when they either moved their foot, their finger or their tongue. Researcher know that there are different regions in the brain in charge when people move their foot, finger or tongue.

Hence, unsurprisingly, when people moved different regions of their body, different regions of the brain was active.

Afterwards, the researchers looked at the parts of the brain that were active when participants read action words; kick, pick, lick, actions done with your foot, finger and tongue, respectively.

Researchers found that when participants moved their foot and read the action word associated with foot, the same brain regions were active. This happened for the other words and actions.

This supports that our knowledge of the words kick, pick and lick involve simulations of the actions in our minds as the same motor areas involved in doing the actions were involved in reading about the actions. Therefore, reconciling cognitive neuroscience with embodied cognition.

The idea that semantic knowledge is stored in the anterior temporal lobe is still not reconciled by Hauk et al’s (2004) findings, so what theory reconciles it all?

Hub-and-spoke model.

It says that the hub (located in the anterior temporal lobe) is a modality specific concept centre where all the abstract information is stored. Hence, it gives us access to information all over the brain.

Then, it says that there are spokes which are modality specific sensory and motor areas which are activated by the hub and contain specific information about sensory/motor neural representations of the item.

ie. if you want know what an apple looks like, you activate a visual spoke. If you want to know what an apple sounds like, you activate a auditory spoke.

How do we know the inferior parietal lobe is a spoke and not a hub? And how do we know the anterior parietal lobe is a hub, not a spoke?

In a study TMS, a technique used to disrupt neuron’s in a specific part of the brain, was applied to the anterior temporal lobe (ATL) and inferior parietal lobe (IPL).

While this happened, participants were shown a picture and then tasked to name the picture; non-living or living. The naming speed was measured.

When TMS was applied to ATL, for both non-living and living things, naming speed was slowed under TMS. Therefore, indicating the ATL is involved in naming all things—the hub.

However, when TMS was applied to IPL, the naming speed only decreased for non-living things, but not living things. Thus, suggesting that IPL is involved with accessing knowledge for non-living things, but not living things.

Therefore, supporting the idea that IPL is a spoke because performance only decreased for some knowledge (non-living things). This makes sense as you typically only manipulate non-living things, rather than living things and so it makes sense why the spoke would be sensitive to things related to sensory/motor interactions.

Moreover, supporting the idea that ATL is a hub because TMS did not affect performance regardless of the knowledge type.

What other study was conducted to further solidify the idea that IPL is a spoke?

The same experiment was conducted, but instead of non-living and living things being tested, only non-living things were used.

Low represents how hard it is to manipulate something with your hands (ie. couch), and high means you can manipulate it very easily with your hands (ie. clay).

When TMS was added to IPL, naming speed decreased a lot when the object was very easily manipulated, but did not change when it was not easily manipulated. Suggesting, that IPL is involved in storing knowledge about non-living things that are easily manipulated with your hands, and not non-living things that are not easily manipulated.

Also, when TMS was added to ATL, naming speed decreased for both low and high manipulability objects, further supporting that it’s a hub that involves knowledge of all items.

Do you know all the learning objectives?

***check the other side too for more learning objectives!