Lecture 1: body schema and multisensory integration

Moravec’s paradox

-easy to make computers exhibit adult level performance on intelligence tests

-difficult or impossible to give them the skills of a one-year-old when it comes to perception and mobility

-find high level cognition relatively easy

-find low level suprisingly hard

Johansson - method

-fine motor skill task → grab match and light it

-two conditions:

1. hand anesthetized → no touch sensation from fingers but does not affect motor control

2. control condition

Johansson - results

-control condition nothing unusual and could light match in about 5 seconds

-anaesthesia condition struggle to pick up match and strike it properly, took about 25 seconds despite motor control being unaffected by anaesthesia → result of lack of tactile feedback

multisensory perception

-if senses are lacking then performance degrades

-touch tells us the force pushing against, shape of something, how we are holding it

-vision is seeing the object and where we need to use it

-but these multiple senses need to be used together to form perception

multisensory integration

-need to integrate information across many senses so that it can be used jointly/together

-end product is coherent perception of the object

challenges in multisensory integration

1. how to transform representations from different senses into a common representational space? → information comes in different formats and hard to directly compare so needs to be put into common representational space

2. how to integrate information from different senses into a coherent percept? → sensory information does not always agree with each other so need to know what to agree with

localising sensory percepts

• vision → eye-centred, location of visual stimulus on the retina

• audition → head-centred, location of sound source with respect to the ears

• touch → body-centred, location of tactile stimulus on skin

-need to convert between these reference frames and to external space

coordinate transformations

• eye-to-head → need to know orientation of eyes

• head-to-body → need to know orientation of head

-converting between reference frames requires us to know the position and orientation of body parts

-need a body schema

body schema

-representation of the positions of body parts in space, which is updated during body movement

body schema vs body image

• body schema → sensorimotor representation that guides action, doesn’t require awareness

• body image → how imagine your body, how feel about body, conscious visual representation

characteristics of body schema

1. spatially coded

2. modular

3. updated with movement

4. adaptable

5. supramodal

6. coherent

7. interpersonal

spatially coded (characteristics of body schema)

-position and orientation of each body part in external space

-purpose is to put things into space and keep track of this

modular (characteristics of body schema)

-different body parts are processed separately

-not a singular process

-different parts of the body schema track position and orientation of different body parts

updated with movement (characteristics of body schema)

-automatic and cannot be turned off

-can interfere with perception in some instances when it should not or doesn’t need to

adaptable (characteristics of body schema)

-changes when body changes

-can be maladaptive

-body schema needs to adapt to size and proportion of current body we inhabit

supramodal (characteristics of body schema)

-gets information from lots of senses

-combined input from multiple senses

-any senses useful for body schema, proprioception, any relevant spatial information is also incorporated

coherent (characteristics of body schema)

-spatial continuity when resolving perceptual conflicts

-feel our body is a single entity, never have feeling that a limb is disembodied or feeling broken up into parts

interpersonal (characteristics of body schema)

-observed actions by others are translated into our own body schema to reproduce these different actions

pinocchio illusion

-holding nose with hand

-bicep is vibrated to activate proprioceptors which signals to brain that arm is extending

-perceptual consequence is that body schema thinks arm is extending while still holding our nose → so feels as though nose is very long

-feel nose is still attached as this fits principle of spatial unity

rubber hand illusion

-participant’s real hand hidden from view

-experimenter strokes real and rubber hand simultaneously

-participants sees and feels rubber hand being stroked → multisensory integration important to illusion

-participant starts to feel as if rubber hand is real hand

-around 25% don’t feel illusion at all

explaining rubber hand illusion

-if sensory modalities are in conflict when viewing rubber hand, plausible visual information about arm or hand location can dominate proprioception

-but proprioceptive information can play the same role, shown in the absence of visual information about limb position

temporal order judgement task

-two body parts stimulated in quick succession

-have to indicate which part was stimulated first

Heed - method (does body posture affect perception)

-stimulate both hands in random order

-participants blindfolded and have to indicate which hand was stimulated first

-vary delay between stimulation to make task harder

Heed - conditions (does body posture affect perception)

1. arms uncrossed

2. arms crossed

-should not affect task performance

-should not require access to body schema because don’t need to convert between two reference frames

Heed - results (does body posture affect perception)

-participants mix up which hand was stimulated when arms are crossed

-solving the task does not need input from body schema → performance shows body schema affecting basic perception

Heed - psychophysical curve (does body posture affect perception)

• uncrossed posture → black line, line is close to zero (clustered scores) so people are mostly guessing correctly

• crossed posture → red line, big gap around zero, much more sloped, so people are more inaccurate

begum ali - method (how does the body schema develop?)

-toddlers make an orientating response (reach) towards a foot if it is stimulated

-temporal judgement task

-two conditions:

• feet crossed

• feet uncrossed

begum ali - results (how does the body schema develop?)

-at 4 months no difference in performance at crossed and uncrossed, hard to see where they are reaching → no interference from body schema

-at 6 months uncrossed condition was the same at 4 months

→ but crossed condition was at chance level

-shows body schema interferes with their perception in this task and tactile orientating response

maravita - method (cross-modal integration)

-participants receive a tactile stimulus (buzz) and respond with the location → up (index finger) or down (thumb)

-visual distractor presented on same hand with the tactile stimulus or on other hand without the tactile stimulus

• congruent condition → tactile stimulus and visual distractor in same location

• incongruent condition → tactile stimulus and visual distractor in different locations

maravita - results (cross-modal integration)

-congruency effect = RT for incongruent condition minus RT for congruent condition

-congruency effect exists for both hands → crossmodal congruency effect

-but effect was greater for same hand (62ms vs 20ms)

-evidence for cross-modal integration and visual interference

maravita - second method (cross-modal integration and body schema )

-arm-crossing changes the mapping of the body schema

-conditions:

• tactile stimulus in the same side of the body

• visual stimulus on different side of the body

-effect of visual distractor moves with the hand during arm-crossing

-body schema mediates the integration between vision and tactile stimulation

maravita - second result (cross-modal integration and body schema)

-visual stimuli closest to the current hand position produce the largest crossmodal congruency effects on the tactile judgements for the hand (67 ms vs 3ms)

-the combinations of retinal visual stimulation and somatotopic stimulation that produce the largest interference change with postural changes

crossmodal precueing

-presentation of a cue can enhance judgements for a target that is presented shortly afterwards in spatial proximity

-doesn’t matter if target is in the same or different sensory modality as the cue

-modulated by the current posture of the hands

alice in wonderland syndrome (disorder of body schema)

-distortion in size perception

• microsomatognosia → body parts appear smaller

• macrosomatognosia → body parts appear larger

-can affect whole body

-associated with childhood and migraines

autopagnosia (disorder of body schema)

-inability to locate body parts

-loss of spatial unity of body

-patients can name body parts but relative order lost

-can name body parts but cannot arrange body parts in the way they are arranged in their own body

finger agnosia (disorder of body schema)

-fused percept of fingers, cannot feel them individually

-feel fused or merged

phantom limbs (disorder of body schema)

-presence of limb is still felt after limb loss

-inability of body schema to adapt after traumatic loss

-can include agency over limb

-often associated with pain

• telescoping → phantom limb can change size over time

peripersonal space

-space immediately surrounding our bodies

-objects within this space can be grasped and manipulated immediately

-can contract and expand

neurons incorporating tools (peripersonal space)

-recorded visuotactile neurons in parietal cortex

-individual neurons had high response whenever object moves into peripersonal space

-trained monkey in using rake as a tool to extend reachable space

-use of rake induced an expansion of response space and found same neurons responded to stimuli at far end of tool which would have been unreachable without it

-shows body schema can extend along a wielded tool

tool use - method (cross-modal integration and body schema)

-maravita method

-visual distractors at end of long tools

-tools held in crossed or uncrossed arrangement

-would crossing the tools reverse the impact of left vs right visual distractors?

tool use - results (cross-modal integration and body schema)

-if tools were uncrossed the crossmodal congruency effects were stronger from a visual distractor on the same side as the vibration (74ms vs 54 ms)

-if tools were crossed (right hand to left visual field) then crossmodal congruency effects became larger from visual distractors in the visual field opposite to the stimulated hand (79ms vs 37ms)

-shows tools become part of body schema