Lecture 7: hyperdimensional computing

what was Kanerva’s paper about?

• types of computations we should consider when building a model

• how are brain and computers different, what can one do that the other cannot

what are the different aspects of training models? (2)

• developmental psychology: types of input and language that kids receive (spoken ≠ written)

• demographic: characteristics of language input (how time and place influence the type of language you learn)

historically, on what do we based our understanding of how the brain works?

according to metaphors, especially of sophisticated technology from that time (ex: computer, sewage systems, telephone)

by who Pavlov (classical conditioning) was influenced?

• by Descartes who suggested that our thinking and higher order cognition are part of our soul (immaterial)

• our neural and nervous systems would be driven by pumps and sewage systems: nervous system sends transmission the same way water flows down a pipe

who thought that cognition worked like a telephone network?

behaviourists: different signals are sent different ways

what’s part of a computer’s architecture? (4)

• CPU: central processor

• RAM: main memory

• hard drive: secondary memory

• input and outputs

according to Atkinson and Shiffrin, how is memory organized?

• you have sensory registers (short term storage for perpetual information that is selected by attention)

• short term memory maps onto main memory (AKA working memory where you maintain info and use it to solve problems)

• then it gets into long term memory (permanent recordings of what happened to you and your knowledge)

• this is then mapped onto secondary memory

*this is a pretty accurate representation of memory

how does our brain use information?

• take input

• turn it into logic based code

• turn into central processing reasoning system and use it to solve problems

what is the fundamental question in cognitive science?

• nature VS nurture: are your brains with equal skills and cognitive abilities because they are…

• nature: wired in

• nurture: acquired in the same fashion based in environmental constraints

how does Konerva understand the difference between the brain and computers?

• most computers have similar architecture: they are compatible (we can put a PC component in another PC)

• but every brains are different: there is a lot of variance terms of how we differ… but does it mean that we still are equivalent in problem solving skills and cognitive abilities?

what’s the difference between fMRI and PET?

• fMRI: look at where the hydrogen atoms are in the brain

• PET: inject reactive chemical in circulatory system to see where the blood flows in the brain

*for both, we look at the activation pattern in the brain

define the “functional specialization” idea

your brain is highly specialized in terms of certain areas to do specific things

in which brain area is facial recognition done?

in the fusiform face gyrus

true or false: the fusiform face gyrus is specialized in facial recognition only

false: while it is a specialized area, it isn’t for facial recognition only, it’s an expertise area for classifying visual information

explain the main theory of language evolution

language evolved by overtaking domain general capabilities like memory and attention (and these were instead integrated in language processing)

what do we need to understand about the neural system to know if a model is accurate? (2)

• how they are learning information

• how the information is being stored and represented

what’s a problem when we try to create neural networks?

• they are realistic at a small level, but once they get bigger they aren’t really realistic anymore

• we try to use backpropagation but that’s controversial

true or false: there is dependency in representations in a computer

true: the represented info of a computer is transformed into a binary code because it’s the only language the CPU can use

true or false: Konerva assumed that we could take perceptual information and transform it into a logic based representation which would would be used by our cognitive mechanisms in daily life

false: Konerva believed that representation is fundamental and drive what can propose in terms of cognitive mechanisms (we need to understand the representations first)

what are the representations used by a computer?

binary representations

according to Kanerva, how can neurons have similar representation-processing connections to binary representations?

• you would need to ignore a bunch of neuroscience (ion channels, membrane potentials) and focus on the mathematical properties of neural representations

• we look at the mathematical properties about how a neural code can be used to drive an information processing machine

what are the properties of neural representations? (2)

• hyperdimensionality: neural circuits are large (neurons and synapses), so any representations used has to be massive in the number of things being involved

• robustness: our brain are resilient towards damage (especially compared to computers)

define “hyperdimensionality”

neural circuits are large (neurons and synapses), so any representations used has to be massive in the number of things being involved

define “robustness”

our brains are resilient against damage, especially compared to a computer

define “dimensionality reduction”

instead of directly representing what we see, we can take that information and make it a more efficient representation by reducing its dimensionality (kind of like LSA)

define “holistic representations” (AKA “distributed representations”)

we can represent information in a whole form and nothing is encoding specific features from of that form

what are the different types of representations found in cognitive psychology? (2)

• localist: single element or neuron represents a specific property of an item (one neuron = one concept, ex: grandmother cell)

• holographic: information is distributed in terms of what it is representing (many neurons in pattern = one concept)

define “localist representation”

single element or neuron represents a specific property of an item (one neuron = one concept)

ex: grandmother cell (if that cells die, you can’t recognize your grandma)

define “holographic representation”

information is distributed in terms of what it’s representing (pattern of neurons activated = one concept)

what’s the difference between holistic, holographic and localist representations?

• holistic: we can represent information in a whole form and nothing is encoding specific features from of that form

• holographic: information is distributed in terms of what it is representing (many neurons in pattern = one concept)

• localist: single element or neuron represents a specific property of an item (one neuron = one concept, ex: grandmother cell)

true or false: while holistic representations require the activation of many neurons, it is resistant to noise

true

define “randomness”

mix of genetics and experience factors during development (it’s not nature VS nurture)

why do toddlers lose neurons after birth

• kid loses a lot of neurons because they aren’t being activated

• once a kid is born and interacting, certain pathways become strengthened while others die

what does Kanerva mean when he says that representation are initially (at birth) random?

• we start with no knowledge, we learn through the accumulation of random patterns that come from the environment

• the environment will dictate the cognitive skills required

• since we grow in somewhat similar environments, we end up with the same cognitive skills needed to be part of human society

define “self-organization”

accumulation of random patterns from experience leading to a structured system

define “recursion”

continually embed phrases within a sentence to combine different ideas

ex: the man went to the park → the man who fell down went to the park → the man who love his wife who fell down went to the park

when studying the Piraha language, what did we understand from our cognitive skills?

• cognitive skills you acquire depend on the culture you’re part of

• on the basic level, almost all cultures studied are similar

• the Piraha were a tribe who didn’t use recursion or a numerical system because they didn’t need it