finding structure

Finding Structure in Time

• Author: Jeffrey L. Elman, University of California, San Diego

• Published in: Cognitive Science, 1990.

Overview

• Importance of time in human behavior and cognition, especially in language.

• Discusses representation of time in connectionist models.

• Introduction of recurrent links to implement dynamic memory in networks is highlighted.

Implicit Representation of Time in Networks

• Dynamic Memory: Networks use feedback loops to use internal representations that depend on prior states.

• Simulation Results: From simple problems (like temporal XOR) to complex language features (syntactic/semantic attributes).

• Presents a model that incorporates both task and memory demands, showing context-dependency and generalization across items.

Introduction

• Cognition is linked with time, which influences goal-directed behavior and language.

• Challenges arise in creating models that address the temporal aspects of cognition.

The Problem with Time

• Traditional models struggle with temporal representation:

  • Action plans in motor tasks: whether they specify output literally or abstractly.

  • Linguistic theorists often overlook how temporal sequencing impacts language processing.

• Spatial Representation Limitation:

  • Treating time as a spatial metaphor is not effective for capturing many temporal behaviors.

  • Sequential inputs present challenges in learning and generalizing over time.

Networks with Memory

• Recurrent connections allow networks to maintain state across cycles, lending networks the ability to process information dynamically.

  • Introduces hidden and context units for memory effects in network architecture.

  • Describes mechanisms of information processing using recurrent links.

Temporal Version of XOR

• XOR function identified as significant:

  • Traditional networks cannot learn it due to its structure requiring three layers.

  • The sequential XOR is redefined such that it processes 1-bit inputs in time steps.

• Simulation shows that the network can predict consecutive bits based on previous states, signifying learning of temporal relationships.

Structure in Letter Sequences

• Extended learning tasks demonstrated through binary input patterns simulating speech sounds.

• Simulations show how networks can still make predictions despite increasing complexity of input patterns, illustrating memory capacity and redundancy can aid learning.

Discovering the Notion of "Word"

• Examines how language learners categorize sounds (words, morphemes) and the fluidity in defining these units.

• Simulation: Presented letters as input streams without explicit word boundaries to learn word structure.

  • Context influences predictions and highlights learning based on co-occurrences in sequences.

Discovering Lexical Classes from Word Order

• Language structure, like sequential constraints in word order, may influence processing and learning structures in networks.

• Networks can capture appropriate response probabilities in their predictions.

Type/Token Distinction in PDP Networks

• Discusses representation differences between traditional and PDP models.

  • PDP's strength lies in its ability to handle both types (categories) and tokens (individual instances) as contextual, not rigid.

Conclusions

• Represents several cognitive insights:

  • Many problems change nature when framed temporally, particularly in learning and processing tasks.

  • Performance benefits from implicitness in the representation of tasks over time, leading to task-sensitive memory dynamics.

  • Networks demonstrate potential in developing structured memory representations through representation of time.