Sparse Coding of Sensory Inputs

This paper describes sparse coding of associative memory patterns and its networks usage for large-scale modeling of the brain. The mechanism of associations in human memory was stated earlier by psychologists and philosophers and it has been a topic of research in the field of neuroscience and artificial neural network for over half-century which continues its investigation even today. The working of associative memory in the human brain is observed when we try to remember and cannot recollect a particular bit of information instantly.

At this state, our mind probably contains the features of current scenario and related information (not the appropriate information) about what we are trying to remember which initiates a back-to-back process of associations from one to another (probably by similarity of meaning or logic in human language) eventually arriving to recognize that information which fits appropriately into the context that stimulated our search. So, relating to contextual information our brain probably acts like information system which associates a new output to a particular input. Technically there are three different mechanisms of association process. One is hetero-association which searches one pattern from a category which is associated to another pattern in another category, second is auto-association which searches within a pattern to recall a complete or exact pattern, and third is a special case of auto-association named as a bidirectional association which goes back and forth between two patterns.

Now the question put forth by neuroscientists is how the process of association in mind is related to (or realized in) neurophysiological mechanisms in the brain. A rule named synaptic plasticity formulated by Donald Hebb explains that the strength of the synapses between any two cells is strengthened by repeated firing of one cell to another cell. This is otherwise called Hebbian theory which paved the way for the development of Neural Associative Memory (NAM) models. Neural Associative Memory model stores the weights of synaptic connections between neurons and is stored in a memory storage matrix. The process of storing and retrieving a set of specified patterns is expressed by using an additive rule or binary rule with an appropriate threshold value for a specific set of patterns. Consider a learning process in matrix formation (memory) wherein each learning step at a time, the change of matrix formation (memory) depends upon the product of presynaptic activity and postsynaptic activity at the synapse at that specific time. Thus synaptic change is computed based on additive rule locally across time and space which is termed as local learning rule. The output of this model is binary {0, 1} which contains the stored pattern that is based on Hebb learning rule and the sparseness of these patterns is more productive for storage and retrieval of information.

Efficiency becomes critical when NAM model is used in many technical applications. Vector-matrix multiplication is done for entries in {0, 1} for retrieval and then with counting, threshold and finally, if input patterns are sparse then the retrieval of stored patterns becomes faster. These sparse binary patterns are used in spoken word recognition, face recognition, written letter recognition which has a large number of classes. Video signals are another example where compression codes are based on the sparseness of signal differences. In the primary visual cortex of visual system of humans and animals sparseness works in edge detecting cells and also in antagonistic activation of retinal ganglion cells. Machine learning and signal processing are the other areas where sparseness principle is used other than associative memory. It is even natural to say that human cognition occurs out of sparseness, as we cannot conceive anything as absent like non-dog or non-apple, as it encloses every concept.

NAM functionalities have become important in parallel computing architectures where inter-process communication is a bottleneck, and so the idea of transmitting addresses of few active neurons which saves transmission rate because of sparse activity was adopted into parallel computing. This concept has been used in a pulse coded neural network which focuses on hardware implementation for brain simulations. In neuroscience, NAM has become the surge of interest for spiking neuron models and synchronization. Based on the work done by Fransén & Lansner et.al [1] it is possible to construct a cognitive behavior which is biologically plausible based on associative memory structures by creating cortical modules which can interact in a meaningful way.

An example work done by Fay et.al [2] for modeling cerebral cortex as a network of associative memory modules which controls the robot to perform a specific action based on command sentences was achieved by building a system containing 30 modules. These larger modules containing thousands of neurons may possibly merge the gap between lower level neurophysiological observations and higher cognitive psychological level behavior at a transitional level where it is open to ideas and interpretations from both the fields. Edelman et.al [3] in their work also discusses the possibilities of ideas and projects that deal with associative memory models.

Problem Statement: Sparse coding modeling (cerebellum) with the biologically realistic neural network model.