Clique-Based Neural Associative Memories

Abstract

Auto-associative memories store a set of patterns and retrieve them by resorting to a part of their contents. This thesis focuses on developing and extending a type of associative memories relying on a sort of coded neural networks called clique-based neural networks.

Background. Both associative memories and erasure correcting decoders deal with similar tasks that revolve around retrieving missing pieces of information. However, despite the similarity of their respective tasks, there is a gap in terms of efficiency and performance which motivates applying coding techniques in the design of associative memories. Clique-based neural networks, introduced by Gripon and Berrou, denote a family of associative memories that are inspired by biological considerations as well as concepts from information theory. The usage of error-correcting coding and decoding techniques, borrowed from the field of information theory, considerably boosts the performance of these associative memories. The proposed neural network is organized in clusters of interacting neurons such that patterns can be stored as neural cliques, which in turn can be seen as codewords of a code. The tournament-based neural network is an extension of clique-based neural networks with the ability to store sequences. In this model, sequences of any length can be stored as chains of tournaments. Both clique-based and tournament-based associative memories have considerably larger storage capacity than the Hopfield model, which is commonly considered as the benchmark model for associative memories.

Contribution. The aim of this thesis is to advance the research area in associative memory by generalizing the concepts of clique-based and tournament-based neural networks. The generalization is expected to yield superior efficiency and retrieval performance. In this thesis, we use the following approaches. First, in Paper I, the coding techniques are used in two levels to enhance storage capacity and retrieval of partial erasures. In Paper II a modification to the structure of clique-based neural networks is proposed to enhance the error-tolerance of the memory. Lastly, in Paper III, a modified version of tournament-based neural networks is used for retrieval of a sequence from a given segment by means of forward and backward retrievals. Moreover, the sequence retrieval performance is enhanced with the new retrieval techniques.

Discussion. We achieve the aim of generalizing the clique-based associative memories originally proposed by Gripon, Berrou, and co-authors to more resilient memories via using coding theory and graph theory approaches while maintaining their biologically plausible structures. The proposed models are quite flexible and can be employed collectively.

Keywords. Neural Associative Memory, Content Addressable Memory, Error-Correcting Codes, Sparse Graphs, Sequence Storage, Clique-Based Neural Networks, Tournament-Based Neural Networks.