Understanding the flexibility of working memory: Compositionality, generative processing, anchors and holistic representations

The typical conception of working memory is a mechanism to temporarily hold multiple discrete objects in service of other cognitive tasks in an item-based representation. In this paper, we expand the conventional idea that working memory represents objects into a more flexible framework that uses compositional and generative mechanisms to code and then re-code visual input according to task demands. Compositionality allows complex scenes or objects to be mentally decomposed into constituents that can be individually manipulated or recombined to form new representations. Generative processing allows purely conceptual information to be reconstructed in a format akin to visual sensory representations that can be manipulated and re-processed by perceptual mechanisms. Together, compositional and generative mechanisms would enable a wide range of cognitive functions including the basis of visual imagery. In our view, working memory items do not need to correspond to discrete objects, but could serve as pointers or anchors to clusters of features that form parts of objects, or alternatively, multiple objects could be encoded as one holistic item depending on the task. We conclude with a conceptual account of such a memory system that can build and re-use information by moving it between different levels of abstraction within a perceptual hierarchy. This model is linked to experimental results from the memory and visual imagery literatures that illustrate the flexibility of such a system for performing cognitive tasks.