Humans adaptively integrate memory and perception based on stimulus history

Visual representations that are actively held in mind are often systematically distorted by new visual stimuli, prompting claims that memories are vulnerable to unwanted integration with new perceptual inputs when they recruit shared sensory representations. Here we challenge this vulnerability perspective by showing that individuals rationally adapt the degree to which memory and perception are integrated based on their recent history of being mutually informative. Across three experiments, we show that the exact same pairs of remembered and perceived stimuli are more likely to be perceived as similar and elicit larger memory biases when preceded by other pairs that were similar as well. These adaptive modulations in integration with stimulus history do not depend on corresponding changes in sensory strength, implying a separate, higher-order mechanism beyond sensory recruitment. These findings are consistent with the predictions of normative cognitive models that have successfully characterized biases across visual perception and memory.