Real-time Multimodal Sensory Detection Using Widefield Hippocampal Calcium Imaging

The hippocampus is a complex structure that has a major role in learning and memory. It also integrates information from multisensory modalities, supporting a comprehensive “cognitive map” for both spatial and non-spatial information. Previous studies have been limited to real-time spatial decoding, typically using electrodes. However, decoding hippocampal non-spatial information in real-time has not been previously described. Here, we have constructed a real-time optical decoder driven by calcium activity of large neuronal ensembles to decode spatial, visual, and auditory information effectively. Using advanced machine learning techniques, our rapid end-to-end decoding achieves high accuracy and provides a “cognitive translation” method. This method enables the real-time investigation of hippocampal neural coding and allows for direct neural communication with animals and patients affected by functional impairments. The ability to decode multimodal sensory inputs in real-time thus forms the basis for a novel all-optical brain-computer interface.