YORU: social behavior detection based on user-defined animal appearance using deep learning

25 The creation of tools using deep learning methodologies for animal behavior analysis has 26 revolutionized neuroethology. They allow researchers to analyze animal behaviors and reveal 27 causal relationships between specific neural circuits and behaviors. However, the application 28 of such annotation/manipulation systems to social behaviors, in which multiple individuals 29 interact dynamically, remains challenging. Here, we applied an object detection algorithm to 30 classify animal social behaviors. Our system, packaged as "YORU" (Your Optimal 31 Recognition Utility), classifies animal behaviors, including social behavior s, based on the 32 shape of the animal as a "behavior object" . It successfully classified several types of social 33 behaviors ranging from vertebrates to insects . We also integrated a closed-loop control 34 system for operating optogenetic devices into the YORU package. YORU enables real-time 35 delivery of photostimulation feedback to specific individuals during specific behaviors, even 36 when multiple individuals are close together. We hope that the YORU system will accelerate 37 the understanding of the neural basis of social behaviors. 38 39

40 Social behaviors such as courtship, aggression, and group formation are important for 41 improving survival rate and reproductive efficiency in a wide range of animals, including 42 invertebrates and vertebrates 1–3. To accelerate our understanding of the neural basis of these 43 behaviors, it is necessary to capture information on the location and type of social interactions 44 that occur between individuals 4. Recent advances in machine learning have led to the 45 establishment of various tools for animal behavior detection, represented by markerless body 46 part tracking 5–7. These emerging tools enable real -time behavior analysis, allowing 47 researchers to manipulate neural activity precisely when the animal exhibits the behavior of 48 interest 8–10. Such a closed-loop approach promises to clarify the causal relationship between 49 the neural activity and behaviors , and indeed has yielded significant success in uncovering 50 neural basis for single individual behavior 7,11,12. However, detection of the social interaction 51 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 3 between multiple individuals with low latency remains challenging 4,6,7. This difficulty arises 52 from the complexity of defining social behavior based on the coordinates of each individual's 53 body parts, preventing the effective use of a closed -loop approach 6,7,13. Therefore, novel 54

are required to enable the detection of social interaction and to perform real -time 55 behavior analysis for neural intervention. 56 Social interactions are often accompanied by individuals striking distinctive postures 57 to interact with others, such as wing extension by courting male fruit flies and mounting on 58 females by male mice attempting copulation 14–16. In this study, we propose the detection of 59 social behaviors of multiple individuals based on their appearance s by defining each as a 60 “behavior object”. To this end, we focused on an object detection algorithm “YOLOv5”, a 61 high-speed object detection algorithm based on a convolutional neural network (CNN) 17,18. 62 The algorithm achieves fast object recognition by framing detection as a single regression 63 problem, using a unified architecture that processes the entire image in one forward pass 17,18. 64 The YOLO-based detection is robust to variations in object orientation, size and background 65 noise 19. YOLOv5 is therefore a promising algorithm to detect various animals’ social 66 behaviors without relying on the body -part coordinates, making it well -suited for adapting 67 behavior analysis and a closed-loop system to investigate social interactions. 68 Here, we established a behavior detection system called “YORU” (Your Optimal 69 Recognition Utility), based on the YOLOv5 algorithm 18. First, to verify the proof of concept, 70 we tested whether YORU could be used to detect social behaviors of various animals ranging 71 from vertebrates to insects. As an application of our behavior analysis, we compared the 72 detection readouts with neural activity imaging in mice to interpret large-scale brain activity. 73 Secondly, we evaluated the inference speed of the detection and feedback latency in YORU's 74 real-time analysis. Finally, as a practical example of the YORU ’s closed-loop system, we 75 experimented with optogenetic neural manipulation focused on courtship behavior in 76 Drosophila. In the presence of multiple flies, we successfully manipulated the neural activity 77 of individuals in an individual selective manner by optogenetic stimulation of the fly 78 exhibiting the behavior of interest. 79 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 4 80

81 YORU is a framework for animal behavior recognition 82 Here we introduce “YORU” (Your Optimal Recognition Utility), an animal behavior 83 detection system with a graphical user interface (GUI). In the YORU system, animal 84 behaviors, either performed by single animals or multiple animals interacting with each other, 85 are classified as a “behavior object” based on their shapes with YOLOv5 18 (Fig. 1a). YORU 86 is an open -source Python software composed of four packages: “Training”, “Evaluation”, 87 “Video Analysis” and “Real -time Process” ( Fig. 1b, Figs. 1a-e). The YORU system is 88 designed to work both offline (video analysis) and online (real-time analysis) to detect animal 89 behavior. During video analysis, YORU allows us to quantify animal behavior according to 90 user-defined shapes of social behavior. For real -time analysis, YORU analyzes animal 91 behavior in real -time, which can be used to output trigger signals accordingly to control 92 external devices, such as LEDs for optogenetic control as set by the user. 93 We set three constraints to the workflow of YORU: ease of use for experimenters, low 94 system latency, and high customizability. To make YORU user -friendly, we designed the 95 system to allow easy quantification of animal behaviors without requiring programming 96 knowledge. To achieve low system latency , special efforts were focused on behavior 97 detection and feedback when operating as a closed-loop system. Here, immediate behavior 98 detection is achieved by YOLOv5 -based object detection algorithm, which processes 99 generating region proposals and classifying subjects simultaneously, resulting in faster 100 detection 17,18. Immediate feedback is achieved by the parallel processing design of YORU 101 real-time process 20: image acquisition, object recognition, and hardware (Arduino, DAQs, 102 etc.) manipulation are not processed serially but simultaneously. To achieve high 103 customizability, i.e., adapting YORU system to various experimental systems easily, YORU 104 implements a trigger output for hardware manipulation . The e xperimenter can customize 105 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 5 YORU’s parameters with no or minimal user programming. These features enable 106 experimenters to easily use high-performance closed-loop neural feedback. 107 Detection of social behaviors using object detection algorithm 108 In previous studies, YOLO-based object detection successfully classified some social 109 behaviors of Drosophila, such as offensive behaviors and copulation 21,22. In this study, we 110 extended this idea and analyzed a variety of animal behaviors with YORU. The following 111 social behaviors were tested to validate the performance : (1) Fruit flies; wing extension 112 behavior of a male toward a female during courtship 14 (Fig. 2a), (2) Ants; mouth-to-mouth 113 food transfer behavior among workers (trophallaxis) 23,24 (Fig. 2b), and (3) Zebrafish; 114 orientation behavior toward another individual behind a partition 25,26 (Fig. 2c). We extracted 115 2000 images from multiple videos and manually labeled their behavior objects with the 116 following definitions. 117 (1) Fruit flies: “wing_extension”; a fly extending one of its wings (Fig. 2a). When the flies 118 were not labeled as “wing_extension”, they were labeled as “fly”. 119 (2) Ants: “trophallaxis”; heads of two ants engaging in a food exchange, with their mouth 120 parts in contact with each other. “no”; the situation of no food exchange although the heads 121 of the two ants were close together (Fig. 2b). No label was given when neither of these two 122 behavior types were detected. 123 (3) Zebrafish: “orientation”; two zebrafish exhibiting orientation behavior as defined in a 124 previous study 25,26, “no_orientation”; a zebrafish exhibiting no orientation behavior (Fig. 2c). 125 We created models and compared their detection accuracies with human annotations using 126 multiple videos that were not used for model creation. The Accuracy score for the model's 127 detection of fruit flies, ants, and zebrafish behaviors were 93.3%, 98.3% and 90.5%, 128 respectively, when compared to human manual annotations (Fig. 2d-f, Supplementary Table 129 1). We also compared zebrafish orientation analyses between the previous method based on 130 body part tracking 25 and human annotations , yielding an Accuracy score of 81.2% 131 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 6 (Supplementary Table 1, Fig. 2f). YORU can thus potentially detect zebrafish orientation 132 quicker and with higher accuracy than the previous tracking methods. These results suggest 133 that YORU’s detection can detect social behaviors with a similar to human annotations. 134 In general, two major factors to consider for the practical application of deep learning-135 based analyses are the amount of training data (the number of labels) and the selection of 136 based networks 5,27. To find optimal conditions in our dataset, we evaluated the accuracy of 137 models with different numbers of training images (200, 500, 1000, 1500, and 2000) and 138 YOLOv5 networks (YOLOv5n, YOLOv5s, YOLOv5m, YOLOv5l, and YOLOv5x). Two 139 metrics were used in this evaluation: “Precision”, the ratio of correctly predicted detections 140 to all predicted detections, and “Recall”, the ratio of correctly predicted detections to all 141 ground-truths. All models detected the target social behavior with Precision greater than 85% 142 and Recall greater than 90%, even when only 200 images were used to create models 143 (Supplementary Table 2). The Precision and Recal l scores increased with the number of 144 training images, finally exceeding 90% and 95%, respectively, when 1,000 or more images 145 were used for training (Supplementary Table 2). We then evaluated typical object detection 146 model indices, intersection over union (IOU) and average precisions (AP) (Fig. 2g-i, Figs. 147 2a-c) 28. These two indexes can be used to compare the accuracy of the model between 148 conditions; the IOU shows the difference in location information between the ground truth 149 and detected bounding boxes while the AP reflects the accuracy of the behavior classes , 150 which includes Precision and Recall values 28. For each behavior, neither the number of 151 images used for training nor the YOLOv5 models tested affected IOU values (Figs. 2a-c). 152 On the other hand, AP values increased with the number of training images , in which the 153 same number of images gave similar AP values across YOLOv5 based models (Fig. 2g-i). 154 These evaluations suggest the primary factor influencing accuracy is the amount of training 155 data, at least in these cases. 156 Next, we increased the number of individuals to test the performance of YOLOv5 157 models in multi-individual conditions. In the first condition, a group of flies was analyzed to 158 detect three objects: “wing extension”, “copulation”, and other behaviors labeled as “fly” 159 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 7 (Fig. 2j, k). In the second condition, a group of ants was used to detect two objects: 160 “trophallaxis” and “no_trophallaxis” (Fig. 2l, m). In the group of flies, Precision and Recall 161 scores were over 95% even when using only 200 images for training; the scores increased 162 with the number of training images (Supplementary Table 2). In the group of ants, Precision 163 and Recall scores exceeded 90% with 500 or more training images (Supplementary Table 2). 164 These findings suggest that YORU’s detection of social behaviors is also applicable to 165 multiple individuals (more than two individuals) with high accuracy (Fig. 2 j-m, Figs. 3a, b, 166 Supplementary Table 2), highlighting its usefulness for analyzing various types of social 167 behaviors. The detection accuracy does not change significantly with different YOLOv5 pre-168 trained models and instead depends strongly on the number of training images. 169 The relationship between behavioral readouts and neural activity interpretation 170 One of the main questions in neurophysiology is the interpretation of which sensations 171 and animal behaviors can explain observed neural activity. One promising methodology to 172 address this question is to combine time series analyses of multiple behavioral types via video 173 analysis with neural activity measurements. To test the performance of YORU for such tasks, 174 we recorded dorsal cortex-wide neural activity with wide -field calcium imaging from mice 175 running in a virtual reality (VR) system which provide s visual feedback coupled to their 176 locomotion 29 (Fig. 3a). A mouse on a treadmill in VR typically exhibits multiple behaviors, 177 such as running, grooming, and eye blinking ( Fig. 3b). First, to validate the behavior 178 classification performance of YORU, we estimated the time series of eight behavior classes 179 from video analysis: “Running”, “Stop”, “Whisker-On”, “Whisker-Off”, “Eye-Open”, “Eye-180 Closed”, “Grooming-On”, and “Grooming-Off”. Precision and Recall scores for the model's 181 detection of these behaviors were 91.8% and 92.7%, respectively, validating this model to 182 detect mouse behaviors as accurately as human manual annotations (Supplementary Table 183 3). For all classes detected by this model, the average IOUs and AP@50 (i.e., the AP value 184 at IOU=50) were above 0.60 and 0.55, respectively (Fig. 4, Supplementary Table 3). Previous 185 studies have reported that rodents actively move their whiskers to seek and identify objects 186 or avoid obstacles in front of them during locomotion 30–33. In line with these reports , the 187 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 8 time series data of active whisker movement (corresponding to “Whisker-On” label) during 188 “Running” periods were positively correlated while “Whisker-Off” and “Running” 189 correlated negatively (Fig. 3b, c). In addition, the "Running" and "Stop" epochs in the YORU 190 readout are negatively correlated, indicating that mutually exclusive behaviors are labeled 191 exclusively (Fig. 3c). These results further confirmed that YORU’s approach to behavior 192 labeling can efficiently detect typical behaviors of mice. 193 Next, we investigated which brain regions in the cortex correlate with YORU’s readout. 194 The “Running” epoch was highly correlated with the neural activity of medial motor areas, 195 somatosensory area s representing forelimb and hindlimb information, visual areas, and 196 posterior association region (retrosplenial cortex) (Fig. 3d). Several whisker movements 197 associated with sniffing during “Stop” period and “Blinking” behavior were also coupled to 198 distinct macroscopic activity pattern s of widespread regions in somatosensory and visual 199 areas. As expected, g rooming behavior correlated specifically with neural activity of 200 forelimb somatosensory and motor areas (Fig. 3d, e). These results demonstrate the 201 applicability of its utility and potential for precise and quantitative interpretation of neural 202 activity with various animal behaviors. 203 Inference speed and system latency of YORU 204 A closed-loop system that relies on live feedback of animal behaviors requires a low-205 latency solution for behavior detection and feedback outputs 12. To assess YORU’s potential 206 for application in closed-loop systems for social behavior analysis, we measured the total 207 time required from frame acquisition to behavior estimation using a simple light detection 208 task (Fig. 4a). Possible major factors that could affect the speed of YORU’s behavior 209 detection are (i) the network structure, (ii) image size , and (iii) computing hardware, 210 especially the graphics processing unit ( GPU). To demonstrate the ir impact on analy zing 211 each frame, we measured the single -frame inference latency with (i) five YOLOv5 212 architectures (YOLOv5n, YOLOv5s, YOLOv5m, YOLOv5l, and YOLOv5x), (ii) two input 213 image sizes (640x480 or 1280x1024 pixels, images were resized before feeding to the neural 214 network), and (iii) a variety of NVIDIA GPUs in Windows PCs ( Supplementary Table 3). 215 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 9 Inference speed was calculated by comparing the time before (t1) and after (t2) the frame 216 was analyzed by a model (Figs. 5a). We found that the smallest network (YOLOv5n) was 217 the fastest and the largest network ( YOLOv5x) had the largest inference latency (Figs. 5b, 218 Supplementary Table3). In addition, the inference speed was faster with a smaller image size 219 and more powerful NVID IA GPUs ( Supplementary Table 3). For example, with NVIDIA 220 RTX 4080 GPU, we achieved inference speed as low as ~ 5.0 ms per frame (~ 200 fps) using 221 YOLOv5s networks (Figs. 5b). These results suggest that the network architecture, input 222 image size, and GPU all influence YORU’s inference speed. 223 The design of closed-loop systems adapted to neuroscience experiments requires the 224 simultaneous control of several processes , such as camera capture and hardware 225 manipulation 34. In addition, several other factors such as camera type and frame rate, trigger 226 destination type, and PC memory can affect system latency. Therefore, w e tested the 227 performance of YORU's closed-loop system from end to end, including the entire process 228 from the camera capturing an image of the LED to the PC detecting whether the LED was lit 229 or not and sending a trigger signal to a data acquisition (DAQ) system upon detecting the 230 “ON” state (Fig. 4a, b). The delay between the timing of the LED turning on and the trigger 231 signal output, both detected by measuring their voltage s using DAQ, was as low as 30 ms 232 per event (Fig. 4c). This suggests that the end-to-end system latency of the YORU system is 233 around 30 ms in this setup , which is sufficiently low in most cases to provide real-time 234 feedback in response to animal behavior. Next, we assessed the factors that affected the end-235 to-end system latency , such as networks, input image size , camera frame rate, and system 236 hardware. The effect of the network differences was almost negligible except YOLOv5l and 237 v5x, which showed larger latency than others (Fig. 4c). On the other hand, t he input image 238 size severely affected the latency; the average latency of the smaller image (640 x 480 pixels) 239 was ~30 ms while that of the larger image (1280 x 1024 pixels) was ~75 ms ( Fig. 4c). 240 According to the camera fps latency results, even if the camera frames were acquired as fast 241 as the model's inference speed (~200 fps), the system delay would not be significant, due to 242 YORU's multi-processing system (Fig. 4d). In a multi -processing system, the size and 243 processing speed of the random access memory (RAM) affect the processing speed of the 244 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 10 system due to the shared memory. Interestingly, in YORU's real-time process, the size of the 245 RAM (between 16 GB and 32 GB) had less effect on the system latencies, suggesting that 246 16GB RAM size is minimally sufficient to operate YORU's real-time process (Figs. 6a). The 247 effect on system latency due to hardware differences (cameras and trigger devices) was small 248 (Figs. 6b, c). These results suggest that inference speed and the input image size are the 249 primary factors affecting end-to-end system latency. In addition, YORU's system processing 250 speed was very fast compared to previously reported tracking-based systems 7. 251 YORU application for real-time optogenetic system 252 Next, to assess the practical applicability of the YORU closed-loop system, we applied 253 it to event-triggered optogenetic manipulation. During courtship, the male fruit fly extends 254 his wing to serenade the female with a unique sound known as the courtship song. Upon 255 hearing it, female flies gradually increase their receptivity to copulation 35. We hypothesized 256 that if male wing extension behavior is inhibited when a male attempts to extend his wing, 257 copulation rates would be reduced. Using a split -GAL4 strain that specifically drives gene 258 expression in pIP10 neurons, which are descending neurons regulat ing courtship song 259 production 36,37, we expressed the green light-gated anion channel GtACR1 38 in male pIP10 260 neurons (Fig. 5a). We then paired individual mutant males with a wild -type female in a 261 chamber and allowed YORU to detect the single-wing extension of a fly. In this system, 262 when YORU detects an object labeled as "wing extension," YORU introduces green 263 photostimulation to the entire chamber (Fig. 5b, c). As a control for photo-stimulation, we 264 used an event-triggered light with a 1-second delay that illuminated the entire chamber (Fig. 265 5c, ‘Delayed’ group). Male flies expressing GFP in pIP10 neurons were also used as a genetic 266 control group. We then analyzed the wing extension ratio during courtship and the copulation 267 rate during the 30-min observation period (Fig. 5d, e). In the experimental group, males 268 decreased the amount of wing extension during courtship (Fig. 5d), validating the optogenetic 269 inhibition of pIP10-induced behavior. In line with this, the cumulative copulation rate was 270 significantly lower in the experimental group than in the control groups (Fig. 5e). These 271

confirm the importance of male pIP10 neurons for inducing the wing extension 272 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 11 behaviors observed in the previous reports 36,37,39, which subsequently leads to male 273 copulation success. They also validate YORU’s performance in operating real-time 274 manipulation of neural activity in response to the detection of a specific behavior. 275 YORU application for individual-focused photo-stimulation 276 Finally, we applied YORU for individual-selective neural manipulation in response to 277 social behavior between multiple individuals. We included a projector in YORU's closed-278 loop system to control the light pattern for optogenetic stimulation (Fig. 6a-c). When YORU 279 detects the target behavior, it sends location information to focus the projector light. We 280 utilized the fly courtship assay to test the usability of the system by suppressing female 281 hearing only when the male sings a courtship song, as exhibited by his wing extension. We 282 hypothesized that disrupting hearing in females during male courtship song production would 283 suppress female mating receptivity, leading to a reduced copulation rate. Using the JO15-2-284 Gal4 strain that selectively labels auditory sensory neurons , we express ed GtACR1 38 in 285 female auditory sensory neurons (i.e., JO-A and JO -B neurons) (Fig. 6d, Fig. 7). We then 286 paired these females each with a wild -type male in a chamber and utilized the online 287 capabilities of YORU. In this system, when YORU detects an object labeled "single -wing 288 extension," the YORU -operated projector illuminates the object labeled as "fly", which 289 typically is the female courted by the male (Fig. 6b, c, e). As a control for individual-focused 290 photo-stimulation, we used pattern light (3s On, 4s Off) illuminating the entire chamber (Fig. 291 6e). In the experimental group (JO15-2>GtACR1 female with event-triggered light 292 condition), the copulation rate was significantly lower than in the control groups (Fig. 6e, f). 293 This result confirms the importance of auditory sensory neurons for females to detect the 294 male's courtship song to enhance copulation receptivity. Again, YORU was able to 295 optogenetically manipulate neural activity using individual-focused illumination, even when 296 multiple individuals were moving in a chamber at the same time. Th ose experiments 297 validated the usefulness of the YORU system, which can manipulate various devices , such 298 as a projector as well as a DAQ and Arduino , by trigger output. These proof-of-principle 299 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 12 experiments demonstrate the usefulness of YORU for the online detection of social behaviors 300 and manipulation of individual-focused neural activity through optogenetics. 301 302

303 Here, we presented YORU, an animal behavior detection system using a YOLOv5-based 304 object detection algorithm. YORU allowed t he detection of social behaviors, as well as 305 single-animal behaviors. Furthermore, by introducing real-time analysis, YORU can operate 306 a closed-loop system with low latencies and high user scalability. We also demonstrated the 307 practical applicability of real-time neuronal manipulation using the fly courtship behavior. 308 In particular, we created an individual-focused illumination system to manipulate the neural 309 activity of selected individuals in response to a specific behavior. The YORU’s closed-loop 310 system is thus a powerful approach for social behavior research. On the user side, YORU can 311 be used entirely through its GUI without any programming. Just as body parts tracking-based 312 analysis tools such as DeepLabCut 5 have revolutionized neuroscience research, YORU will 313 meet the needs of many biologists and stimulate the generation of novel, testable hypotheses. 314 In tracking -based behavioral analyses, behavior is typically defined based on the 315 location of body parts. In addition, clustering analysis (e.g., t -SNE, PCA, UMAP) of body 316 part location coordinates can be used to classify behaviors and detect unknown behavior 317 patterns 5. However, it is challenging for tracking-based analysis to classify already known 318 behaviors with high accuracy in real -time. For example, tracking-based analysis for 319 recognizing behaviors such as wing extension in Drosophila requires recognition of wings 320 and body axes and definition of behaviors by their angles, and accordingly, it would fail to 321 capture behaviors if body parts are not correctly tracked even partially. In addition, to capture 322 the behavior of multiple individuals, we need to know which body part belongs to which 323 animal: Despite various algorithms being proposed to overcome these challenges, capturing 324 the behavior of multiple individuals in real time is still difficult 6,7,13. YORU’s object 325 detection algorithm can sufficiently compensate for the shortage of body part tracking-based 326 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 13 behavior quantification even in multiple animal conditions 6,7. YORU’s processing time is 327 fast enough for real -time processing, even at camera resolutions sufficient for animal 328 behavior quantification. 329 In behavioral neuroscience, o ptogenetics serves as a powerful approach to cell-type- 330 and spatiotemporal-specific control of neural activity, especially for investigating the causal 331 relationship between neural circuits and behavior 8,40. With state-of-the-art genetic tools, it is 332 now possible to express channelrhodopsin only in specific neurons to perform neural activity 333 intervention 8,38. In addition, the development of computer science has made it possible to 334 create closed-loop experimental setups and manipulate neural activity during behavior, 335 allowing us to explore the causal relationship between neural circuits and behavior in more 336 detail 34,41,42. In the research of the neural bases of social behaviors, optogenetic manipulation 337 of only specific individuals, even in the presence of multiple individuals, can be a powerful 338 approach 43. However, the difficulty of capturing the behavior of multiple individuals 339 simultaneously in real -time has made it extremely difficult to conduct online behavior 340 analysis in the presence of multiple individuals . The YORU system allows us to create a 341 closed-loop system that can analyze animal behavior and operate photostimulation for 342 optogenetics in real-time. Furthermore, the YORU system allows us to conduct individual-343 focused optogenetics experiments with widely available equipment: projectors, cameras, and 344 personal computers. 345 Due to YORU’s concept of detecting behaviors by their “snapshot” appearances, it has 346

in detecting behaviors that cannot be defined within a single frame and instead 347 require time-series data for definitions 21, such as foraging or mating attempts . To capture 348 these behaviors, it is necessary to adapt object detection algorithms to these time-series 349 behaviors, or to perform additional analysis using other tools such as DeepLabCut. Another 350

lies in the hardware operation. Although the system delay in our condition was ~30 351 msec, an additional delay in trigger processing depending on the external hardware should 352 be considered . In our experiments of individual -focused phtostimulation, there was an 353 additional delay on the projector side before the patterned image signal was sent and 354 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 14 projected. This time lag may cause fast-moving individuals to move out from the area of the 355 light stimulus. Possible solutions for such cases include incorporating predictive algorithms, 356 or using a high-speed projector such as a low latency gaming projector 44,45. Overcoming 357 these two limitations would further widen the potential options for studying dynamic animal 358 social behaviors. 359 Various analysis tools, driven by advances in deep learning , have contributed to 360 biology research. YORU is a user -friendly system that allows all analyses to be performed 361 via a GUI, making it easy for anyone to perform analysis both in offline and online modes. 362 The object detection paradigm has the potential not only for animal behavior quantification, 363 but also for capturing various biological phenomena . In particular, it has been incorporated 364 as a tracking method for humans and fish, and has been applied to the behavioral 365 classification of animals and plants (e.g., Drosophila mating behaviors and the stomatal 366 opening and closing of Arabidopsis) 21,22,46,47. The YORU system can contribute to making 367 the use of object recognition algorithms more widespread in biology and to significantly 368 reduce the labor of biologists. 369 370

371 Development of YORU system 372 YORU is written in Python. The GUI of YORU is developed using DearPyGui library, 373 which enabled GUI development in a fast, interactive manner, and plotting of acquisition 374 data in real-time. To use various image acquisition devices such as webcams or other high-375 performance machine vision cameras, we used OpenCV API for image acquisition. All 376 processing and data streams were performed in a multiprocessing manner with the Python 377 library “multiprocessing”. To label animal behavior s, YORU use s the open-source 378 annotation software “LabelImg” ( https://github.com/HumanSignal/labelImg). To detect 379 “behavior objects”, YORU uses YOLOv5 packages (https://github.com/ultralytics/yolov5). 380 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 15 The code for YORU is available at (https://github.com/Kamikouchi-lab/YORU) as open-381 source software. In this study, we used ChatGPT (https://chatgpt.com), developed by OpenAI, 382 as a tool to assist in various programming tasks. ChatGPT was used for code generation, 383 debugging, refactoring suggestions, and answering technical questions. While ChatGPT was 384 used to assist in specific tasks, all generated output was reviewed and validated by the 385 researchers to ensure accuracy and reliability. 386 Datasets 387 To evaluate the performance of YORU, we prepared different datasets collected under 388 various conditions. The datasets include “Fly - wing extension”, “Ant - trophallaxis”, “Fly - 389 group courtship”, “Ant - group trophallaxis” , “Zebrafish - orientation”, “LED lighting - 390 Small”, and “LED lighting - Large”. The “Zebrafish - orientation” dataset was created using 391 zebrafish videos from a previous study 26, while the other datasets were generated from 392 videos obtained in this study. Each dataset consisted of images (frames manually extracted 393 from videos) and "behavioral object" labels. Each image was then manually labeled using 394 LabelImg. Supplementary Table 5 shows the detailed condition of each dataset. 395 Flies - wing extension 396 This dataset includes a pair of wild-type fruit flies consisting of a male and a female. 397 Fruit flies (Drosophila melanogaster, Canton-S strain) were raised on standard yeast -based 398 media on a 12 h light/12 h dark (12 h L/D) cycle. Both sexes of flies were collected within 8 399 h after eclosion to ensure their virgin status. They were maintained at 25°C under a 12 h L/D 400 cycle and transferred to new tubes every 2 to 4 days, except on the day of the experiment. 401 Male flies were kept singly in a plastic tube (1.5 mL, Eppendorf) containing ∼200 μL fly 402 food, while females were kept in groups of 10 to 30. Experiments were conducted using 403 males and females 4 to 8 days post-eclosion, with each individual used only once. The video 404 recordings were performed between Zeitgeber Time (ZT) = 1-11 at 25°C and 40-60% relative 405 humidity. 406 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 16 Courtship behavior was monitored in a round courtship chamber with a sloped wall 407 (20 mm top diameter, 12 mm bottom diameter, 4 mm height , and 6 mm radius fillet) made 408 of transparent polylactic acid filament by 3D printer (Sermoon D1, Creality 3D Technology 409 Co., Ltd.). The chamber was enclosed with a slide glass and a white acrylic plate as a lid and 410 bottom, respectively. The chamber was illuminated from the bottom by an infrared LED light 411 (ISL-150×150-II94-BT, 940 nm, CCS INC.) to enable recordings in dark conditions. Male 412 and female flies were gently introduced into chamber s by aspiration without anesthesia. 413 Videos were captured from the top , with a monochrome CMOS camera (DMK33UX273, 414 The Imaging Source Asia Co., Ltd.) equipped with a 25 mm focal length lens (TC2514-3MP, 415 KenkoTokina Corporation ) and a light -absorbing and infrared transmitting filter (IR -82, 416 FUJIFILM), at a resolution of 640 x 480 pixels and 30 fps for 30 min for each pair using IC 417 Capture (The Imaging Source Asia Co., Ltd.). In this dataset, we labeled two behavior object 418 classes, “fly” and “wing_extension” , without a female or male identification. The 419 “wing_extension” was labeled on the fly when it extended one of its wings. The “fly” 420 indicates a fly not showing wing extension. 421 Flies - group courtship 422 This dataset includes a group of wild-type fruit flies consisting of four males and four 423 females. We prepared these flies under the same condition described in the "Fly - wing 424 Extension" dataset. Behaviors were monitored in a fly bowl chamber with a sloped wall (60 425 mm in diameter, 3.5 mm in depth ) 48, illuminated by a visible LED light to facilitate 426 recordings. Four male and four female fl ies were gently introduced into the chamber by 427 aspiration without anesthesia. Videos were captured from the top as described in the "Fly - 428 wing Extension" dataset . In this dataset, we labeled three behavior object classes: “fly” , 429 “wing_extension”, and “copulation” without a female or male identification. The definitions 430 of “wing_extension” and “fly” were based on those in the “Fly - wing extension” dataset. 431 The “copulation” was defined by genital coupling between a male and a female. 432 Ants - trophallaxis 433 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 17 This dataset includes t wo worker ants. Workers of Camponotus japonicu s were 434 collected in October 2023 at the Higashiyama Campus of Nagoya University (35°09'15.3" N, 435 136°58'15.8" E). Ant species were identified with the Encyclopedia of Japanese Ant 49. After 436 collection, they were kept individually in Falcon tubes with moistened paper for 24 h at 22°C 437 with 40-60% relative humidity under dark conditions. The video recordings were conducted 438 at 22°C and 40-60% relative humidity. 439 The trophallaxis behavior was monitored in a custom-made chamber (25 mm width, 440 15 mm length, 5 mm depth ) made of transparent polylactic acid filament by 3D printer 441 (Sermoon D1, Creality 3D Technology Co., Ltd.). The chamber was enclosed with a glass 442 ceiling. Two worker ants were used for a single video recording: one individual was fed 1 M 443 sucrose solution as much as desired immediately before observation, while the other was not 444 fed anything. These two workers were transferred to a custom-made chamber. Videos were 445 captured from the top , with a color CMOS camera (D FK33UP1300, The Imaging Source 446 Asia Co., Ltd.) equipped with a 50 mm focal length lens (MVL50M23, Thorlabs, Inc.) , at a 447 resolution of 1280 x 960 pixels and 30 fps for 30 min for each pair using IC Capture (The 448 Imaging Source Asia Co., Ltd.) . In this dataset, we labeled two behavior object classes: 449 “trophallaxis” and “no”. The “trophallaxis” class was defined as a situation when the heads 450 of two individuals were in proximity and the palps were in contact with each other. The “no” 451 class was defined when the heads of the two individuals were in proximity, but the palps 452 were not in contact with each other. 453 Ant - group trophallaxis 454 This dataset includes a group of ants consisting of six worker ants. Among six workers, 455 three individuals were fed 1 M sucrose solution as much as desired immediately before 456 observation, while the other three were not fed anything. These workers were then transferred 457 to the polystyrene chamber (87mm width, 57mm length,19mm depth) with a glass ceiling, 458 and video recording was started. Videos were recorded from the top with a monochrome 459 CMOS camera (FLIR GS3-U3-15S5; Edmund optics) equipped with a zoom lens ( M0814-460 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 18 MP2, CBC Optics Co., Ltd.). Video recordings and behavior object definitions were 461 performed in the same manner as for the “Ant - trophallaxis” dataset. 462 Zebrafish - orientation 463 This dataset includes a pair of wild-type zebrafish. Adult zebrafish, aged four to six 464 months, with the Oregon AB genetic background were used without sex identification. Fish 465 were maintained in a 14/10 h L/D cycle at 28.5°C. The experiments were conducted as 466 described in previous studies 25,26. Fish were isolated in individual tanks the day before the 467 experiments, with white paper placed between the tanks to prevent visual contact . The 468 experiments were conducted at ZT = 0-14. 469 The orientation behavior was monitored in two custom-made acrylic tanks (90 mm 470 length, 180 mm width, and 60 mm height). These tanks were separated by a divider made 471 from a polymer dispersed liquid crystal (PDLC) film (Sunice Film) attached to 2 mm thick 472 acrylic sheets. Fish were placed individually into water tanks with a depth of 57 mm for 20 473 min. Following this acclimation period, the fish were recorded for 5 min at 30 fps with the 474 opaque divider condition (invisible condition ). The divider was then made transparent to 475 allow the fish to see each other, and the recording continued for an additional 5 min (visible 476 condition). Videos were captured from below, with a monochrome CMOS camera (FLIR 477 FL3-U3-13E4, Edmund optics) at a resolution of 1280 x 1024 pixels and 30 fps. The divider 478 condition (opaque or transparent) was controlled with a DAQ interface (USB-6008; National 479 Instruments Co.) and custom-made software written in LabVIEW (National Instruments Co.). 480 In this dataset, we labeled two behavior object classes : “orientation” and “no_orientation”. 481 The “orientation” class was defined as a situation when two zebrafish showed orientation 482 behavior as defined in a previous study 25. The “no_orientation” class was defined when two 483 zebrafish showed no orientation behavior (Fig. 2f). 484 LED – ON or OFF 485 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 19 This dataset includes a blue LED (OSB5YU3Z74A, OptoSupply). We created “LED 486 lighting - Small” and “LED lighting - Large” datasets; “LED lighting - Small” consists of 487 640x480 pixel videos , and “LED lighting - Large” consists of 1280x1024 pixel videos. 488 Videos were captured with a color CMOS camera (DFK33UP1300, The Imaging Source 489 Asia Co., Ltd) equipped with a 50 mm focal length lens (MVL50M23, Thorlabs, Inc.). In 490 each dataset, we created models based on each YOLOv5 model (YOLOv5n, YOLOv5s, 491 YOLOv5m, YOLOv5l, and YOLOv5x). In these models, there were two classes: “ON” and 492 “OFF”, indicating that LED turned on or off, respectively (Fig. 4a). 493 Model creation for YORU-based analysis 494 We created YORU models using the “Training” package. YORU randomly splits the 495 datasets containing the images and labels into two datasets: 80% for training and 20% for 496 validation. The models were trained with the training and validation datasets for 300 epochs. 497 If the training loss was low enough, training was finished before 300 epochs. For comparison 498 with human manual annotations, we created YORU models based on a YOLOv5s pre-trained 499 model, in which 2000 images were used. Human manual annotations were conducted using 500 BORIS 50. For model evaluation using “Evaluation” package, we extracted the specified 501 number of images from each dataset and created each model based on YOLOv5 pre-trained 502 models. 503 Comparison with YORU and human manual annotations in animal videos 504 To obtain human manual annotation data , we analyzed the behaviors of flies (10 505 videos and ~40 min in total ), ants (3 videos and ~90 min in total ), and zebrafish (2 videos 506 and ~ 10 min in total) , following the behavior def initions (Supplementary table 5). Four 507 parameters, Accuracy, Precision, Recall, and F1 score, were used to compare the 508 performances between the human manual annotation and YORU. To obtain values for these 509 parameters, each annotation was first classified as follows: 510 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 20 ・True Positive (TP): Correct annotation. 511 ・False Positive (FP): Incorrect annotation, such as annotating a non-existing object or a 512 misplaced annotation. 513 ・False Negative (FN): Undetected ground-truth. 514 ・True Negative (TN): Correct no-annotation. 515 Subsequently, Accuracy, Precision, Recall, and F1 score were calculated as follows: 516 Accuracy = TP + TN TP + FP + FN + TN 517 Precision = TP TP + FP 518 Recall = TP TP + FN 519 F1 score = 2 × Recall × Precision Recall + Precision 520 Evaluation of YORU models 521 We evaluated the models using YORU’s “Evaluation" package. Several hundred 522 images, which were not used in the model creation, were manually labeled with ground-truth 523 bounding boxes using LabelImg. YORU then predicted bounding boxes on the same images 524 using the models. By comparing the ground -truth and predicted boxes, we calculated the 525 Precision and Recall of the models. In addition, intersection over union (IOU) and average 526 precisions (AP), two of the typical object detection model indexes , were used 28. IOU 527 represents how close the ground-truth box and the predicted box are, based on Jaccard Index 528 that evaluates the overlap between two bounding boxes , and AP represents the accuracy of 529 the behavior classes , reflecting the “Precision” and “Recall” values 28. Since the AP is 530 affected by the IOU threshold, we used two different threshold values to calculate the APs: 531 "AP@50" with a threshold value of IOU = 50% and "AP@75" with a threshold value of IOU 532 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 21 = 75%. The AP ranges from 0 to 1, with a value of 1 indicating perfect consistency with the 533 ground-truth labels. As with the PASCAL Visual Object Classes Challenge 51, model 534 evaluation in YORU adopts the AP using the all-point interpolation method by the Riemann 535 integral to compare the models. 536 Mice – running in virtual reality and recording neural activity 537 All mice procedures followed institutional and national guidelines and were approved 538 by the Animal Care and Use Committee of Nagoya University. All efforts were made to 539 reduce the number of animals used and minimize the suffering and pain of the animals. The 540 “Mouse - treadmill” dataset consists of 3 videos of single mouse behaviors in the treadmill 541 environment. Referring to a previous study 29, the videos were collected. C57BL/6J mice 542 were purchased from Nihon SLC. These animals were maintained in a temperature -543 controlled room (24°C) under a 12 h light/dark cycle with ad libitum access to food and water. 544 Due to the body size needed for adaptation in head-fixed VR, mice aged 11−28 weeks were 545 used in the VR experiments. The VR environment used in this study was referred to in the 546 previous study 29. Briefly, mice were head -fixed and free to run on a one -dimensional 547 treadmill. We recorded mice behavior with a monochrome CMOS camera (DMX33UX174, 548 The Imaging Source) at 30 fps and 640 × 480 pixels resolution. 549 For the “Mouse - treadmill” dataset, we labeled 2171 frames with eight behavior object 550 classes: “Running”, “Stop”, “Whisker -On”, “Whisker -Off”, “Eye -Open”, “Eye -Closed”, 551 “Grooming-On”, and “Grooming-Off” (Fig. 3b). “Eye-Open” and “Eye-Closed” indicate a 552 mouse that opens or closes the eye. “Grooming-On” was defined as the situation where the 553 forelegs of the mouse are touching its head. “Grooming-Off” was defined as all situations 554 that do not satisfy the criteria of “Grooming-On”. “Running” and “Stop” indicate a mouse 555 that runs or stops. “Running” was defined as the situation where the roller was rotating, and 556 “Stop” was defined as the situation where the roller was stationary. “Whisker-On” was 557 defined as the situation where the whisker was oriented in the anterior direction , and 558 “Whisker-Off” was defined as all situations that did not satisfy the criteria of “Whisker-On”. 559 Labels were randomly split into two datasets: creating a model dataset (1975 images) and a 560 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 22 test dataset (196 images). Using creating a model dataset, we generated a model based on the 561 YOLOv5s pre-trained model. 562 Surgery for the head plate implantation and virus injection were performed as 563 previously described 29. Briefly, Retro-orbital virus injection , The AAV-PHP.eB-hSyn-564 jGCaMP7f was injected at 100 µL into the retro-orbital sinus of mice using a 30-gauge needle. 565 Head plate implantation was performed 14 days after the virus injection. Skin and membrane 566 tissue on the skull were carefully removed, and the surface was covered with clear dental 567 cement (204610402CL, Sun Medical). A custom-made metal head-plate was implanted onto 568 the skull for s table head fixation. During surgery, the eyes were covered with ofloxacin 569 ointment (0.3%) to prevent dry eye and unexpected injuries, and body temperature was 570 maintained with a heating pad. All procedures of surgery were performed under deep 571 anesthesia with a mixture of medetomidine hydrochloride (0.75 mg/kg; Nihon Zenyaku), 572 midazolam (4 mg/kg; Sandoz), and butorphanol tartrate (5 mg/kg; Meiji Seika). After surgery, 573 mice were injected with atipamezole hydrochloride solution (0.75 mg/kg; Meiji Seika) for 574 rapid recovery from the effect of medetomidine hydrochloride. 575 The recording of cortex-wide neural activity was performed as previously described 29. 576 Briefly, we used a tandem lens design (a pair of Plan Apo 1×, WD=61.5 mm, Leica). To filter 577 the calcium independent artifacts 52, an alternating blue (M470L4, Thorlabs, Inc.) and violet 578 (M405L3, Thorlabs, Inc. ) excitation LED s were used as a light source for excitation, 579 combined with band-pass filters (86-352, Edmund; FBH400-40, Thorlabs, Inc., respectively). 580 Fluorescent emission was passed through a dichroic mirror (FF495-Di03) and filters 581 (FEL0500 and FESH0650, Thorlabs, Inc.) to a scientific CMOS (sCMOS) camera (ORCA-582 Fusion, Hamamatsu Photonics). The timing of the excitation light was controlled by a global 583 exposure timing signal from the camera, processed with an FPGA-based logic circuit (Analog 584 discovery 2, Digilent), equipped with binary-counter IC (TC4520BP, Toshiba). 585 Speed benchmarking – inference speed 586 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 23 We measured the inference speed of the YOLOv5 detection using a custom Python 587 code of the YORU’s detection function in the “Real -time Process” package . The list of 588 computers used for this analysis is shown in Supplementary Table 6. In the custom Python 589 code, the times before and after the YOLO detection step (𝑡1 and 𝑡2, respectively) were 590 logged by the “perf_counter()” function of the “Time” module. Then, w e calculated the 591 inference speed of one frame detection as 𝑡2 − 𝑡1. For the analyses on the model size and 592 frame size dependency, we used the 640 × 480 pixel s videos for “LED lightning - Small” 593 models and 1280 × 1024 pixels videos for “LED lightning - Large” models. 50000 frames 594 (60 fps, ~14 min, ~420 events) were used to calculate the inference speed. 595 Speed benchmarking – real-time system 596 To estimate the latency of the YORU’s “Real-time Process” package, we measured the 597 end-to-end latency of the LED light detection task by running YORU on a Windows desktop 598 PC (CPU, Core i7 -13700KF 16core; GPU, NVIDIA RTX 4080; RAM, 32GB or 16GB 599 DDR5). The CMOS camera (DFK 33UP1300, The Imaging Source Asia Co., Ltd or ELP-600 USBFHD08S-MFV, Autocastle) equipped with a 50 mm focal length lens (MVL50M23, 601 Thorlabs, Inc.) captured an LED and streamed frames . The frames were then processed to 602 detect whether the LED state was on or off using the “LED lighting - Small” or “LED lighting 603 - Large” model. When YORU detected the ON state, it sent a signal to a trigger controller, 604 which then emitted a transistor-transistor logic (TTL) voltage pulse. As the trigger controller, 605 DAQ (USB-6008, National Instruments Co.) or a microcontroller (Arduino Uno, Arduino 606 CC) was used. A recording DAQ (USB-6212, National Instruments Co.) logged the TTL 607 voltage from the trigger controller (Fig. 4b). The voltage data were then processed using a 608 50 Hz low-pass filter (“butterworth_filter()” function of “scipy” package) to block the high-609 frequency noises. The delay between the timing of LED voltage and that of the trigger TTL 610 was used to estimate the full -system latency, which includes overhead from hardware 611 communication and other software layers. 612 Optogenetic assays – fly preparation 613 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 24 D. melanogaster were raised as described above . Canton-S was used as a wild -type 614 strain. UAS-GtACR1.d.EYFP (attP2) 38 (RRID: BDSC_32194) and split-GAL4 strain that 615 labels pIP10 neurons specifically (w; VT040556-p65.AD; VT040347-GAL4.DBD) 36 (RRID: 616 BDSC_87691) were obtained from the Bloomington Drosophila Stock Center. JO15-2-GAL4 617 53 was a kind gift from Dr. D. F. Eberl (University of Iowa). 618 For optogenetic assays, pIP10 neurons specific split-GAL4>GtACR1 male or JO15-619 2>GtACR1 female flies were paired with wild-type adult males or females, respectively, as 620 mating partners. Flies used for the behavior assay were collected within 8 h after eclosion to 621 ensure their virgin status. Wild-type male, transgenic male and female flies were kept singly 622 in a plastic tube (1.5 mL, Eppendorf) containing ∼200 μL fly food. Wild-type females were 623 kept in groups of 10 to 30. They were transferred to new tubes every 2 to 3 days, but not on 624 the day of the experiment. Males and females , 5 to 8 days after eclosion , were used for 625 experiments and were used only once. All experiments were performed between ZT = 1-11 626 at 25°C and 40-60% relative humidity. 627 Optogenetic assays – dissection and immunolabeling 628 Dissections of the male genitalia and female head were performed as described 629 previously with minor modifications 11. Briefly, male genitalia were dissected in phosphate-630 buffered saline (PBS: Takara Bio Inc., #T900; pH 7.4 at 25°C), kept in 50% VECTASHIELD 631 mounting medium (Vector Laboratories, #H-1000; RRID: AB_2336789) in deionized water 632 for ∼5 min, and mounted on glass slides (Matsunami Glass IND., LTD, Osaka, Japan) using 633 VECTASHIELD mounting medium. 634 Immunolabeling of the brains and ventral nerve cord was performed as described 635 previously with minor modifications 54. Briefly, brains were dissected in PBS (pH 7.4 at 636 25ºC), fixed with 4% paraformaldehyde for 60 -90 min at 4ºC, and subjected to antibody 637 labeling. Brains were kept in 50% glycerol in PBS for ∼1 h, 80% glycerol in deionized water 638 for ∼30 min, and then mounted. Rabbit polyclonal anti -GFP (Invitrogen, #A11122; RRID: 639 AB_221569; 1:1000 dilution) was used for detecting the mCD8::GFP. Mouse anti -640 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 25 Bruchpilot nc82 (Developmental Studies Hybridoma Bank, #nc82, RRID:AB_2314866; 641 1:20 dilution) was used to visualize neuropils in the brain. Secondary antibodies used in this 642 study were as follows: Alexa Fluor 488 -conjugated anti-rabbit IgG (Invitrogen, #A11034; 643 RRID:AB_2576217; 1:300 dilution) and Alexa Fluor 647 -conjugated anti -mouse IgG 644 (Invitrogen, #A21236; RRID: AB_2535805; 1:300 dilution). 645 Optogenetic assays – confocal microscopy and image processing 646 Serial optical sections were obtained at 0.84 μm intervals with a resolution of 512 × 512 647 pixels using an FV1200 laser -scanning confocal microscope (Olympus) equipped with a 648 silicone-oil-immersion 30× lens (UPLSAPO30XSIR, NA = 1.05; Olympus). Images of 649 neurons were registered to the Drosophila brain template55 by using groupwise registration56 650 with the Computational Morphometry Toolkit (CMTK) registration software . Brain 651 registration, image size, contrast, and brightness were adjusted using Fiji software (version 652 2.14.0; RRID: SCR_002285). 653 Optogenetic assays – retinal feeding 654 Transgenic male and female flies were maintained under a dark condition for 4-6 days 655 after eclosion and then transferred to a plastic tube (1.5 mL, Eppendorf) containing ~200 µL 656 of fly food. Plastic tubes that contain males were divided into experimental and control 657 groups. For the experimental group, 2 μL of all-trans-retinal (R2500, Sigma -Aldrich), 25 658 mg/mL dissolved in 99.5% ethanol (14033-80, KANTO KAGAKU), was placed on the food 659 surface. The male and female flies were kept on the food for 1 day and 2 days, respectively, 660 before being used for the assays. 661 Optogenetic assays – a closed-loop system for event-triggered photostimulation 662 For the event -triggered optogenetic assay (Fig. 5), we used the YORU system on a 663 desktop Windows PC (the same machine used in the speed benchmarking experiments). 664 Green LED light (M530L4, Thorlabs, Inc.) was used as the light source. The cameras used 665 to record the fly behavior and photostimulations, respectively, are as follows: a monochrome 666 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 26 CMSO camera (DMK33UX273, The Imaging Source Asia Co., Ltd.) equipped with a light-667 absorbing and infrared transmitting filter (IR-82, FUJIFILM); a color CMOS camera (DFK 668 33UP1300, The Imaging Source Asia Co., Ltd). The monochrome CMOS and color CMOS 669 cameras were equipped with a 25 mm focal length lens ( TC2514-3MP, KenkoTokina 670 Corporation) and zoom lens (MLM3X-MP, Computar), respectively. The fly behaviors were 671 recorded at a resolution of 640 by 480 pixels and a frame rate of 100 fps for ~40 min for each 672 fly pair using YORU. The recordings of the photostimulation were performed at a resolution 673 of 1280 x 1080 pixels resolution and 30 fps. The chamber was illuminated from the bottom 674 by an infrared LED light (ISL -150×150-II94-BT, 940 nm, CCS INC.) to enable recordings 675 in dark conditions. We used a round courtship chamber with a sloped wall (19 mm top 676 diameter, 12 bottom diameter, 4 mm height, and 6 mm radius fillet) made of the transparent 677 polylactic acid filament produced by a 3D printer (Sermoon D1, Creality 3D Technology Co., 678 Ltd.). The chamber was enclosed with a slide glass and a white acrylic plate as a lid and 679 bottom, respectively. 680 For analyzing the behaviors, a male and a female fly were introduced into the chamber 681 by gentle aspiration without anesthesia. YORU captured and detected two behavior object 682 classes. When YORU detected a “wing extension”, a green light (light intensity (530 nm): 683 2.0 mW/cm2) illuminated the entire chamber. As a control experiment of the light condition, 684 we used an event-triggered light with a 1-second delay to illuminate the entire chamber (Fig. 685 5c). The light intensity of photostimulation was calibrated with a n optical power meter 686 (PM100D, Thorlabs, Inc.). 687 688 Optogenetic assays – a closed-loop system for individual-focused photostimulation 689 For the individual-focused optogenetic assay (Fig. 6), we used the YORU system on 690 a desktop Windows PC (the same as for the speed benchmarking experiments). A projector 691 (HORIZON Pro, XGIMI Technology Co.) was used as a light source to stimulate specific 692 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 27 individuals. Camera captures, chambers, and background light setup were configured in the 693 same manner as in the event-triggered photostimulation experiments. 694 Before the experiments, the positions of the camera and projector plane s were 695 calibrated using a circle grid pattern (5 circles height x 8 circle s width). The 696 “findHomography” function of “OpenCV” package provided the homography matrix that 697 linearly transformed the position from the camera plane to the projector plane. 698 For analyzing the behaviors, a male and a female fly were introduced into the chamber 699 by gentle aspiration without anesthesia , and analyzed by YORU. When YORU detected a 700 “wing extension”, a circle-shaped illumination (~6.5 mm diameter with 1.14 mW/cm2 light 701 intensity at 530 nm) was delivered to the individual that was detected as a “fly”. The center 702 coordinates of the predicted bounding boxes were homography transformed using the 703 homography matrix. As a control for the light condition, the projector displayed a patterned 704 light consisting of alternating 3-s of green and 4 -s of black ”, covering the entire chamber. 705 This was controlled using a custom Python code. This green/black time ratio was determined 706 based on the calculation of courtship duration of wild-type Drosophila in courtship assay 707 videos. The light intensity of photostimulation was calibrated with a n optical power meter 708 (PM100D, Thorlabs, Inc.). 709 Quantification and statistical analysis 710 Statistical analyses were conducted using Jupyter (Python version 3.9.12) and RStudio 711 (R version 4.3.2). Aligned rank transform one -way analysis of variance (ART one -way 712 ANOVA) was performed to compare the wing extension ratio between conditions in the 713 event-triggered optogenetic assay. All statistical analyses were performed after verifying the 714 equality of variance (Bartlett’s test for three -groups comparisons; F -tests for two -groups 715 comparisons) and normality of the values (Shapiro -Wilk test). Kaplan-Meier curves were 716 generated using R , and a pairwise Log-rank test was performed to compare females’ 717 cumulative copulation rates between conditions in the individual-focused optogenetic assay. 718 After ART one-way ANOVA or pairwise Log-rank tests, p values were adjusted using the 719 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 28 Benjamini-Hochberg method in the post hoc test. For the ART one -way ANOVA, the 720 ARTool package (version 0.11.1) was used ( https://github.com/mjskay/ARTool/) 57,58. For 721 Kaplan-Meier curves and pairwise Log-rank test, the survival package (version 3.5.7) was 722 used (https://github.com/therneau/survival). Statistical significance was set at p < 0.05. 723 Boxplots were drawn using the R package ggplot2 (https://ggplot2.tidyverse.org/). Boxplots 724 represent the median and interquartile range (the distance between the first and third 725 quartiles), and whiskers denote 1.5 × the interquartile range. 726 727 Data and code availability 728 ・YORU software and all original code ha ve been deposited at GitHub and are publicly 729 available at https://github.com/Kamikouchi-lab/YORU. 730 731 Declaration of generative AI and AI-assisted technologies in the writing process 732 During the preparation of this work the authors used DeepL ( https://www.deepl.com) and 733 ChatGPT (https://chatgpt.com) in order to improve language. After using this tool, the 734 authors reviewed and edited the content as needed and take full responsibility for the content 735 of the publication. 736 737 Acknowledgments 738 We thank Mayu Yamanouchi for the software logo design; Masahiko Hibi and Shiori 739 Hosaka for collecting animal behavior videos; Ryota Nishimura (the Research Equipment 740 Development Group, Technical Center of Nagoya University) for the chambers used in 741 behavioral experiments, and construction virtual reality environment for mice ; Matthew P. 742 Su, Kentaro Noma, Tomoka Nozaki and Haruka Ando for discussions; Kei Ito, Daniel F. 743 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 29 Eberl and the Bloomington Drosophila Stock Center (Indiana University, Bloomington, IN, 744 USA) for fly stocks; Developmental Studies Hybridoma Bank for antibodies. This study was 745 financially supported by the MEXT KAKENHI Grant -in-Aid for Transformative Research 746 Areas (A) “iPlasticity” (JP23H04228 to A.K.), Grant-in-Aid for Transformative Research 747 Areas (A) Hierarchical Bio-Navigation (JP22H05650 and JP24H01433 to R.T.) , Grants-in-748 Aid for Scientific Research ( C) (JP23K05846 to R.T. and JP23K05845 to T.S. ), Grant-in-749 Aid for Early-Career Scientists (JP20K16464 to R.F.T. and JP21K15137 to R.T. ) JST 750 FOREST (JPMJFR2147 to A.K.), the Japan Society for the Promotion of Science (JSPS) 751 Grant-in-Aid for JSPS Fellows (No. JP24KJ1290 to H.M.Y.) Japan. 752 753

754 1. Silk, M. J., Finn, K. R., Porter, M. A. & Pinter-Wollman, N. Can Multilayer Networks 755 Advance Animal Behavior Research? Trends Ecol. Evol. 33, 376–378 (2018). 756 2. Han, J. et al. Analysis of social interactions in group -housed animals using dyadic 757 linear models. Appl. Anim. Behav. Sci. 256, 105747 (2022). 758 3. Batista, G., Levine, J. D. & Silva, A. Editorial: The Neuroethology of Social Behavior. 759 Front. Neural Circuits 16, 897273 (2022). 760 4. Bordes, J., Miranda, L., Müller -Myhsok, B. & Schmidt, M. V. Advancing social 761 behavioral neuroscience by integrating ethology and comparative psychology methods 762 through machine learning. Neurosci. Biobehav. Rev. 151, 105243 (2023). 763 5. Mathis, A. et al. DeepLabCut: markerless pose estimation of user-defined body parts 764 with deep learning. Nat. Neurosci. 21, 1281–1289 (2018). 765 6. Lauer, J. et al. Multi-animal pose estimation, identification and tracking with 766 DeepLabCut. Nat. Methods 19, 496–504 (2022). 767 7. Pereira, T. D. et al. SLEAP: A deep learning system for multi-animal pose tracking. 768 Nat. Methods 19, 486–495 (2022). 769 8. Bernstein, J. G., Garrity, P. A. & Boyden, E. S. Optogenetics and thermogenetics: 770 technologies for controlling the activity of targeted cells within intact neural circuits. Curr. 771 Opin. Neurobiol. 22, 61–71 (2012). 772 9. Poth, K. M., Texakalidis, P. & Boulis, N. M. Chemogenetics: Beyond Lesions and 773 Electrodes. Neurosurgery 89, 185 (2021). 774 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 30 10. Fenno, L., Yizhar, O. & Deisseroth, K. The Development and Application of 775 Optogenetics. Annu. Rev. Neurosci. 34, 389–412 (2011). 776 11. Yamanouchi, H. M., Tanaka, R. & Kamikouchi, A. Piezo -mediated 777 mechanosensation contributes to stabilizing copulation posture and reproductive success in 778 Drosophila males. iScience 26, 106617 (2023). 779 12. Kane, G. A., Lopes, G., Saunders, J. L., Mathis, A. & Mathis, M. W. Real-time, low-780 latency closed-loop feedback using markerless posture tracking. eLife 9, 1–29 (2020). 781 13. Biderman, D. et al. Lightning Pose: improved animal pose estimation via semi -782 supervised learning, Bayesian ensembling and cloud-native open-source tools. Nat. Methods 783 21, 1316–1328 (2024). 784 14. Villella, A. & Hall, J. C. Chapter 3 Neurogenetics of courtship and mating in 785 Drosophila. Adv. Genet. 62, 67–184 (2008). 786 15. Burns-Cusato, M., Scordalakes, E. M. & Rissman, E. F. Of mice and missing data: 787 what we know (and need to learn) about male sexual behavior. Physiol. Behav. 83, 217–232 788 (2004). 789 16. McGill, T. E. Sexual Behavior in Three Inbred Strains of Mice. Behaviour 19, 341–790 350 (1962). 791 17. Redmon, J., Divvala, S., Girshick, R. & Farhadi, A. You Only Look Once: Unified, 792 Real-Time Object Detection. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. CVPR (2016) 793 doi:10.1145/3243394.3243692. 794 18. Jocher, G. YOLOv5 by Ultralytics. https://doi.org/10.5281/zenodo.3908559 (2020). 795 19. Du, J. Understanding of Object Detection Based on CNN Family and YOLO. J. Phys. 796 Conf. Ser. 1004, 012029 (2018). 797 20. A. Aziz, Z., Naseradeen Abdulqader, D., Sallow, A. B. & Khalid Omer, H. Python 798 Parallel Processing and Multiprocessing: A Rivew. Acad. J. Nawroz Univ. 10, 345 –354 799 (2021). 800 21. Amino, K. & Matsuo, T. Automated Behavior Analysis Using a YOLO-Based Object 801 Detection System. Behavioral Neurogenetics vol. 181 (2022). 802 22. Yamanouchi, H. M., Tanaka, R. & Kamikouchi, A. Event-triggered feedback system 803 using YOLO for optogenetic manipulation of neural activity. 2023 IEEE Int. Conf. Pervasive 804 Comput. Commun. Workshop Affil. Events PerCom Workshop BiRD 2023 312–315 (2023) 805 doi:10.1109/PerComWorkshops56833.2023.10150245. 806 23. Negroni, M. A. & LeBoeuf, A. C. Metabolic division of labor in social insects. Curr. 807 Opin. Insect Sci. 59, 101085 (2023). 808 24. Piqueret, B. & d’Ettorre, P. Communication in Ant Societies. in The Cambridge 809 Handbook of Animal Cognition (eds. Kaufman, A. B., Kaufman, J. C. & Call, J.) 36 –55 810 (Cambridge University Press, Cambridge, 2021). doi:10.1017/9781108564113.004. 811 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 31 25. Stednitz, S. J. et al. Forebrain Control of Behaviorally Driven Social Orienting in 812 Zebrafish. Curr. Biol. 28, 2445-2451.e3 (2018). 813 26. Hosaka, S., Hosokawa, M., Hibi, M. & Shimizu, T. The zebrafish cerebellar neural 814 circuits are involved in orienting behavior. eNeuro (2024) doi:10.1523/ENEURO.0141 -815 24.2024. 816 27. Mathis, M. W. & Mathis, A. Deep learning tools for the measurement of animal 817 behavior in neuroscience. Curr. Opin. Neurobiol. 60, 1–11 (2020). 818 28. Padilla, R., Passos, W. L., Dias, T. L. B., Netto, S. L. & Da Silva, E. A. B. A 819 comparative analysis of object detection metrics with a companion open -source toolkit. 820 Electron. Switz. 10, 1–28 (2021). 821 29. Takeuchi, R. F. et al. Posteromedial cortical networks encode visuomotor prediction 822 errors. 2022.08.16.504075 Preprint at https://doi.org/10.1101/2022.08.16.504075 (2024). 823 30. Petersen, C. C. H. Sensorimotor processing in the rodent barrel cortex. Nat. Rev. 824 Neurosci. 20, 533–546 (2019). 825 31. Warren, J. A. et al. The HIV-1 latent reservoir is largely sensitive to circulating T 826 cells. eLife 9, e57246 (2020). 827 32. Vincent, S. B. The Function of the Vibrissae in the Behavior of the White Rat. (Holt, 828 1912). 829 33. Sofroniew, N. J., Cohen, J. D., Lee, A. K. & Svoboda, K. Natural Whisker -Guided 830 Behavior by Head-Fixed Mice in Tactile Virtual Reality. J. Neurosci. 34, 9537–9550 (2014). 831 34. Lopes, G. et al. Bonsai: An event -based framework for processing and controlling 832 data streams. Front. Neuroinformatics 9, 1–14 (2015). 833 35. Ishimoto, H. & Kamikouchi, A. Molecular and neural mechanisms regulating sexual 834 motivation of virgin female Drosophila. Cell. Mol. Life Sci. 78, 4805–4819 (2021). 835 36. Ding, Y. et al. Neural Evolution of Context -Dependent Fly Song. Curr. Biol. 29, 836 1089-1099.e7 (2019). 837 37. von Philipsborn, A. C. et al. Neuronal Control of Drosophila Courtship Song. Neuron 838 69, 509–522 (2011). 839 38. Mohammad, F. et al. Optogenetic inhibition of behavior with anion 840 channelrhodopsins. Nat. Methods 14, 271–274 (2017). 841 39. Lillvis, J. L. et al. Nested neural circuits generate distinct acoustic signals during 842 Drosophila courtship. Curr. Biol. 34, 808-824.e6 (2024). 843 40. Krakauer, J. W., Ghazanfar, A. A., Gomez-Marin, A., MacIver, M. A. & Poeppel, D. 844 Neuroscience Needs Behavior: Correcting a Reductionist Bias. Neuron 93, 480–490 (2017). 845 41. Lopes, G. & Monteiro, P. New Open -Source Tools: Using Bonsai for Behavioral 846 Tracking and Closed-Loop Experiments. Front. Behav. Neurosci. 15, (2021). 847 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 32 42. Buccino, A. P. et al. Open source modules for tracking animal behavior and closed -848 loop stimulation based on Open Ephys and Bonsai. J. Neural Eng. 15, 055002 (2018). 849 43. Wu, M.-C. et al. Optogenetic control of selective neural activity in multiple freely 850 moving Drosophila adults. Proc. Natl. Acad. Sci. 111, 5367–5372 (2014). 851 44. Watanabe, Y. et al. High-speed 8-bit image projector at 1,000 fps with 3 ms delay. in 852 vol. 3 1421–1422 (2015). 853 45. Kagami, S. & Hashimoto, K. Interactive Stickies: Low -latency projection mapping 854 for dynamic interaction with projected images on a movable surface. in ACM SIGGRAPH 855 2020 Emerging Technologies 1–2 (Association for Computing Machinery, New York, NY, 856 USA, 2020). doi:10.1145/3388534.3407291. 857 46. Casado-García, A. et al. LabelStoma: A tool for stomata detection based on the 858 YOLO algorithm. Comput. Electron. Agric. 178, 105751 (2020). 859 47. Walter, T. & Couzin, I. D. Trex, a fast multi-animal tracking system with markerless 860 identi cation, and 2d estimation of posture and visual elds. eLife 10, 1–73 (2021). 861 48. Simon, J. C. & Dickinson, M. H. A New Chamber for Studying the Behavior of 862 Drosophila. PLOS ONE 5, e8793 (2010). 863 49. Terayama, M., Kubota, S. & Eguchi, K. Encyclopedia of Japanese Ants. 864 50. Friard, O. & Gamba, M. BORIS: a free, versatile open-source event-logging software 865 for video/audio coding and live observations. Methods Ecol. Evol. 7, 1325–1330 (2016). 866 51. Everingham, M. et al. The Pascal Visual Object Classes Challenge: A Retrospective. 867 Int. J. Comput. Vis. 111, 98–136 (2015). 868 52. Allen, W. E. et al. Global Representations of Goal-Directed Behavior in Distinct Cell 869 Types of Mouse Neocortex. Neuron 94, 891-907.e6 (2017). 870 53. Dolezal, D. M., Joiner, M. A. & Eberl, D. F. Two distinct functions of Lim1 in the 871 Drosophila antenna. MicroPublication Biol. (2024) doi:10.17912/micropub.biology.001229. 872 54. Yamada, X. D. et al. GABAergic local interneurons shape female fruit fly response 873 to mating songs. J. Neurosci. 38, 4329–4347 (2018). 874 55. Bogovic, J. A. et al. An unbiased template of the Drosophila brain and ventral nerve 875 cord. PLOS ONE 15, e0236495 (2020). 876 56. Avants, B. & Gee, J. C. Geodesic estimation for large deformation anatomical shape 877 averaging and interpolation. NeuroImage 23, S139–S150 (2004). 878 57. Wobbrock, J. O., Findlater, L., Gergle, D. & Higgins, J. J. The aligned rank transform 879 for nonparametric factorial analyses using only anova procedures. 143 –146 (2011) 880 doi:10.1145/1978942.1978963. 881 58. Kay, M., Elkin, L. A., Higgins, J. J. & Wobbrock, J. O. ARTool: Aligned rank 882 transform for nonparametric factorial anovas. R package version 0.11.1, 883 https://github.com/mjskay/ARTool. 594511 (2021) doi:10.5281/zenodo.594511. 884 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 33 Supplemental file legends 885 Supplementary Table 1: Summary of model accuracy compared to human annotations 886 Supplementary Table 2: Summary of model Precision and Recall in each pretrained 887 model and in each training image number 888 Supplementary Table 3: Summary of model accuracy of “mouse – treadmill” model 889 Supplementary Table 4: Summary of inference speed of one frame detection 890 Supplementary Table 5: Summary of dataset conditions 891 Supplementary Table 6: Lists of PC hardware information for benchmarking tests 892 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint Figure 1 a b Evaluation Real-time Process Video AnalysisTraining Extract frames Label frames (LabelImg) Train network (create model) GroomingRunning Behavior object Copulation Prepare ground-truth labels Calculate model accuracy Input videos Analyze videos Output results (csv and videos)Output results Input camera frames Analyze frames in real-time Process the results Output triggers (Control hardware) Single-animal behavior Multi-animal behavior Fig. 1: YORU detects animal behaviors as a behavior object. a, Illustrations of behavior objects. YORU can adapt to single-animal (Left) and multi-animal (Right) behaviors, including social behaviors. Animal behaviors (Top) can be classified as behavior objects (Bottom). b, Diagram of YORU. YORU includes four packages: Training, Evaluation, Video Analysis, and Real-time Process. .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint AP@50 AP@75 trophallaxis 0 200 400 600 800 1000 0 200 400 600 800 1000 0.0 0.2 0.4 0.6 0.8 1.0 Training images AP Value YOLOv5n YOLOv5s YOLOv5m YOLOv5l YOLOv5x AP@50 AP@75 fly wing extension 0 500 1000 1500 2000 0 500 1000 1500 2000 0.6 0.8 1.0 0.6 0.8 1.0 Training images AP Value YOLOv5n YOLOv5s YOLOv5m YOLOv5l YOLOv5x AP@50 AP@75 orientation 0 500 1000 1500 2000 0 500 1000 1500 2000 0.4 0.6 0.8 1.0 Training images AP Value YOLOv5n YOLOv5s YOLOv5m YOLOv5l YOLOv5x AP@50 AP@75 trophallaxis 0 500 1000 1500 2000 0 500 1000 1500 2000 0.4 0.6 0.8 1.0 Training images AP Value YOLOv5n YOLOv5s YOLOv5m YOLOv5l YOLOv5x a b c wing_extension fly d g e h f i no orientation no_orientation trophallaxis YORU Manual 270 275 280 285 290 YORU Manual 0 120 240 360 480 Manual YORU Time (s) Time (min) Manual YORU YORU Manual 0 5 10 15 20 25 30 YORU Manual 12 13 14 YORU Manual Definition 0 30 60 90 120 150 180 210 240 270 300 YORU Manual Definition 60 70 80 90 Definition

Manual YORU Time (s) j k l m Group flies (4 males, 4 females) AP@50 AP@75 fly wing extension copulation 0 500 1000 1500 2000 0 500 1000 1500 2000 0.6 0.8 1.0 0.6 0.8 1.0 0.6 0.8 1.0 Training images AP Value YOLOv5n YOLOv5s YOLOv5m YOLOv5l YOLOv5x Group ants (6 ants) Figure 2 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint Fig. 2: Detection of animal behaviors by YORU. a, “Fly - wing extension” dataset. Two behavior classes were defined: “wing_extension” ; a fly extending one of its wings, and “fly” ; a fly that does not belong to the “wing extension” class. b, “Ant - trophallaxis” dataset. Two behavior classes were defined: “trophallaxis” ; two ants showing nutrition exchange, and “no” ; the situation of no nutrition exchange but the heads of the two ants are close together. c, “Zebrafish - orientation” dataset. Two behavior classes were defined: “orientation” ; two zebrafish showing orientation behavior, and “no_orientation” ; zebrafish showing no orientation behavior. d-f, Ethogram of the fly wing extension (d), ant trophallaxis (e), and zebrafish orientation (f) behaviors. Top panels show the example view of each behavior. The circles show detections by manual analysis (pink), YORU analysis (blue), or analysis using a previous tracking method (green). The horizontal axes in the ethogram represent the observation period. The colored area of the ethogram shows the occurrence of the behavior detected with YORU analysis (blue), manual analysis using BORIS (pink), and the previous tracking method (green). g-i, The AP values of “Fly - wing extension” (g), “Ant - trophallaxis” (h), and “Zebrafish - orientation” (i) models. The values at IOU=50% (AP@50) (Left) and IOU=75% (AP@75) (Right) are shown. The horizontal and vertical axes show the number of images used for training and the AP values, respectively. Each colored line shows the AP values of different pre-trained models (Also in k, m). j, “Fly - group courtship” datasets. Three behavior classes were defined: “copulation” ; a pair of flies showing copulation, “wing_extension” ; a fly extending one of its wings, and “fly” ; a fly that does not belong to the “copulation” and “wing extension” classes. k, The AP values for “Fly - group courtship” models. l, “Ant - group trophallaxis” dataset. Two behavior classes were defined: “trophallaxis” ; two ants showing nutrition exchange, and “no_trophallaxis” ; the heads of two ants are close together without exchanging nutrition. .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint Running Stop Whisker-OnWhisker-OffEye-OpenedEye-ClosedGrooming-OnGrooming-Off Running Stop Whisker-On Whisker-Off Eye-Opened Eye-Closed Grooming-On Grooming-Off -1 0 1 GroomingSniffing (with “Stop”)Running Blinking -0.3 0.0 0.3 Correlation (r) Locomotion speed Running Grooming Blinking Sniffing 20 s Images Correlation: YORU-readout vs. Neural activityCorrelation of behavior Run Stop Blink Grooming YORU-readout a c e b Monitors Treadmill Virtual reality Cortex-wide Ca2+ imaging Time Visual Auditory Association Motor Somatosensory ʜ d Correlation (r) Mouse atlas (Right) Anterior Lateral ** Figure 3 Fig. 3: YORU uncovers the relationship between behavioral readouts and neural activity interpretation. a, The setup for virtual reality (VR) and cortex-wide imaging in mice. b, Time-series of locomotion speed (red, calculated from rotary encoder signals of VR) and YORU readout (black). Representative images are shown at the bottom. c, Correlation matrix between each behavior. Correlation indices are derived from the YORU readout time series data. d, Top view of the Allen Common Coordinate atlas of the dorsal cortex. Five rough divisions are auditory areas (yellow), association areas (magenta), somatosensory areas (cyan), and motor areas (blue). Black asterisks show the somatosensory areas representing forelimb and hindlimb information. e, Pseudo-colormap of Spearman's correlation coefficient (YORU-readout vs. Neural activity of each pixel). .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint 640x480 (100fps) 1280x1024 (100fps) YOLOv5n YOLOv5s YOLOv5m YOLOv5l YOLOv5x YOLOv5n YOLOv5s YOLOv5m YOLOv5l YOLOv5x 0 50 100 150 200 Duration (ms) 640x480 30fps 60fps 100fps 130fps 160fps 200fps 0 50 100 150 200 Duration (ms) a b d DAQRecording DAQ LED Camera Trigger controller YORU DAQ ON OFF c Figure 4 Fig. 4: Validation of YORU’ s operation speed. a, “LED lighting” dataset. Two state classes were defined: “ON” and “OFF” , indicating that LED light was turned on and off, respectively. b, Schematic of system latency measurements. The camera captures the LED light, YORU detects a frame, and the trigger-controller DAQ outputs the TTL voltage based on detection by YORU. The recording DAQ logged the TTL pulse from the trigger controller DAQ and the LED voltage. c, The system latency of “LED lighting - Small” (Left) and “LED lighting - Large” (Right) models. The system latency of each model was calculated using camera images with resolutions of 640x480 pixels and 1240x1024 pixels, respectively. d, The system latency at different camera frame rates. The “LED lighting - Small” model was used. c,d, Violin plot represents the probability density of individual data points within the range of possible values. .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint a b “wing extension” in results Output a trigger Behavior detection YORU (Online) DAQ Photostimulation Yes No Camera Green LED c d pIP10 split-GAL4 (male) (FOPUZQF pIP10 neurons > GtACR1 -JHIU
DPOEJUJPO 4BNQMF
TJ[F 33 27 27Event Event DelayedpIP10 neurons > GtACR1 pIP10 neurons > GFP e Event Delayed 1 sec Wing extension Photostimulation 0.0 0.2 0.4 0.6 0.8 1.0 0 5 10 15 20 25 30 Copulation Latency (minutes) Cumulative Copulation n.s. ** * n.s. * 0.0 0.2 0.4 0.6 0.8 Wing Extension Ratio TITLE Figure 5 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint Fig. 5: Neural manipulation in response to male wing extension using YORU. a, Microscopy image of pIP10 split-GAL4 expression pattern in a representative male brain and ventral nerve cord. Scale bar, 100 μm. Signals of the GFP marker (green) and counter-labeling with the nc82 antibody (magenta) are shown. b, Schematic of the YORU’ s closed-loop conditions. YORU analyzes camera frames. Then, if YORU detects “wing extension” , it sends signals to the trigger controller (DAQ) and operates the LED light. c, Schematic of light conditions. As the experimental photostimulation condition, YORU introduces green photostimulation to the entire chamber when it detects a fly showing wing extension (Event). As a photostimulation control, we used an event-triggered light with a 1-second delay that illuminates the entire chamber (Delayed). pIP10 specific split-GAL4>UAS-GtACR1 (pIP10 neurons > GtACR1) males were used for these two groups. In addition, pIP10 split-GAL4>20XUAS-IVS-mCD8::GFP (pIP10 neurons > GFP) males were used as a genetic control. The following color code was used: experimental group (blue), photostimulation control group (pink), and genetic control group (dark blue) (Also in d, e). d, Ratio of time spent displaying wing extension before copulation. The aligned rank transform one-way analysis of variance (ART one-way ANOVA) test corrected with the Benjamini-Hochberg method was used for statistical analysis. Boxplots display the medians (horizontal white line in each box) with 25th and 75th percentiles and whiskers denote 1.5x the interquartile range. Each point indicates individual data. e, Cumulative copulation rate of pIP10 split-GAL4>GtACR1 males. Pairwise comparisons using Log-Rank test corrected with the Benjamini-Hochberg method were used for statistical analysis. d,e, Not significant (n.s.), p > 0.05; *, p < 0.05. .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint a b f ~20 mm 4 mm IR camera Chamber IR LED Light IR Pass Filter Projector ~12 mm “wing extension” in results Draw a green circle Select the “fly” box Behavior detection YORU (Online) Projecter Photostimulation Yes No Camera (FOPUZQF JO15>GtACR1 JO15>GtACR1 +>GtACR1 -JHIU
DPOEJUJPO 4BNQMF
TJ[F 34 32 28Event-triggered Event-triggered Pattern c d Female head JO15-2-GAL4 > 20xUAS-IVS-mCD8::GFP GFP e Event-triggered Pattern 3 sec Wing extension Photostimulation 4 sec Photostimulation Event-triggered Pattern *** n.s. 0.0 0.2 0.4 0.6 0.8 1.0 0 5 10 15 20 25 30 Copulation latency (minutes) Cumulative Copulation *** 0.00 s 0.10 s 0.23 s D V Figure 6 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint Fig. 6: Individual-specific neural manipulation by YORU. a, Experimental setup of individual-focused photostimulation. An infrared (IR) camera is used for observing flies. IR LED light is used for IR camera recording. IR pass filter allows only IR light to be captured by the IR camera, preventing visible light noise. The projector is used for introducing individual-focused photostimulation. b, Schematic of the closed-loop system for individual-focused photostimulation assay. YORU analyzes camera frames. If YORU detects “wing extension” , it draws a green circle on the “fly” bounding box and sends the image to the projector. The projector introduces green-circled photostimulation to the fly. c, A representative situation during individual-focused photostimulation. White arrowheads show the fly displaying wing extension. d, JO15-2-GAL4 expression in female antenna. GFP markers driven by JO15-2-GAL4 (JO15-2-GAL4>20XUAS-IVS-mCD8::GFP) are detected in Johnstons Organs. White arrowheads show JO neurons. D and V indicate the dorsal and ventral sides, respectively. Scale bar, 100μm. e, Schematic of light conditions. In the experimental photostimulation condition, when YORU detects a fly showing wing extension, YORU introduces individual-focused photostimulation to the other fly (Event-triggered). In the photostimulation control, we used pattern light (3s On, 4s Off) to illuminate the entire chamber irrespective of the wing extension event (Pattern). f, Cumulative copulation rate of JO15-2>GtACR1 females. We used JO15-2-GAL4>UAS-GtACR1 (JO15-2>GtACR1) females for the experimental group. As a genetic control, we used +>UAS- GtACR1 (+>GtACR1) females. The following color code was used: experimental group (blue), photostimulation control group (pink), and genetic control group (dark blue). Pairwise comparisons using Log-Rank test corrected with the Benjamini-Hochberg method used for statistical analysis. Not significant (n.s.), p > 0.05; ***, p < 0.001. .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted November 14, 2024. ; https://doi.org/10.1101/2024.11.12.623320doi: bioRxiv preprint