OCTRON - a general purpose segmentation and tracking pipeline for behavioral experiments

OCTRON is a pipeline for markerless segmentation and tracking of animals in behavioral experiments. By combining Segment Anything Models (SAM 2) for rapid annotation, YOLO11 models for training, and state-of-the-art multi-object trackers, OCTRON enables unsupervised segmentation and tracking of multiple animals with complex, deformable body plans. We validate its versatility across species - from transparent marine annelids to camouflaging cuttlefish - demonstrating robust, general-purpose applicability for behavioral analysis. Main text Quantifying animal behavior has traditionally relied on tracking an animal’s position and pose, typically through markerless detection of key points on the body. While such models, like DeepLabCut (Mathis et al. 2018; Lauer et al. 2021) and SLEAP (Pereira et al. 2022) achieve excellent performance across species and experimental contexts, their reliance on fixed anchor points makes them less suitable for the analysis of organisms with highly deformable body plans, such as hydra or cephalopods. In contrast, segmentation provides a richer representation than sparse key points, by capturing the full shape and spatial extent of the animal. This approach also enables precise quantification of full-body deformations while simultaneously generating a mask of the tracked body region. That mask can then be leveraged to extract additional features of the body itself, unlocking new levels of detail in behavioral analyses. To enable generalised tracking of the vast diversity of morphable, as well as non-morphable, organisms, we created OCTRON: A complete pipeline for the markerless segmentation and tracking of animals in behavioral experiments that is openly available and has a user-friendly graphical user interface. OCTRON is built as a napari (Napari: Napari: A Fast, Interactive, Multi-Dimensional Image Viewer for Python, n.d.) plugin and utilizes state-of-the-art segmentation models (Segment Anything Model 2 [SAM 2]) (Ravi et al. 2024) to rapidly collect training data across video frames. These training data can then be used to train custom instance segmentation models (using YOLO11, Ultralytics (Jocher and Qiu 2024)) for fast and unsupervised segmentation of animals in behavioral videos (Fig. 1a). In addition, state-of-the-art multi-object trackers that support both position-only tracking and more advanced ReID (Re-Identification) models (Broström, n.d.; Zhang et al. 2021; Aharon et al. 2022; Cao et al. 2022; Yang et al. 2023; Du et al. 2022; Maggiolino et al. 2023) are integrated. At each stage - annotation, training, and tracking - OCTRON offers users multiple models as plug-and-play 2 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint solutions, ensuring coverage of a wide range of tracking scenarios. While OCTRON is not the first pipeline to enable segmentation or tracking in soft-bodied animals (Romero-Ferrero et al. 2019; Allan et al. 2025; Hebert et al. 2021; Banerjee et al. 2023), it is the first fully trainable pipeline for multi-instance segmentation tracking that is generally applicable across experimental setups and species. Comprehensive documentation and best practices for the pipeline and library are available on github under https://github.com/OCTRON-tracking and our documentation website https://octron-tracking.github.io/OCTRON-docs. We here demonstrate the versatility of this pipeline by using it to segment and track a number of non-model organisms whose behaviour is challenging to analyze with current methods, including camouflaging cuttlefish, octopus paralarvae, transparent marine annelids and ctenophores, as well as highly morphable hydra (Fig. 1b,c). We first loaded videos of each species in OCTRON. Next, we created labels for the objects to be tracked and selected these objects in one or a few frames with single clicks or user-drawn outlines. This rapidly creates a video-specific memory in SAM 2 that can be used to perform unsupervised prediction on subsequent frames in that same video (Fig. 1b). With this forward prediction, we could annotate hundreds of frames within minutes to yield large training datasets, with the possibility to refine masks if necessary. SAM 2-assisted annotation reliably segmented regions of interest after just a few mouse clicks (Fig. 1c). These individual training datasets were then used to train YOLO segmentation models (YOLO11m-seg or YOLO11l-seg). By applying the trained model on (non-trained) test data, we found that OCTRON reliably segments and tracks individuals from all six tested species (Ascarosepion bandense, Hydra vulgaris, Mnemiopsis leidyi, Octopus vulgaris, Sepia officinalis, and Tomopteris helgolandica) during behavior (two are shown in Fig. 1d). We then compared OCTRON-extracted masks with those created manually by human observers (ground truth annotations). When benchmarked across varying training dataset sizes, OCTRON-trained models converged rapidly and exhibited stronger alignment with the consensus of five annotators than with any single annotator (Fig.  1e,f and Extended Data Fig. 1), ultimately surpassing human performance. To test instance segmentation in conjunction with multi-object tracking, we segmented and tracked the head and mantle of three octopus paralarvae (Octopus vulgaris) swimming in the same arena (Fig. 2a). The X and Y positions of one individual’s head were strongly correlated with those of its mantle (Fig.  2b) and significantly more so than with the head-mantle tracks of other individuals (Fig.  2c). Thus confirming reliable tracking without ID switches, even when 3 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint animals crossed paths multiple times (Supplemental Video 1). Further, mask time series of mantle and head capture information about both shape and size, and when analyzed using principal component (PC) analysis, they can reveal distinct body shapes. We found that in our videos of swimming octopus paralarvae, shape PCs reflected posture changes, suggesting discrete behavioral states (Fig.  2d). While frames sorted by PC1 scores distinguished gross changes in animals’ alignment to the camera, frames sorted by PC2 scores distinguished between a more compact body shape and an elongated one, the latter indicative of a streamlined configuration during swimming bouts when the animals propel themselves forward. Indeed, a correlation with speed confirmed that the elongated body form (low PC2 score) corresponded to higher swimming speeds (Fig. 2e). Combined, we show that access to segmentation unlocks two complementary strengths: robust multi-object tracking for reliable identity preservation and rich shape information for high-resolution behavioral and morphological analyses. Not all animals have body features that make their orientation or posture easy to infer. Some, such as ctenophores (Fig.  1c), lack distinct ‘head’ and ‘tail’ regions, while in certain annelids (Fig. 2f), the head is so small relative to the body that achieving the image resolution needed for reliable detection is impractical. However, the segmentation masks produced by OCTRON enable keypoint-like extraction of distinct body points during secondary analysis. For example, after skeletonizing masks of a pelagic annelid (Tomopteris helgolandica), the centerline endpoints can be classified as ‘head’ or ‘tail’ based on the median medial-axis thickness of each endpoint (Fig.  2g), since the head region is rounder and bulkier than the tail. Similarly, access to the full shape of the animal allows assigning head and tail points with a confidence score derived from the observed head-to-tail asymmetry (Fig. 2g). When swimming, T. helgolandica exhibits lateral body undulations (Gouveneaux et al. 2018), which results in a steady head scanning motion (Fig. 2i, Supplemental Video 2). By extracting points at fixed fractions along the centerline to define head angle relative to body angle, we quantified this motion across five animals, revealing a stable peak near 2  Hz (Fig. 2j). This is in line with known kinematics for this species (Daniels et al. 2021) and might have important implications for how they process sensory stimuli during locomotion. Thus, in addition to analysis of full body changes, the rich shape information provided by OCTRON also enables secondary keypoints-like analysis of behavior. 4 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint Finally, we evaluated OCTRON on an exceptionally challenging segmentation task: camouflaging Sepia officinalis cuttlefish. Individual animals were placed in an open, circular tank divided into three equally-sized areas that contained images of coarse, medium, and fine sand on its floor and walls. This arena type is designed to evoke distinct camouflage body patterns (disruptive, mottle, and uniform, respectively; Fig.  3a,b; Extended Data Fig. 2a,b) (Barbosa et al. 2008; Allen et al. 2010; Chiao et al. 2005; Hanlon et al. 2009; Chiao et al. 2015). We found that our trained model was able to reliably extract head and mantle regions across frames spanning a range of positions and skin patterns (Fig. 3b; Extended Data Fig. 2b, Supplemental Video 3). By segmenting out background (Extended Data Fig. 2c) we were able to precisely align the tank regions and animal position data across videos (Extended Data Fig. 2d). We next used the extracted head and mantle masks to derotate cuttlefish across all frames (Fig. 3c). A subset of those rotated and cropped cuttlefish frames were used to train a convolutional autoencoder (Extended Data Fig. 3a) to extract 32 latent embeddings, which were reduced to two dimensions via principal component analysis (PCA) for visualization and clustering. Unsupervised clustering with HDBSCAN assigned cluster labels to all frames (Extended Data Fig. 3b, c). These skin pattern clusters were then manually grouped into two classes: one combining predominantly mottle patterns with those extending into uniform patterns (merged due to the limited expression of purely uniform patterns in our experiment; Extended Data Fig. 3c) and a second class capturing a variety of disruptive patterns (Fig. 3d; Extended Data Fig. 3c). By combining pattern class information (Fig.  3d) with position and head-direction data (Fig. 3e, top), we could precisely map the spatial occurrence of the two pattern classes and normalize by occupancy across the environment (Fig.  3e, bottom). As predicted, we found that mottle/uniform camouflage patterns were typically expressed in more uniformly patterned sectors of the arena (fine and medium sand), whereas a disruptive pattern appeared preferentially over the coarse background (Fig.  3f). OCTRON thus enables quantitative analysis of body patterns in addition to position, revealing behavioral insights inaccessible to location-only methods. By combining powerful, state-of-the-art tools for the rapid annotation, segmentation, and multi-object tracking of animals, as well as visualization and export possibilities in a user-friendly graphical user interface, OCTRON provides a complete pipeline for markerless segmentation and tracking. The rich information supplied by the generated masks can benefit the behavioral analysis of all species, but are particularly well suited for animals that have complex, deformable body plans or blend into the background. While segmentation masks provide valuable 5 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint information about body shape and can be used to generate keypoints-like skeletons, they make analyzing complex poses - such as joint position - challenging. Such analyses are still best achieved using keypoint-based methods. Importantly, the YOLO models that OCTRON uses already support keypoint tracking in principle, features we plan to integrate in the near future. Looking ahead, we also envision the pipeline supporting multi-camera angle data, enabling 3D reconstruction of body shapes and movements and thereby bringing volumetric behavioral analysis within reach for animals with malleable bodies. 6 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint

Agassiz. 1865. “On the Metamorphoses Undergone by Certain Fishes before Acquiring the Adult Form.” The Annals & Magazine of Natural History 16 (91): 69–70. https://doi.org/10.1080/00222936508679376. Aharon, Nir, Roy Orfaig, and Ben-Zion Bobrovsky. 2022. “BoT-SORT: Robust Associations Multi-Pedestrian Tracking.” In arXiv [cs.CV]. June 29. arXiv. https://doi.org/10.48550/arXiv.2206.14651. Allan, Daniel B., Thomas Caswell, Nathan C. Keim, Casper M. van der Wel, and Ruben W. Verweij. 2025. Soft-Matter/trackpy: v0.7. Zenodo. https://doi.org/10.5281/ZENODO.16089574. Allen, Justine J., Lydia M. Mäthger, Alexandra Barbosa, et al. 2010. “Cuttlefish Dynamic Camouflage: Responses to Substrate Choice and Integration of Multiple Visual Cues.” Proceedings. Biological Sciences / The Royal Society 277 (1684): 1031–1039. https://doi.org/10.1098/rspb.2009.1694. Banerjee, Shoubhik Chandan, Khursheed Ahmad Khan, and Rati Sharma. 2023. “Deep-Worm-Tracker: Deep Learning Methods for Accurate Detection and Tracking for Behavioral Studies in C. Elegans.” Applied Animal Behaviour Science 266 (106024): 106024. https://doi.org/10.1016/j.applanim.2023.106024. Barbosa, Alexandra, Lydia M. Mäthger, Kendra C. Buresch, et al. 2008. “Cuttlefish Camouflage: The Effects of Substrate Contrast and Size in Evoking Uniform, Mottle or Disruptive Body Patterns.” Vision Research 48 (10): 1242–1253. https://doi.org/10.1016/j.visres.2008.02.011. Broström, Mikel. n.d. BOXMOT. Version 15.0.0. https://github.com/mikel-brostrom/boxmot. Cao, Jinkun, Jiangmiao Pang, Xinshuo Weng, Rawal Khirodkar, and Kris Kitani. 2022. “Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking.” In arXiv [cs.CV]. March 27. arXiv. https://doi.org/10.48550/arXiv.2203.14360. Chiao, Chuan-Chin, Charles Chubb, and Roger T. Hanlon. 2015. “A Review of Visual Perception Mechanisms That Regulate Rapid Adaptive Camouflage in Cuttlefish.” Journal of Comparative Physiology. A, Neuroethology, Sensory, Neural, and Behavioral Physiology 201 (9): 933–945. https://doi.org/10.1007/s00359-015-0988-5. Chiao, Chuan-Chin, Emma J. Kelman, and Roger T. Hanlon. 2005. “Disruptive Body Patterning of Cuttlefish (Sepia Officinalis) Requires Visual Information Regarding Edges and Contrast of Objects in Natural Substrate Backgrounds.” The Biological Bulletin 208 (1): 7–11. https://doi.org/10.2307/3593095. Daniels, Joost, Nadège Aoki, Josh Havassy, Kakani Katija, and Karen J. Osborn. 2021. “Metachronal Swimming with Flexible Legs: A Kinematics Analysis of the Midwater Polychaete Tomopteris.” Integrative and Comparative Biology 61 (5): 1658–1673. https://doi.org/10.1093/icb/icab059. Donoghue, Thomas, Matar Haller, Erik J. Peterson, et al. 2020. “Parameterizing Neural Power 7 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint Spectra into Periodic and Aperiodic Components.” Nature Neuroscience 23 (12): 1655–1665. https://doi.org/10.1038/s41593-020-00744-x. Du, Yunhao, Zhicheng Zhao, Yang Song, et al. 2022. “StrongSORT: Make DeepSORT Great Again.” In arXiv [cs.CV]. February 27. arXiv. https://doi.org/10.48550/arXiv.2202.13514. Gouveneaux, Anaïd, Marie-Charlotte Gielen, and Jérôme Mallefet. 2018. “Behavioural Responses of the Yellow Emitting Annelid Tomopteris Helgolandica to Photic Stimuli.” Luminescence: The Journal of Biological and Chemical Luminescence 33 (3): 511–520. https://doi.org/10.1002/bio.3440. Hanlon, R. T., C-C Chiao, L. M. Mäthger, A. Barbosa, K. C. Buresch, and C. Chubb. 2009. “Cephalopod Dynamic Camouflage: Bridging the Continuum between Background Matching and Disruptive Coloration.” Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 364 (1516): 429–437. https://doi.org/10.1098/rstb.2008.0270. Han, Shuting, Ekaterina Taralova, Christophe Dupre, and Rafael Yuste. 2018. “Comprehensive Machine Learning Analysis of Hydra Behavior Reveals a Stable Basal Behavioral Repertoire.” eLife 7 (March). https://doi.org/10.7554/eLife.32605. Harris, Charles R., K. Jarrod Millman, Stéfan J. van der Walt, et al. 2020. “Array Programming with NumPy.” Nature 585 (7825): 357–362. https://doi.org/10.1038/s41586-020-2649-2. Hebert, Laetitia, Tosif Ahamed, Antonio C. Costa, Liam O’Shaughnessy, and Greg J. Stephens. 2021. “WormPose: Image Synthesis and Convolutional Networks for Pose Estimation in C. Elegans.” PLoS Computational Biology 17 (4): e1008914. https://doi.org/10.1371/journal.pcbi.1008914. Hunter, J. D. 2007. “Matplotlib: A 2D Graphics Environment.” Computing in Science & Engineering 9 (3): 90–95. https://doi.org/10.1109/MCSE.2007.55. Jocher, Glenn, and Jing Qiu. 2024. Ultralytics YOLO11. Version 11.0.0. https://github.com/ultralytics/ultralytics. Ke, Lei, Mingqiao Ye, Martin Danelljan, et al. 2023. “Segment Anything in High Quality.” In arXiv [cs.CV]. June 2. arXiv. https://doi.org/10.48550/arXiv.2306.01567. Lauer, Jessy, Mu Zhou, Shaokai Ye, et al. 2021. “Multi-Animal Pose Estimation and Tracking with DeepLabCut.” In bioRxiv. April 30. https://doi.org/10.1101/2021.04.30.442096. Lopes, Gonçalo, Niccolò Bonacchi, João Frazão, et al. 2015. “Bonsai: An Event-Based Framework for Processing and Controlling Data Streams.” Frontiers in Neuroinformatics 9 (April): 7. https://doi.org/10.3389/fninf.2015.00007. Maggiolino, Gerard, Adnan Ahmad, Jinkun Cao, and Kris Kitani. 2023. “Deep OC-SORT: Multi-Pedestrian Tracking by Adaptive Re-Identification.” In arXiv [cs.CV]. February 23. arXiv. https://doi.org/10.48550/arXiv.2302.11813. Martín Abadi, Ashish Agarwal, Paul Barham, et al. 2015. “TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems.” Preprint. https://www.tensorflow.org/. Mathis, Alexander, Pranav Mamidanna, Kevin M. Cury, et al. 2018. “DeepLabCut: Markerless Pose Estimation of User-Defined Body Parts with Deep Learning.” Nature Neuroscience, 8 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint August 20, 1. https://doi.org/10.1038/s41593-018-0209-y. McInnes, Leland, John Healy, and Steve Astels. 2017. “Hdbscan: Hierarchical Density Based Clustering.” Journal of Open Source Software 2 (11): 205. https://doi.org/10.21105/joss.00205. Napari: Napari: A Fast, Interactive, Multi-Dimensional Image Viewer for Python. n.d. Github. Accessed December 10, 2025. https://github.com/napari/napari?tab=readme-ov-file. Pedregosa, Fabian, Gaël Varoquaux, Alexandre Gramfort, et al. 2011. “Scikit-Learn: Machine Learning in Python.” Journal of Machine Learning Research: JMLR 12 (85): 2825–2830. http://jmlr.org/papers/v12/pedregosa11a.html. Pereira, Talmo D., Nathaniel Tabris, Arie Matsliah, et al. 2022. “SLEAP: A Deep Learning System for Multi-Animal Pose Tracking.” Nature Methods 19 (4): 486–495. https://doi.org/10.1038/s41592-022-01426-1. Ravi, Nikhila, Valentin Gabeur, Yuan-Ting Hu, et al. 2024. “SAM 2: Segment Anything in Images and Videos.” In arXiv [cs.CV]. August 1. arXiv. https://doi.org/10.48550/arXiv.2408.00714. Romero-Ferrero, Francisco, Mattia G. Bergomi, Robert C. Hinz, Francisco J. H. Heras, and Gonzalo G. de Polavieja. 2019. “Idtracker.Ai: Tracking All Individuals in Small or Large Collectives of Unmarked Animals.” Nature Methods 16 (2): 179–182. https://doi.org/10.1038/s41592-018-0295-5. Stanojević, Vukašin, and Branimir Todorović. 2025. “BoostTrack++: Using Tracklet Information to Detect More Objects in Multiple Object Tracking.” In arXiv [cs.CV]. August 20. arXiv. https://doi.org/10.48550/arXiv.2408.13003. Team, The Pandas Development. 2020. Pandas-Dev/pandas: Pandas. Version latest. Zenodo. https://doi.org/10.5281/zenodo.3509134. Velden, Ellert van der. 2020. “CMasher: Scientific Colormaps for Making Accessible, Informative and ‘Cmashing’ Plots.” Journal of Open Source Software 5 (46): 2004. https://doi.org/10.21105/joss.02004. Virtanen, Pauli, Ralf Gommers, Travis E. Oliphant, et al. 2020. “SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.” Nature Methods 17 (3): 261–272. https://doi.org/10.1038/s41592-019-0686-2. Walt, Stéfan van der, Johannes L. Schönberger, Juan Nunez-Iglesias, et al. 2014. “Scikit-Image: Image Processing in Python.” PeerJ 2 (June): e453. https://doi.org/10.7717/peerj.453. Waskom, Michael L. 2021. “Seaborn: Statistical Data Visualization.” Journal of Open Source Software 6 (60): 3021. https://doi.org/10.21105/joss.03021. Yang, Mingzhan, Guangxin Han, Bin Yan, et al. 2023. “Hybrid-SORT: Weak Cues Matter for Online Multi-Object Tracking.” In arXiv [cs.CV]. August 1. arXiv. https://doi.org/10.48550/arXiv.2308.00783. Zhang, Yifu, Peize Sun, Yi Jiang, et al. 2021. “ByteTrack: Multi-Object Tracking by Associating Every Detection Box.” In arXiv [cs.CV]. October 13. arXiv. https://doi.org/10.48550/arXiv.2110.06864. 9 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint

Pipeline OCTRON consists of several modules, each of which harnesses powerful openly available tools. The implementation of these tools in each step of the OCTRON pipeline is described below. Project organization OCTRON operates on data contained within a user‑defined project folder that consolidates all annotation and training data for a given project. The workflow begins with the user selecting a video for annotation, which can optionally be re‑encoded with ffmpeg (https://www.ffmpeg.org) before being imported into OCTRON. During the import process, OCTRON creates a dedicated subfolder for the video inside the project directory, named according to a file hash of the video itself. This mechanism prevents mix‑ups and safeguards data integrity, even if a newer video with the same filename unintentionally replaces an older one in the project folder. Annotation subfolders from multiple users who have annotated videos on their own machines can be incorporated into the project. This is achieved simply by copying and pasting the annotation folders along with their corresponding video files into the original project directory. This approach allows larger projects to be easily distributed across multiple users and later consolidated for the training stage (see below). Annotation Module OCTRON uses SAM 2 models for semi-automated annotation of video frames. Users select from three model sizes (SAM 2 Base Plus (Ravi et al. 2024), SAM 2 Large (Ravi et al. 2024), SAM 2 HQ (Ke et al. 2023)) based on available GPU resources, with larger models providing greater precision. Annotation is performed by one of two methods: point-based prompts, where users left-click to include regions and right-click to exclude them, and shape-based prompts, where users draw outlines (rectangles, circles, or polygons) around objects of interest. SAM 2 generates region predictions (masks) in real-time based on user input, which are automatically saved to disk. Batch (forward) prediction functionality extends single-frame annotations across subsequent frames, with adjustable skip intervals to accelerate annotation of static or slowly-changing subjects. The annotation GUI organizes labels through a layer manager, where each label is associated with a point/shape layer for user input and a mask layer displaying SAM 2's predictions. Annotations are stored as compressed zarr archives (python zarr ≥ 3.0, 10 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint https://github.com/zarr-developers/zarr-python), and label and video metadata are organized in JSON format. Users can reopen previously annotated videos and continue working on them. Also, SAM 2's memory can be reset if batch predictions slow down due to accumulated frame context. Model Training Module Training data generation splits annotated frames into training and testing subsets, with optional pruning to exclude frames where not all labeled objects are present and watershedding to separate touching instances of the same object class. OCTRON trains YOLO11 instance segmentation models (Jocher and Qiu 2024) in two size variants: YOLO11m-seg (medium, recommended for initial training), YOLO11l-seg (large, for increased accuracy with sufficient training data). Training parameters include image size (640 or 1024 pixels, selected based on native video resolution and desired level of detail), number of epochs (with automatic early stopping after 100 epochs without improvement), and save period (checkpoint frequency). Training progress is monitored via terminal output or optionally through TensorBoard (Martín Abadi et al. 2015) for real-time metric visualization. Evaluation metrics include mean Average Precision (mAP) at Intersection over Union (IoU) thresholds of 0.5 and 0.5:0.95, precision-recall curves, confusion matrices, and validation batch predictions. All training outputs, including model weights (epoch checkpoints [.pt], best.pt, and last.pt), results.csv, and diagnostic images (results.png, confusion_matrix.png, val_batchX_pred.jpg), are saved in the project folder's model/training subdirectory. Video Analysis Module The analysis module applies a user-trained YOLO11 model to new videos using BoxMOT (Broström, n.d.) implemented tracking algorithms. Six tracking methods are available: motion-only trackers (ByteTrack (Zhang et al. 2021) and OcSort (Cao et al. 2022), computationally efficient but limited ability to track similar-looking objects) and motion+ReID trackers (BotSort (Aharon et al. 2022), DeepOCSORT (Maggiolino et al. 2023), HybridSort (Yang et al. 2023), and BoostTrack (Stanojević and Todorović 2025), which incorporate visual feature matching for robust identity preservation during occlusions and interactions). Users can tune tracker parameters, enable single-subject mode (bypassing multi-object tracking), and activate detailed mode to enable the extraction of additional morphological features (via skimage.measure.regionprops (van der Walt et al. 2014)). Analysis parameters include morphological opening (disk radius for noise reduction), confidence threshold (0 to 1, filtering 11 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint out low-confidence detections), intersection-over-union (IoU) threshold (controlling overlapping mask separation), and frame skip interval (for accelerated processing of videos that, for example, contain little changes over time). Output data is organized per video in an octron_predictions folder, with each analyzed video generating a subfolder containing per-track-ID CSV files and zarr-compressed mask archives. CSV files include frame indices, track IDs, label names, confidence scores, bounding box coordinates, and (in detailed mode) mask area, centroid position, eccentricity, solidity, and orientation derived from region properties. Prediction results can be visualized directly in napari with adjustable trajectory length and colormaps. File System Organization The project folder contains annotation subfolders (one per video, named by abbreviated video file hash) with label-specific zarr archives (label_name masks.zarr), video frame data (video data.zarr), and metadata (object_organizer.json, video_info.txt). Hash-based naming ensures video-annotation correspondence, preventing loading of video data that does not match the original annotation data exactly. Training generates a model subfolder containing training_data (train/test/val splits plus yolo_config.yaml), and a training subfolder populated during training with weights (best.pt, last.pt, epoch checkpoints), metrics (results.csv), and evaluation plots. Analysis outputs are stored in an octron_predictions folder within the video directory, structured as video_file_tracker/label_track_id.csv plus predictions.zarr and predictions_metadata.json. All data uses zarr ≥ 3.0 for compressed storage of masks and frame data, with CSV files providing tabular feature extraction compatible with pandas or other analysis libraries. Accuracy evaluation To generate an independent ground truth dataset for model evaluation, random frames were extracted as follows; For every species-specific project (videos of Ascarosepion bandense, Sepia officinalis, Mnemiopsis leidyi, Hydra vulgaris, Octopus vulgaris, Tomopteris helgolandica), 25 frames were randomly sampled without replacement from each of several videos and loaded into a stack. Ground truth annotations were generated by manual annotation (labels layer, brush tool) of every frame in the stack in napari. Multiple annotators (n=5) independently painted binary masks for each label. 12 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint To assess model performance as a function of training data quantity, multiple YOLO11m models were trained on subsets of the full training set. The complete training dataset was split into train/test/val partitions using fixed fractions (70%/15%/15%). From the training partition, five subset sizes were randomly sampled: n=60, 120, 240, 480, and 960 frames (each maintaining class balance). Models were trained for 350 epochs with maximum input image size of 640 pixels in one dimension, and cosine learning rate scheduling (initial lr=0.01). Mosaic augmentation was applied with probability 0.25 and disabled during the final 10 epochs to stabilize training. Data augmentation was applied within YOLO11 and comprised: (1) Color space jitter: HSV shifts with ranges ±0.25 for hue, saturation, and value channels; (2) Geometric transforms: random rotation ±180°, translation ±10% of image dimensions, scaling ±25%, shear ±2°, no perspective distortion, horizontal and vertical flips each with probability 0.5; (3) Advanced augmentations: mosaic (probability 0.25, combining 4 images), mixup (probability 0.25, alpha-blending two images), copy-paste (probability 0.25 in mixup mode, pasting objects from one image onto another). Masks were generated with overlap_mask=true to preserve overlapping instances and mask_ratio=2 for higher-resolution mask outputs during training (default is 4). Model checkpoints were saved every 10 epochs, with best-performing models retained for evaluation. Model predictions were generated for the human-annotated test frames using each trained model. Frames were passed through YOLO segmentation with confidence threshold 0.5. For frames containing multiple detections of the same class, masks were combined via logical OR to produce a single unified mask per class, and confidence scores were averaged. Intersection over Union (IoU) was computed as the ratio of overlapping pixels to total pixels covered by either mask: IoU = |A ∩ B| / |A ∪ B|, where A is the predicted mask and B is the human-annotated mask. IoU was calculated frame-by-frame for each annotator separately. Additionally, an "average human annotation" was created by averaging binary masks across all annotators and thresholding at 0.5. Model predictions were compared against this consensus annotation to obtain IoU relative to collective human agreement. Frames lacking predictions for a given class were assigned IoU = NaN. IoU distributions were summarized as mean ± standard deviation across frames for each training set size and label class (Fig. 1f, Extended Data Fig. 1). To compare model performance against single-annotator versus consensus annotations, independent-samples t-tests were used to assess significance (alpha=0.05). 13 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint Animals To demonstrate the versatility of OCTRON we curated videos from our own work and one openly available dataset representing a variety of species as described below in alphabetical order. Ascarosepion bandense Juvenile (3-4 months old) Ascarosepion bandense cuttlefish, also known as dwarf or stumpy-spined cuttlefish, were sourced from the Cephalopod Initiative breeding program at the Marine Biological Laboratory (MBL), Woods Hole, MA. In the absence of nationwide regulations for coleoid cephalopods, all procedures followed MBL’s Institutional Animal Care and Use Committee (IACUC) approved protocols under veterinary supervision. Stress was minimized throughout experiments, and animal health was monitored daily; any signs of stress (inking, excessive movement) paused experiments until recovery. Cuttlefish were housed individually in white, food-grade fiberglass pans (McMaster-Carr 4727T76-4727T761) divided into three compartments with opaque corrugated plastic sheets, within a closed-loop seawater system (“A Frame”) with a >30-gallon sump supplying all tanks. Salinity (31–33 ppt) was checked daily, and evaporative loss was compensated with filtered, desalinated water. The animals were fed twice daily with shrimp (Palaemonetes spp.). Behavior was recorded using overhead cameras (Basler acA2040-55uc, Thorlabs MVL25M1 lenses) mounted above enclosures at 20 fps (1544 × 1544 px). Frames and environmental data (temperature, brightness) were time-stamped via Bonsai (https://bonsai-rx.org; (Lopes et al. 2015)). Hydra vulgaris Dataset collected and published by (Han et al. 2018) and openly available on Dryad (https://datadryad.org/dataset/doi:10.5061/dryad.f6v067r). The animals were imaged from above at 5 fps with a sCMOS camera attached to a dissecting microscope. Mnemiopsis leidyi Cydippid-stage Mnemiopsis leidyi (Agassiz 1865) from the F71 laboratory strain (a gift from Dr. William Browne, University of Miami) were reared at the University of California, Berkeley. Animals were maintained in artificial seawater (Instant Ocean dissolved in distilled water, adjusted to a salinity of 26 ± 2 ppt) and fed a diet of S-type rotifers and tiger pods (Tigriopus californicus) until they reached the lobate (adult) stage, approximately 1 cm in length. Once the 14 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint animals were the desired size, they were placed in a tightly sealed 35 mm petri dish connected to a syringe filled with artificial seawater. Videos were taken with a Sony HDR-CX900 camera at 30 fps for 7 min. Video contrast was enhanced using Adobe Premiere Pro. Octopus vulgaris Embryos of Octopus vulgaris were obtained from Eduardo Almansa (CSIC, Instituto Español de Oceanografía, Tenerife, Spain). Upon arrival in Belgium, they were maintained in a closed standalone system at the Laboratory of Developmental Neurobiology, KU Leuven, where subsequent experiments were conducted. All animals used in this study originated from a single spawning event in November 2022. Following hatching, paralarvae were kept unfed in beakers under low-light conditions. For behavioral recordings, one to three paralarvae (~2 mm mantle length) were placed in a transparent 3 cm diameter Petri dish positioned on a custom-made adjustable transparent stage (Thorlabs). The stage was uniformly illuminated from below using a red light panel (ELF-200X200RD, Vision Light Tech). Recordings were captured from above using an enclosed monochrome CMOS camera with a trigger (PL-D734MU-NIR-T, Pixelink), equipped with a modular lens system (Navitar). The setup was enclosed by black curtains to eliminate ambient light interference. A red light filter was applied during acquisition, resulting in grayscale video recordings. Videos were recorded at a resolution of 2048 × 2048 pixels and a frame rate of 30.00 frames per second using Pixelink Capture software. All procedures involving paralarvae were approved by the KU Leuven Ethical Committee for Animal Experimentation (permit P099/2021) and conducted in accordance with Directive 2010/63/EU. OCTRON analyses We trained the YOLO “L” model (YOLO11l-seg) in OCTRON on 7425 annotated images across a total of 28 videos of swimming Octopus vulgaris paralarvae. For multi-animal tracking we used HybridSort in combination with a “osnet_x1_0_market1501.pt” for Re-ID. For the trajectory plots in Fig. 2a we extracted spatial paths of identified mantle-head pairs (those with high correlation over the whole video) over a specified history window (500 frames, ~16.7 seconds) ending 10 frames before the displayed frame. Background images were generated from single video frames with a circular mask applied to isolate the arena region. Trajectories were rendered using matplotlib (Hunter 2007) LineCollection with line width proportional to mantle area (normalized to 99th percentile, scaled ranging 0.5-25 pixels) and color representing instantaneous speed. Line segment opacity increased linearly from 0 to 1 across the history 15 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint window to emphasize recent movement. Position and area values were smoothed with Gaussian filters (σ=3 for position). For individual animal visualizations (Fig. 2a, right column), cropped regions were extracted centered on the midpoint between mantle and head positions. Frames were contrast-adjusted using 0th and 99.5th percentile values as vmin/vmax. Binary masks for mantle and head were extracted from segmentation outputs, converted to polygon coordinates using the imantics library (https://github.com/jsbroks/imantics), smoothed with a Gaussian filter (σ=2), and plotted as closed contours. Direction arrows indicating head-to-mantle orientation were drawn from mantle centroid to head centroid. Mantle-Head Pairing To identify which mantle and head tracks belonged to the same individual, we computed Pearson’s correlations between all mantle-head position combinations separately for X and Y coordinates. Pairs exhibiting correlation >0.9 in both dimensions were classified as high-correlation pairs representing individual animals; all other combinations were retained as controls. Correlation matrices were visualized as heatmaps with high-correlation pairs positioned on the diagonal (Fig. 2b). Position correlation values (mean of X and Y correlations) were pooled across three separate datasets with video lengths: 1:39 min, 5 min and 5 min (Fig. 2c). Morphology analyses For each identified mantle-head pair, masks from both body parts were combined in each frame, rotated to a standardized "head-up" orientation using scipy.ndimage.rotate with nearest-neighbor interpolation, cropped to minimum bounding box. Up to 2500 frames per pair were processed. All rotated masks from all pairs across all datasets were flattened and combined into a single data matrix. PCA was performed using scikit-learn (Pedregosa et al. 2011). The fitted PCA transformation was then applied individually to each pair's mask set to obtain PC scores for every frame. For visualization, frames with the highest and lowest 50 PC scores along each component were averaged and differenced (Fig. 2d). Then, the mantle position was differentiated and converted to speed (pixels/frame). PC2 scores were aligned to corresponding frames via frame index matching. Time series of speed and PC2 were smoothed with a Gaussian filter (σ=6) and plotted on dual y-axes for a representative pair (Fig. 2e, top). Pearson’s correlation between log₁₀-transformed speed and PC2 scores was computed across all frames from all pairs. A linear regression in log-speed space (y = PC2, x = log₁₀(speed)) was fitted to visualize the relationship (Fig. 2e, bottom). Scatter plots display every 15th data point 16 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint for visual clarity. Sepia officinalis Four juvenile Sepia officinalis were hatched from wild eggs collected in southern England and reared at the Marine Biological Laboratory Marine Resources Center (Woods Hole, MA, U.S.A.). At ~9 months old, each animal was housed individually in an enriched environment with 15  °C perfused seawater, following MBL’s Cephalopod Care Policy. Trials were conducted in an isolated tank containing a cylinder (diameter: 27 cm) with tri-patterned laminated substrates (fine, medium, and coarse sand) on the walls and floor. Black fabric enclosed the setup to block external stimuli. An iPhone 14 mounted above the cylinder filmed each 10-minute trial at 1080p/30 fps under ring-light illumination. OCTRON analyses We trained the YOLO “L” model (YOLO11l-seg) in OCTRON on 1158 images across a subset of 4 videos of S. officinalis. For analyses in Fig. 3, six videos of length 10 minutes of one animal were processed using OCTRON to identify the mantle and head regions. Detected regions were tracked across frames, and smoothed via Gaussian smoothing (σ = 2). For each frame, we extracted crops centered on the mantle region, rotated to align the mantle-to-head axis vertically, and registered them to a common canvas (600 × 400 pixels, Fig. 3c,d; Extended Data Fig. 3b, Extended Data Fig. 3c). The L* channel from LAB color space was used for analysis, with NaN values replaced with 50 (mid-range L* value) and normalized to [0, 1] by dividing by 100 (normalized frames) (Fig. 3d, Extended Data Fig. 3c). A convolutional autoencoder with a 32-dimensional bottleneck was trained on 1,000 randomly selected frames (70% training, 30% validation). The architecture consisted of symmetric encoder and decoder networks with convolutional layers, batch normalization, and ReLU activations. The model was optimized using Adam (lr = 2×10⁻⁴ , weight decay = 1×10⁻⁵ ) with mean squared error loss for 50 epochs (Extended Data Fig.3 a). The trained encoder was applied to extract 32 latent embeddings from all training frames. Principal component analysis (PCA) reduced embeddings to 2D for visualization. HDBSCAN (McInnes et al. 2017) (min_cluster_size = 20, min_samples = 10) was used for unsupervised clustering with approximate prediction to assign cluster labels to all frames (Extended Data Fig. 3b). Frames assigned label -1 were classified as noise. To achieve skin pattern classification across videos, for each video, cropped and masked frames (extracted every 10 frames) were preprocessed for 17 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint pattern analysis. Normalized frames were processed through the pre-trained convolutional autoencoder in batches of 32 frames. The encoder generated 32-dimensional latent embeddings for each frame. Latent embeddings were reduced to 2D using the previously fitted PCA transformer. The pre-trained HDBSCAN clusterer assigned cluster labels via approximate prediction. We then manually associated cluster labels to two pattern classes (disruptive and a combined mottle/uniform class, reflecting the absence of a sharp distinction and the weak expression of fully uniform patterns in our experiment, Extended Data Fig. 3c). For arena alignment across the six videos, video frames were analyzed to detect the circular tank boundary using ground segmentation masks from OCTRON predictions (Extended Data Fig. 2c). For each video, we fitted a circle to the arena boundary by identifying three cardinal points (transitions between fine, medium and coarse background regions) and calculating the best-fit circle through these points. To enable cross-video comparisons, we then computed affine transformation matrices that aligned each arena to a reference frame (the first video in the dataset). The reference arena's center and boundary points were used as the target, and transformations were calculated affine transformations in scikit image (van der Walt et al. 2014). An average circle was computed by transforming all individual arena circles to the reference frame and calculating the mean center and radius. Mantle and head coordinates from OCTRON predictions and centroid-based tracking were transformed to the reference frame using the pre-computed affine matrices (Extended Data Fig. 2d). The center point between mantle and head was computed as the midpoint of the two body regions and used for all subsequent position analyses. Aligned tracking data and pattern classifications were merged on frame index. Occupancy maps were computed by binning animal positions into 45 bins (in x and y) within the circular arena and smoothed with Gaussian filtering (σ = 2 bins). Occupancy was normalized by frame time (1/30 s) to yield time spent in each spatial bin (Fig. 3e). Pattern-specific tuning maps were then generated by filtering tracking data by pattern class (disruptive or mottle/uniform) and computing occupancy maps restricted to frames where each pattern was expressed. Tuning maps were normalized by the overall occupancy map to yield the probability of expressing each pattern at each spatial location. Gaussian smoothing (σ = 2 bins) was applied to reduce noise in the final tuning maps (Fig. 3f). Tomopteris helgolandica Specimens were collected with plankton nets in the Trondheimfjord and the Stjørnfjord in Norway from 40-600 m depth. The specimens were housed in complete darkness in a temperature controlled room (12-14 °C) in fresh seawater pumped directly from the 18 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint Trondheimfjord at 100 m depth. Once collected, the specimens were only handled under red and/or infrared light. A 3D-printed circular arena (Ø 105  mm, height 37  mm), designed in Blender 4.1 (Blender Online Community, 2024), was placed in an aquarium filled with freshly filtered seawater (5  µm polypropylene filter cartridge, 9-PA-5). To prevent the arena from floating, two external compartments were designed to hold small weights. The arena also accommodated LEDs inserted through 5  mm holes (spaced 7  mm apart and 20  mm high), controlled via an Arduino UNO microcontroller. The arena was illuminated from below with three infrared lights and filmed from above with an infrared camera (Basler acA2000-165umNIR with a 16 mm, 1:1.4 lens) at a resolution of 1000x1000 pixels with 16  ms exposure and a final frame rate of 7 fps. OCTRON analyses We trained the YOLO “L” (YOLO11l-seg) model in OCTRON on 1660 images across a total of 14 videos of swimming T. helgolandica. For multi-animal tracking we used HybridSort in combination with “osnet_x1_0_market1501.pt” for Re-ID. Head-tail identification Representative frames were analyzed to extract head and tail positions from binary animal masks. Masks were first skeletonized using morphological thinning to produce centerline path coordinates. The skeleton was converted to a pixel graph representation, and the longest path through the graph was identified using all-pairs shortest path algorithms. This longest path defined the animal's primary body axis from one end to the other. Path coordinates were smoothed with a Gaussian filter (σ=3) to reduce pixelation artifacts. To discriminate head from tail, a distance transform was computed on the binary mask to obtain local thickness values at every pixel. For each endpoint of the longest skeleton path, a circular region (radius = 10% of mean major axis length across high-area frames) was defined. Within each circular region, the median distance transform value was calculated, representing the local body thickness. The endpoint with the greater median thickness was classified as the head (anterior end with larger diameter), while the opposite endpoint was classified as the tail. A confidence metric (dist_conf) was derived as 1 - (thickness_ratio), where thickness_ratio = min(thickness_head, thickness_tail) / max(thickness_head, thickness_tail). To ensure reliability, detections with confidence below threshold (dist_conf < 0.35) were set to NaN and linearly interpolated with a strict limit of 10 consecutive frames to maintain temporal continuity without over-smoothing gaps. Optional Gaussian filtering reduced residual jitter in derived coordinates. 19 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint An auxiliary angle reference point was defined by indexing into the reordered skeleton path at 20% of the total skeleton length from the head end (i.e., at position −int(0.2 × skeleton_length) in the tail-to-head ordered array). This reference point, located proximally along the anterior body, provided a stable local orientation vector. The allocentric angle was computed as arctan2(head_y - ref_y, head_x - ref_x), converted to degrees, and normalized to span the range [0°, 360°]. Body curvature was computed frame-by-frame by integrating perpendicular distances of skeleton points from the head-to-tail axis. For each skeleton coordinate, the cross product of the point vector (relative to tail) with the head-tail line vector determined whether the point lay to the left (positive cross product) or right (negative cross product) of the straight axis. The curvature metric was defined as (left_ratio - right_ratio), where left_ratio/right_ratio = fraction of skeleton points on each side. The metric also reported a straightness index computed as the ratio of Euclidean head to tail distance to total path arc length. Curvature time series were smoothed (Gaussian σ=3) for visualization (Fig. 2h). Composite backgrounds in Fig. 2h were created by randomly sampling 50 frames from the first 200 frames of each video, then averaging across time to produce an arena reference image. Animal paths were visualized using matplotlib (Hunter 2007) LineCollection. Speed was computed from framewise centroid differences and Gaussian-smoothed (σ=2). Normalized speed scaled segment linewidths (range ≈1–19 pixels) to convey movement intensity. Head angle series (allocentric, arena coordinates) were unwrapped to eliminate 2π discontinuities, then detrended by subtracting a running median computed over a 5-second sliding window (window size = fps × 5 frames) to remove slow drift while preserving oscillatory components. The detrended signal was plotted alongside Gaussian-smoothed speed (σ=2) on dual y-axes (angle: black; speed: pink) for a 20-second window (Fig. 2i). For frequency analyses, the instantaneous amplitude envelope and phase were computed via the Hilbert transform. Instantaneous frequency was derived as the time derivative of the unwrapped phase, scaled by fps and divided by 2π, then Gaussian-smoothed (σ=4). The amplitude envelope was similarly smoothed (σ=4) for stability. The power spectral density of the detrended head angle series was computed using Welch's method with segment length nperseg=280 frames, overlap n=200 frames. Resulting power spectra were lightly smoothed (Gaussian σ=3) for visual clarity and plotted on linear frequency axes (1–3 Hz range) (Fig. 2j). Peak frequencies were identified using the FOOOF library (Donoghue et al. 2020). Data processing and analyses All analyses were performed in Python using numpy (Harris et al. 2020), pandas (Team 2020), scikit-learn (Pedregosa et al. 2011), scipy (Virtanen et al. 2020), and matplotlib/seaborn (Hunter 20 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint 2007; Waskom 2021). Most plots used seaborn themes, and perceptually uniform colormaps via cmasher (van der Velden 2020). Statistics Statistical tests were run using scipy (Virtanen et al. 2020). When normality could not be assumed, non-parametric tests were used to provide more reliable results. Deployment options OCTRON can be installed and run on macOS, Windows, and Linux operating systems. Data availability The A. bandense dataset was collected by Horst A. Obenhaus at the Marine Biological Laboratory in Woods Hole, MA, U.S.A., and will be published elsewhere. The H. vulgaris dataset has been published previously by Han et al (Han et al. 2018) and is openly available on Dryad (https://datadryad.org/dataset/doi:10.5061/dryad.f6v067r). The M. leidyi dataset was collected by Oscar M. Arenas at the University of California, Berkeley, and will be published elsewhere. The O. vulgaris dataset was collected by Sofia Maccuro in the Laboratory for Developmental Neurobiology led by Eve Seuntjens (KU Leuven, Belgium) The S. officinalis dataset was collected by Johnston Renton at the Marine Biological Laboratory (Woods Hole, MA, U.S.A.) and will be published elsewhere. The T. helgolandica dataset was collected by Nadia M van Eekelen in the Marine Neuroscience Laboratory led by R. Irene Jacobsen at NTNU (Trondheim, Norway) and will be published elsewhere. Code availability The OCTRON codebase and GUI is available on https://github.com/OCTRON-tracking and is documented on https://octron-tracking.github.io/OCTRON-docs. 21 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint

We thank May-Britt and Edvard Moser for supporting our pursuit of novel research directions, as well as colleagues at both the Kavli Institute for Systems Neuroscience and the Department of Biology at NTNU for invaluable technical support. In particular Kyrre Haugen, Klaus Jøran Jenssen, Endre Kråkvik, Haagen Waade, Dag Altin, Stig Koteng, and Siv Anina Etter. We also thank Carsten Wolff at the MBL for technical support, as well as Abhishek Kumar and Carsten Wolff for providing access to computing resources at the MBL. We thank Roger Hanlon at the MBL for his expert guidance on experiments that were conducted with Sepia officinalis in his lab and for providing data for this paper. We thank Lisa Abbo, David Remsen, Dan Calzarette, Nolan Gibbons, Brian McGonagle, and Jennifer Camerlengo from the Marine Resources Center at MBL for their support in setup building, maintenance and animal care, as well as Taylor Sakmar and Bret Grasse from the Cephalopod Initiative at MBL for their expert guidance in cephalopod care. We also thank Anne Sylvester and Briana Bertochi at MBL for their administrative support during the Grass and Whitman Fellowships (HAO and OMA). We thank Eduardo Almansa for providing the Octopus vulgaris embryos. This work was supported by funding from Gidske and Peter Jacob Sørensens Fund (RIJ), Nansenfondet / Norsk Hydro (RIJ), the Helen Hay Whitney Foundation (OMA), the Grass Foundation (HAO, OMA), the Kavli Foundation (HAO) and the MBL Whitman Centre (HAO). SM and EvR were funded through KU Leuven grant C14/21/065 to ES and William Schafer and Fonds Wetenschappelijk Onderzoek grant G040124N to ES. Contributions HAO designed and wrote the OCTRON pipeline, performed analyses and generated figures. Testing and feedback on early versions of OCTRON were done by RIJ, NMvE, and LH. Experiments were designed and/or implemented by RIJ, NMvE, JRe, OMA, EAL, SM, and HAO, with supervision from RIJ, EAL, KB, ES, and HAO. Ground truth annotations were performed by RIJ, NMvE, LH, EvR, and JRi. Writing was carried out by RIJ, NMvE, EvR, JRe, ES, and HAO with feedback from all authors. Funding was acquired by RIJ, OMA, ES and HAO. Corresponding author Correspondence to Horst A. Obenhaus. Competing interests The authors declare no competing interests 22 .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint Main figure legends Fig.1 OCTRON workflow and performance. a. Pipeline schematic: I) load raw data in napari; II) create labels and use forward prediction for rapid training data generation; III) train and evaluate a new model; IV) predict on novel data with optional refinement; V) visualize

in OCTRON and/or use OCTRON library to work with the output. b. Annotation workflow examples. Top: Tomopteris helgolandica (pink) and LEDs (green). Scale bars: 10 mm and 5 mm (insets). Bottom: Ascarosepion bandense (green and orange) and shrimp (Palaemonetes spp., pink). Scale bars: 25 mm and 12 mm (insets). Round dots: masked object, crosses: background. Right: Unsupervised forward prediction yields stable predictions after annotation of only a few frames and thereby accelerates training data generation (+797 frames after last annotated frame for T. helgolandica, +219 frames for A. bandense). c. Additional annotation examples across species: Sepia officinalis, Hydra vulgaris, Mnemiopsis leidyi, Octopus vulgaris paralarva. d. Prediction examples: stable segmentation and centroid tracking of T. helgolandica (top) and H. vulgaris (bottom). T. helgolandica masks are shown across three distinct episodes, color-coded by time. Row order is maintained across panels (e) and (f). e. Outlines of manually drawn masks by five different human annotators (first and third rows, color-coded by annotator) and masks used for benchmarking mask precision (second and fourth rows; human average - green, OCTRON prediction - purple, overlap - black). f. Segmentation accuracy vs. human labels: Mean ± SD Intersection over Union (IoU) and box plots for models trained on varying training dataset sizes (YOLO M model trained on 60, 120, 240, 480, 960 frames) and on the full training dataset (YOLO M, and YOLO L at either 640px or 1024px max training image size). Top row (T. helgolandica), model: M640, t-stat: -3.77, p-value: 0.0005***, model: L1024, t-stat: -3.874, p-value: 0.0004***. Bottom row (Hydra vulgaris - tentacle region), model: M640, t-stat: -2.998, p-value: 0.0046**, model: L640, t-stat: -2.684, p-value: 0.011*). Fig. 2 Multi-animal tracking and keypoint extraction. a. Octopus paralarvae open-field trajectories with size-coded mantle area and color-coded speed for three individuals. Right column: Zoomed in images of the same individuals, the arrow connects the centroid of the mantle (blue) to the head (orange). Heading angle is denoted in degrees in the bottom left. b. Position-correlation heatmaps (x, y) between mantle and head tracks; high-correlation pairs are denoted on the diagonal. c. Box plot comparing position correlations of high-correlation pairs (n=9) and all others (n=18). Mann-Whitney U test: U=162.0, p=3.47e-05***. Single data points overlaid on top. d. PCA on cropped and rotated mantle and head masks: average of highest and lowest 50 PC scores and difference map along each component for PC1 (top) and PC2 (bottom). PC1 distinguishes animals swimming horizontally (high) versus away or towards the camera (low), whereas PC2 distinguishes a more compact (high) versus streamlined body configuration (low), often seen during high speed swimming bouts (jet propulsion). e. Top: Speed (black) and PC2 (blue) time series; bottom: Speed versus PC2 (Pearson’s r=-0.574, p < 0.001). f. Experimental arena with two T. helgolandica annelids. g. Head-tail extraction from OCTRON-predicted masks: left, skeletonized centerline (white), endpoints classified as head (magenta) and tail (cyan) using local mask thickness; right, centerline (white), endpoints, reference point for head angle (black) and head direction (arrow) overlaid on raw frame. Confidence values ([0,1]) indicate certainty of head-tail assignment. h. Open-field tracks per annelid (same data as in f and g): line width indicates speed, hue indicates curvature direction and its strength; background is an average frame. i. Detrended head angle (black) and speed (pink) over time for one example animal. j. Left: Head-angle power spectral density (PSD, peak ≈1.9 Hz) for animal in (i). Right: box plot of peak frequencies across annelids (n=5 animals); median (pink line), interquartile range (gray box), and whiskers extending to min/max values to disable outlier classification. Individual data points were overlaid as dots. .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint Fig. 3 Tracking a camouflaging animal. a. Circular arena divided into three parts with images of various substrates (coarse, medium, and fine sand) covering the floor and walls. b. Example frame (left) and segmentation overlays (right) showing mantle (pink), head (cyan), and head direction (black arrow) of Sepia officinalis. Scale bar: 3 cm. c. Cropped and derotated dorsal view of cuttlefish from the image in (b). d. Example cropped and derotated L* channel frames from autoencoder clusters assigned to disruptive (clusters 0, 2) and mottle/uniform (cluster 3) pattern groups. e. Top: 60 min animal trajectory (pooled data from six individual videos). Color indicates head direction; line width indicates normalized speed. Bottom: heatmap of occupancy within the arena. f. Pattern tuning maps: disruptive (top) and mottle/uniform (bottom) expression probability normalized by occupancy; warmer colors indicate more frequent expression of the pattern. Extended data figure legends Extended Data Fig. 3. Autoencoder, clustering of patterns and clustered pattern examples. a. Autoencoder training and test loss (mean squared error, MSE) over 50 epochs. b. PCA projection of 32D latent embeddings with HDBSCAN cluster labels. c. Representative derotated L* channel frames from each cluster (0–4), illustrating disruptive (clusters 0, 2, 4) and mottle/uniform (clusters 1, 3) pattern groups. Supplementary video legends Supplemental Video 1 Three Octopus vulgaris paralarvae filmed inside a 3 cm diameter petri dish. Same data as shown in Fig. 2a. Mantle (pink) and head masks (green) are shown as overlays for each tracked octopus. Trailing paths are color-coded by octopus ID with saturation reflecting passed time. The data were tracked via HybridSort (see methods). Extended Data Fig. 1 Additional benchmarking results across training sizes and models. As in Fig. 1f. Segmentation accuracy versus human labels: Mean ± SD Intersection over Union (IoU) and box plots for models trained on varying sizes (YOLO M model trained on 60, 120, 240, 480, 960 frames) and on the full training dataset (YOLO M, and YOLO L at either 640px or 1024px max training image size) for four different species. Statistics: * p < 0.05, ** p < 0.01 Extended Data Fig. 2 Example segmentations and arena alignment. a. Raw video frame of the circular arena with three substrate sectors and one Sepia officinalis cuttlefish. b. Example frames: raw view (left), OCTRON detections overlaid (middle; mantle in pink, head in cyan, white arrow = head direction), and corresponding cropped and derotated animal images (right). Top (video frame 980) middle image also shows the detected background regions in gray as an overlay. c. OCTRON-derived background masks with fitted tank boundary (dashed circle) and sector boundaries (Medium-Fine, Fine-Coarse, Coarse-Medium). d. Fitted tank boundaries and substrate boundary points for each video (left) and after affine alignment to a common

frame (right); gray tones denote individual recordings. .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint Supplemental Video 2 Two Tomopteris helgolandica. Same data as shown in Fig. 2f. Masks, IDs and trailing paths of mask centroids are shown for both annelids. Video is playing back at original speed (7fps). The extracted centreline (white), head (pink dot), tail (blue dot) and anchor point for head angle extraction (black dot) are shown as overlays for each animal. Supplemental Video 3 Sepia officinalis. A single Sepia officinalis cuttlefish. Excerpt of data shown in Fig. 3. The masks of the head (green) and mantle (pink) regions are shown, along with the trailing paths of their respective centroids. Throughout the video, the cuttlefish undergoes multiple changes in its skin pattern. .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint .CC-BY-NC-ND 4.0 International licenseavailable under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made The copyright holder for this preprintthis version posted December 23, 2025. ; https://doi.org/10.64898/2025.12.20.695663doi: bioRxiv preprint