{"paper_id":"0ab3e9a1-dd18-44d2-9bba-dc12e10eece1","body_text":"New idtracker.ai: rethinking1\nmulti-animal tracking as a2\nrepresentation learning problem to3\nincrease accuracy and reduce4\ntracking times5\nJordi Torrents1†, Tiago Costa 1†, Gonzalo G. de Polavieja 1*6\n*For correspondence:\ngonzalo.polavieja@neuro.\nfchampalimaud.org\n†These authors contributed\nequally to this work\n1Champalimaud Research, Champalimaud Center for the Unknown - Lisbon, Portugal7\n8\nAbstract idTracker and idtracker.ai approach multi-animal tracking from video as an image9\nclassification problem. For this classification, both rely on segments of video where all animals10\nare visible to extract images and their identity labels. When these segments are too short,11\ntracking can become slow and inaccurate and, if they are absent, tracking is impossible. Here, we12\nintroduce a new idtracker.ai that reframes multi-animal tracking as a representation learning13\nproblem rather than a classification task. Specifically, we apply contrastive learning to image14\npairs that, based on video structure, are known to belong to the same or different identities. This15\napproach maps animal images into a representation space where they cluster by animal identity.16\nAs a result, the new idtracker.ai eliminates the need for video segments with all animals visible, is17\nmore accurate, and tracks up to 440 times faster.18\n19\nVideo-tracking systems that attempt to follow individuals frame-by-frame can fail during oc-20\nclusions, resulting in identity swaps that accumulate over time Branson et al. (2009); Plum (2024);21\nChen et al. (2023); Chiara and Kim (2023); Liu et al. (2023); Bernardes et al. (2021). idTracker Pérez-22\nEscudero et al. (2014) introduced the paradigm of animal tracking by identification from the animal23\nimages. This approach, unfeasible for humans, avoids the accumulation of errors by identity swaps24\nduring occlusions. Its successor, idtracker.ai Romero-Ferrero et al. (2019), built on this paradigm25\nby incorporating deep learning and achieved accuracies often exceeding 99.9% in videos of up to26\n100 animals.27\nWhile both idTracker and idtracker.ai perform well in high-quality video, they share a limitation28\nthat can be critical in videos of lower quality or with many occlusions. To understand this limitation,29\nconsider the schematics of a video in Figure 1a. The first step of both idTracker and idtracker.ai30\nconsists in detecting instances when animals touch or cross paths ( Figure 1a, shown as boxes with31\ndashed borders and containing images of overlapping fish in this example). The video is then di-32\nvided into individual fragments, each consisting of the set of images of a single individual between33\ntwo animal crossings (Figure 1a shows 14 of them as rectangles with a gray background). A global34\nfragment for a video with 𝑁 animals is a collection of 𝑁 fragments that coexist in one or more con-35\nsecutive frames in the video ( Figure 1a, the 5 fragments with blue borders are a global fragment).36\nThe significance of a global fragment is that it provides a set of images and identity labels for all37\nthe animals in the video.38\n1 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\nThe core idea of idTracker and the original idtracker.ai is to use global fragments for the classi-39\nfication of images of animals into identities. In idtracker.ai, this process starts by training a convo-40\nlutional neural network (CNN) with the images and labels of the global fragment that contains the41\nlongest fragment for the animal that moves the least. Once trained, the network assigns identities42\nto all animal images in the remaining global fragments. Only global fragments meeting strict qual-43\nity criteria, such as ensuring all animals in a global fragment have unique identities, are retained44\nfor further training. This iterative process of training, assigning, and selecting continues until most45\nof the video have images assigned to identities. A second algorithm then tracks animals during46\ncrossings given that animals are already identified outside crossings.47\nFigure 2a (blue line) shows the accuracies of the original idtracker.ai (version 4 of the software)48\nfor a benchmark of 33 videos of zebrafish, flies and mice. These accuracies were computed using49\nall the images of animals in the videos excluding animal crossings. Figure 2—figure Supplement 1a50\nshows the same results but for the complete trajectory with animal crossings. The names of the51\nvideos start with a letter for the species (z,f,m), followed by the number of animals in the video,52\nand possibly an extra number to distinguish the video if there are several of the same species and53\nanimal group size. The videos in this figure are ordered by decreasing accuracy of the original54\nidtracker.ai results for ease of visualization. The first 15 videos are videos of zebrafish, flies and55\nmice with an accuracy of > 99.9%. The accuracy in the remaining videos gradually decreases to56\n92.67% in video 𝑚_4_2, and a value of 50.4% outside the figure for video 𝑑_100_3.57\nFigure 2b (blue line) shows the times that the original idtracker.ai takes to track each of the58\nvideos in the benchmark. The videos are ordered by increasing tracking times for ease of visualiza-59\ntion. The original idtracker.ai has a faster protocol, “Protocol 2”, which works well for the simplest60\nvideos and its tracking times ranging from a few minutes to several hours. However, for complex61\nvideos, the software may switch from “Protocol 2” to “Protocol 3”, with Protocol 3 a two-step pro-62\ncess. In the first step, all the global fragments are used to train the CNN filters. The second step63\nproceeds like Protocol 2 but with the initial weights of the CNN filters obtained from the first step.64\nWhile effective, this approach can be extremely slow, often requiring several days or weeks for a65\nsingle video. Since it is stochastic whether a video is tracked using Protocol 2 or 3 ( Figure 2—figure66\nSupplement 2), a reasonable strategy to use the original idtracker.ai is to track each video multiple67\ntimes until Protocol 2 successfully tracks the entire video or, when a patience threshold is reached68\n(here set to 5 attempts), switch to Protocol 3. The tracking times shown in Figure 2b (blue line)69\ncorrespond to this procedure, with the time being the accumulated time of the multiple attempts70\nmade by the software until final tracking. Some of the videos take a few minutes to track, others a71\nfew hours, and six videos take more than three days, one nearly two weeks. If we were to run id-72\ntracker.ai a single time instead of following this protocol, the tracking times for some of the videos73\nwould be longer.74\nWe first optimized idtracker.ai by improving data loading protocols and redefining the main ob-75\njects in the software (animal images and fragments) and their properties (see Methods for details).76\nThis version of the optimized original idtracker.ai (version 5 of the software) achieved better ac-77\ncuracies, Figure 2a (orange line), and Figure 2—figure Supplement 1a (orange line) for accuracies78\nincluding animal crossings. The mean accuracy across the benchmark for this optimized version is79\n99.58% and 99.40% including or not animal crossings, respectively, while for the original idtracker.ai80\nare 97.52% and 97.38%.81\nEven if this version also uses Protocols 2 and 3, we obtain much shorter tracking times, never82\nlonger than a day Figure 2b (orange line). On average, tracking is 13.6 times faster than with the83\noriginal idtracker.ai and, for the more difficult videos, 118.4 times faster. However, waiting a day84\nto track some videos can make a tracking pipeline too slow. To further improve accuracy and85\ntracking times, we retained these optimizations while also changing the main logic of idtracker.ai.86\nIn the original idtracker.ai, when global fragments are short, the quality of the initial CNN is low,87\nleading to either reduced accuracy or the triggering of the very slow Protocol 3. The new system88\nhad to be able to track without global fragments.89\n2 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\nRepresentation\nspace\nTraining batches Silhouette score\nTime\n2,000\n0 batches\n4,000\n 15,000 15,000\na\nb c\nd e\nResNet18\n...\nConv 1\nFully connected\nConv 17\nConv 2\n2,0000 4,000 15,000\nFigure 1. Tracking by identification using deep contrastive learning . a Schematic representation of a video with five fish. It shows 7 portions\nof video with animals crossing or touching (dashed-border boxes), and 14 individual fragments, sequences of images of a single individual\nbetween two crossings (gray-background boxes). The blue-border fragments form a global fragment, as there are as many individual fragments\nas animals and all the individual fragments coexist in one or more frames. Some pairs of images of the same animal identity are highlighted\nwith green borders (positive images) and some images of different identities are highlighted with red borders (negative images). b A ResNet18\nnetwork with 8 outputs generates a representation of each animal image as a point in an 8-dimensional space (here shown in 2D for\nvisualization). Each pair of images corresponds to two points in this space, separated by a Euclidean distance. The ResNet18 network is trained\nto minimize this distance for positive pairs and maximize it for negative pairs. c 2D t-SNE visualizations of the learned 8-dimensional\nrepresentation space. Each dot represents an image of an animal from the video. As training progresses, clusters corresponding to individual\nanimals become clearer, plotted at training with 0, 2,000, 4,000 and 15,000 batches. The t-SNE plot at 15,000 training batches is also shown\ncolor-coded by human-validated ground-truth identities. The pink rectangle at 2,000 batches of training highlights clear clusters and the orange\nsquare fuzzy clusters. d The silhouette score measures cluster coherence and increases during training, as illustrated for a video with 60\nzebrafish. e A silhouette score of 0.91 corresponds to a human-validated error rate of less than 1% per image.\nFigure 1—figure supplement 1. Models comparison\nFigure 1—figure supplement 2. Embedding dimensions comparison\nFigure 1—figure supplement 3. 𝐷neg over 𝐷pos comparison\nFigure 1—figure supplement 4. Batch size comparison\nFigure 1—figure supplement 5. Exploration and exploitation comparison\n3 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\nz_10_1z_80_3z_80_2\nz_20z_7\nz_10_3z_60_3d_72z_10_2z_100_2z_100_3z_10_4m_2_4\nz_5d_60d_38z_60_2z_80_1d_59d_80d_6\nm_2_2d_10d_8\nm_4_1m_2_1z_60_1d_100_1d_100_2m_2_3z_100_1m_4_2d_100_3\n92%\n93%\n94%\n95%\n96%\n97%\n98%\n99%\n100%accuracy without crossings\noriginal idtracker.ai (v4)\noptimized v4 (v5)\nnew idtracker.ai (v6)\nm_2_4m_2_3m_2_2m_2_1m_4_2\nz_7\nm_4_1\nz_5\nz_10_4\nd_6\nz_10_1\nd_8\nz_10_3z_10_2\nz_20\nz_60_3z_60_1z_80_3z_60_2z_80_2z_100_2z_100_3\nd_10d_38z_80_1d_72\nz_100_1\nd_59d_60d_80\nd_100_1d_100_2d_100_3\n0\n1\n2\n3\n4\n5tracking time (hours)\n0\n5\n10\n15(days)\na\nb\nFigure 2. Performance for a benchmark of 33 videos of flies, zebrafish and mice. a. Median accuracy was\ncomputed using all images of animals in the videos excluding animal crossings. b. Median tracking times are\nshown for the scale of hours and, in the inset, for the scale of days. Supplementary Table 1, Supplementary\nTable 2, Supplementary Table 3 give more complete statistics (median, mean and 20-80 percentiles) for the\noriginal idtracker.ai (version 4 of the software), optimized v4 (version 5) and new idtracker.ai (version 6),\nrespectively.\nFigure 2—figure supplement 1. Performance for the benchmark with full trajectories with animal crossings\nFigure 2—figure supplement 2. Protocol 2 failure rate\nFigure 2—figure supplement 3. Memory usage across the different softwares.\nFigure 2—figure supplement 4. Robustness to blurring and light conditions\nWe reformulate multi-animal tracking as a representation learning problem. In representation90\nlearning, we learn a transformation of the input data that makes it easier to perform downstream91\ntasks Xing et al. (2002); Bengio et al. (2013); Ericsson et al. (2022), in our case clustering into animal92\nidentities without needing identity labels. This is possible due to the structure of the video, Fig-93\nure 1a. Note that pairs of images of the same individual can be obtained from the same fragment94\n(Figure 1a , green boxes). Also, pairs of images from different individuals can be obtained from95\ndifferent fragments that coexist in time for one or more frames (Figure 1a , red boxes). These pairs96\ncan be used as “positive” and “negative” pairs of images for contrastive learning, a self-supervised97\nlearning framework designed to learn a representation space in which “positive” examples are98\nclose together, and “negative” examples are far apart Schroff et al. (2015); Dong and Shen (2018);99\nKAYA and BİLGE(2019); Chen et al. (2020a,b); Guo et al. (2020); Wang et al. (2020); Yang et al. (2020).100\nWe first evaluated neural networks suitable for contrastive learning with animal images. In101\naddition to our previous CNN from idtracker.ai, we tested 23 networks from 8 different families102\nof state-of-the-art convolutional neural network architectures, selected for their compatibility with103\nconsumer-grade GPUs and ability to handle small input images (20 × 20 to 100 × 100 pixels) typical104\nin collective animal behavior videos. Among these architectures, ResNet18 He et al. (2016) was the105\nfastest to obtain low errors ( Figure 1—figure Supplement 1).106\nA ResNet18 with 𝑀 outputs maps each input image to a point in an 𝑀-dimensional represen-107\ntation space (illustrated in Figure 1b as a point on a plane). Experiments showed that using 𝑀 = 8108\nachieved faster convergence to low error ( Figure 1—figure Supplement 2). ResNet18 is trained us-109\n4 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\ning a contrastive loss function (Chopra et al. (2005), see Methods for details). Each image in a pos-110\nitive or negative pair is input separately into the network, producing a point in the 8-dimensional111\nrepresentation space. For an image pair, we then obtain two points in an 8-dimensional space,112\nseparated by some (Euclidean) distance. The loss function is used to minimize (or maximize) this113\nEuclidean distance for positive (or negative) pairs until the distance 𝐷pos (or 𝐷neg). The effect of 𝐷pos114\nis to prevent the collapse to a single of the positive images coming from the same fragment, allow-115\ning for a small region of the 8-dimensional representation space for all the positive pairs of the116\nsame identity. The effect of 𝐷neg is to prevent excessive scatter of the points representing images117\nfrom negative pairs. We empirically determined that 𝐷neg∕𝐷pos = 10 results in a faster method to118\nobtain low error ( Figure 1—figure Supplement 3), and we use 𝐷pos = 1 and 𝐷neg = 10.119\nAs the model trains, the representation space becomes increasingly structured, with similar120\ndata points forming coherent clusters. Figure 1c visualizes this progression using 2D t-SNE van der121\nMaaten and Hinton (2008) plots of the 8-dimensional representation space. After 2, 000 training122\nbatches, initial clusters emerge, and by 15,000 batches, distinct clusters corresponding to indi-123\nvidual animals are evident. Ground truth identities verified by humans confirm that each cluster124\ncorresponds to an animal identity (Figure 1c , colored clusters).125\nThe method to select positive and negative pairs is critical for fast learning Awasthi et al. (2022);126\nKhosla et al. (2021); Rösch et al. (2024). This is because not all image pairs contribute equally to127\ntraining. Figure 1c shows at 2, 000 training batches that some clusters well-defined (e.g. those in-128\nside the orange square) while others remain fuzzy (e.g. those inside the pink rectangle). Images129\nin well-defined clusters have negligible impact on the loss or weight updates, as positive pairs130\nare already close and negative pairs are sufficiently separated. Sampling from these well-defined131\nclusters, therefore, wastes time. In contrast, fuzzy clusters contain images that still contribute sig-132\nnificantly to the loss and benefit from further training. To address this, we developed a sampling133\nmethod that prioritizes pairs from underperforming clusters requiring additional learning, while134\nmaintaining baseline sampling for all clusters based on fragment size ( Methods). This ensures con-135\nsistent updates across the representation space and prevents forgetting in well-defined clusters.136\nTo assign identities to animal images, we perform K-means clustering Sculley (2010) on the137\npoints representing all images of the video in the learned 8-dimensional representation space.138\nEach image is then assigned to a cluster with a probability that increases the closer it is to the139\ncluster center. To evaluate clustering quality, we compute the mean Silhouette index Rousseeuw140\n(1987), which quantifies intra-cluster cohesion and inter-cluster separation. A maximum value of141\n1 indicates ideal clustering. During training, the mean Silhouette index increases ( Figure 1d ). We142\nempirically determined that a value of 0.91 for this index corresponds to an identity assignment143\nerror below 1% for a single image ( Figure 1e). As a result, we use 0.91 as the stopping criterion for144\ntraining (Methods).145\nAfter the assignment of identities to images of animals, we run some steps that are common146\nto the previous idtracker.ai. For example, we make a final assignment of all images in fragments147\nas each fragment must have all assignments to be the same, eliminating some errors in individual148\nimages. Also, an algorithm already present in idTracker assigns identities in the animal’s crossings149\ntaking into account that we know the identities before and after.150\nThe new idtracker.ai has a higher accuracy than original idtracker.ai and than its optimized151\nversion, Figure 2a (magenta line). Its average accuracy in the benchmark is 99.92% and 99.78%152\nwithout and with crossings, respectively, an important improvement over the original idtracker.ai153\n(97.52% and 97.38%) and its optimized version (99.58% and 99.40%). It also gives much shorter times154\nthan the original idtracker.ai and its optimized version, Figure 2b (magenta line). It is on average 44155\ntimes faster than the original idtracker.ai and, for the more difficult videos, up to 440 times faster.156\nAs for the original idtracker.ai, the new idtracker.ai can work well with lower resolutions, blur157\nand video compression, and with inhomogeneous light ( Figure 2—figure Supplement 4). We also158\ncompared the new idtracker.ai to TRex Walter and Couzin (2021), which is based on Protocol 2 of159\nidtracker.ai but with additional operations like eroding crossings to make global fragments longer.160\n5 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\nnew idtracker.ai (v6) \noptimized v4 (v5)\nFigure 3. Tracking with strong occlusions . Accuracies when we mask a region of a video defined by an\nangle 𝜃 and the tracking system has no access to the information behind the mask. Light and dark gray region\ncorrespond to the angles for which no global fragments exist in the video. Dark gray regions correspond to\nangles for which the video does not have enough coexisting individual fragments, specifically on average less\nthan 0.25(𝑁 − 1) coexisting individual fragments, with 𝑁 the number of animals in the video. The original\nidtracker.ai (v4) and its optimized version (v5) cannot work in the gray regions, and new idtracker.ai is\nexpected to deteriorate only in the dark gray region.\nTRex gives comparable accuracies to the original idtracker.ai in the benchmark, but by avoiding161\nProtocol 3, it is on average 31 times faster than the original idtracker.ai and up to 315 times faster162\n(Figure 2—figure Supplement 1b ). However, the new idtracker.ai is both more accurate and faster163\nthan TRex (Figure 2—figure Supplement 1). The mean accuracy of TRex across the benchmark is164\n98.14% and 97.89% excluding and including animal crossings, respectively. This is noticeably below165\nthe values for the new idtracker.ai of 99.92% and 99.78%, respectively. Also, the new idtracker.ai is166\non average 3.9 times faster and up to 16.5 times faster than TRex. Additionally, the new idtracker.ai167\nhas a memory peak lower than TRex (Figure 2 —figure Supplement 3).168\nThe new idtracker.ai also works in videos in which the original idtracker.ai does not even track169\nbecause there are no global fragments. Global fragments are absent in videos with very exten-170\nsive animal occlusions, for example because animals touch or cross more frequently, parts of the171\nsetup are covered, or the camera focuses on only a specific region of the setup. To study this sys-172\ntematically, we added a mask on the video with an angle 𝜃 (Figure 3). The tracking systems have173\nno access to the information behind the mask. The light and dark gray regions in Figure 3 corre-174\nspond to videos with no global fragments, and the original idtracker.ai and its optimized version175\ndeclare tracking impossible. The new idtracker.ai, however, works well until approximately 1∕4 of176\nthe setup is visible, and afterward it degrades. This also shows the limit of the new idtracker.ai. For177\nthe clustering process to be successful, we need enough coexisting individual fragments to have178\nboth positive and negative examples. Empirically, we find a deterioration with less than 0.25(𝑁 − 1)179\ncoexisting individual fragments, with 𝑁 the number of animals in the video ( Figure 3, dark gray180\nregion). The new idtracker.ai flags when this condition is not met.181\nThe final output of the new idtracker.ai consists of the 𝑥 − 𝑦 coordinates for each identified ani-182\nmal and video frame. Additionally, it provides several quality metrics: an estimate of the probability183\nof correct identity assignment for each animal and frame, the Silhouette score as a measure of clus-184\ntering quality, and the average number of coexisting individual fragments per fragment divided by185\n(𝑁 − 1), with 𝑁 the number of animal in the video, which when above 0.25(𝑁 − 1) is expected to186\ngive good results. The software can also generate a video with the computed animal trajectories187\nfor visualization, and an individual video per animal to be able to run pose estimators like the ones188\nin Lauer et al. (2022); Pereira et al. (2022); Segalin et al. (2021); Tang et al. (2025); Biderman et al.189\n(2024). For analysis of trajectories and spatial relationships, the user can run our Python package190\n6 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\ntrajectorytools on the trajectories.191\nIn summary, the new idtracker.ai takes an approach to tracking using representational learning192\nto avoid the need for segments of the video in which all animals are visible. This makes the new193\nidtracker.ai work in more videos, more accurately, much faster and with a lower memory peak.194\nAcknowledgments195\nWe thank Alfonso Perez-Escudero, Paco Romero-Ferrero, Francisco J. Hernandez Heras, and Madalena196\nValente for discussions. This work was supported by Fundaçao para a Ciência e Tecnologia PTDC/BIA-197\nCOM/5770/2020 (to G.G.dP.) and Champalimaud Foundation (to G.G.dP.).198\nAuthor contributions199\nT.C. and G.G.dP. devised project and main algorithm, T.C. performed tests of the algorithm as stand200\nalone, J.T. developed version 5, implemented the new algorithm into idtracker.ai architecture and201\nmade final tests with help from T.C., G.G.dP. supervised project, T.C. wrote the Appendices with202\nhelp from J.T and G.G.dP., and G.G.dP. wrote the main text with help from J.T and T.C.203\nMethods204\nSoftware availability205\nidtracker.ai is a free and open source project (license GPL v.3). Information about its installation206\nand usage can be found on the official website https://idtracker.ai. The source code is available in207\ngitlab.com/polavieja_lab/idtrackerai and the package is pip-installable from PyPI. All versions can208\nbe found in these platforms, specifically “original idtracker.ai (v4)” as v4.0.12, “optimized v4 (v5)” as209\nv5.2.12 and “new idtracker.ai (v6)” as v6.0.0.210\nData availability211\nAll videos used in this study, their tracking parameters and human-validated groundtruth can be212\nfound in our data repository at https://idtracker.ai.213\nTested computer specifications214\nThe software idtracker.ai depends on PyTorch and is thus compatible with any machine that can215\nrun PyTorch, including Windows, MacOS, and Linux systems. Although no specific hardware is re-216\nquired, a graphics card is highly recommended for hardware-accelerated machine-learning com-217\nputations.218\nVersion 6 of idtracker.ai was tested on computers running Ubuntu 24.04, Fedora 41, and Win-219\ndows 11 with NVIDIA GPUs from the 1000 to the 4000 series and MacOS 15 with Metal chips. The220\nbenchmark presented in this study was performed on desktop computer running Ubuntu 24.04221\nLTS 64bit with a AMD Ryzen 9 5950X (32 cores at 3.4 GHz) processor, 128 GB RAM and an NVIDIA222\nGeForce RTX 4090.223\nImprovements to original idtracker.ai in version 5224\nFollowing the last publication of idtracker.ai Romero-Ferrero et al. (2019), the software underwent225\ncontinuous maintenance, including feature additions, performance optimizations, and hyperpa-226\nrameter tuning (released via PyPI from March 2023 for v5.0.0 to June 2024 for v5.2.12). These227\nupdates improved the implementation and tracking pipeline but did not alter the core algorithm.228\nSignificant advancements were made in user experience, tool availability, processing speed, and229\nmemory efficiency. Below, we summarize the most notable changes.230\nBlob memory optimization: Blobs are defined as collections of connected pixels belonging to231\none or more animals. In v4, blobs stored pixel indices, causing memory usage to scale quadrati-232\ncally with blob size. In v5, blobs are represented by simplified contours using the Teh-Chin chain233\napproximation Teh and Chin (1989), reducing memory usage by 93% in blob instances. This also234\n7 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\nFigure 4. idtracker.ai new graphic user interface. New graphics user interface (GUI) for versions v5 and v6\nof idtracker.ai. On the left the segmentation GUI. On the right the Validator tool.\naccelerated blob-related computations (centroid, orientation, area, overlap, identification image235\ncreation, etc.).236\nEfficient image loading: Identification images are now efficiently loaded on demand from237\nHDF5 files, eliminating the need to load all images into memory. This enables training with all238\nimages regardless of video length, with minimal memory usage.239\nCode optimization: The source code was revised to eliminate speed bottlenecks. The most240\nimpactful changes include:241\n• Frame segmentation accelerated by 80% through optimized OpenCV usage.242\n• Faster blob-to-blob overlap checks by first evaluating bounding boxes before deeper com-243\nparisons.244\n• Persistent storage of blob overlap checks to avoid redundant computations when reloading245\ndata.246\n• Efficient disk access for identification images by reading them in sorted batches, minimizing247\nI/O overhead.248\n• Reduced bounding box image sizes to the minimum necessary, lowering memory and pro-249\ncessing demands.250\n• Optimized and parallelized Torch data loaders for more efficient model training.251\n• Caching of computationally expensive properties for blobs, fragments, and global fragments.252\n• Sorted Fragment lists to speed up coexistence detection.253\nChanges to the identification protocol: In v4, identity assignments to high-confidence frag-254\nments were fixed and excluded from downstream correction, regardless of later evidence. In v5,255\nthis was relaxed for short fragments (fewer than 4 frames), allowing corrections due to their statis-256\ntical unreliability and frequent image noise.257\nImproved graphical user interface and introduction of Exclusive ROIs: The graphical user258\ninterface was redesigned for improved usability and now includes the \"Exclusive Regions of In-259\nterest\" feature, which allows users to define spatially distinct regions in multi-arena experiments260\nwhere animal identities are treated independently (see Figure 4 left image). It also incorporates a261\nredesigned video generator for visualizing tracking results.262\nValidation application: A standalone GUI for inspecting and correcting tracking results. It al-263\nlows users to navigate video frames, review tracked positions and metadata, detect tracking errors,264\nand apply corrections using integrated plugins (see Figure 4, right image).265\nDirect integration with idmatcher.ai: A utility for matching identities across videos, originally266\nintroduced in Romero-Ferrero et al. (2023). It allows users to propagate consistent identity labels267\nacross multiple recordings, facilitating longitudinal or multi-session experiments. It is now a native268\nfeature of both v5 and v6, fully integrated into the idtracker.ai ecosystem.269\n8 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\nProtocol details for the new idtracker.ai270\nIn this section, we give an overview of the tracking protocol. Please refer to Appendix 1 for details.271\nArchitectures272\nThe contrastive learning network (Figure 1b) is a ResNet18He et al. (2016) with a single channel in the273\nfirst convolutional layer for grayscale images and 8 neurons in the last layer. The network receives274\ngrayscale images because idtracker.ai always works with grayscale converted video frames.275\nLoss function276\nThe contrastive loss 𝐿 for a pair of images (𝐼, 𝐽 ) and label 𝑙 is defined as:277\n(𝐼, 𝐽 , 𝑙)=𝑙𝐼, 𝐽 ⋅ max(0, 𝐷𝐼, 𝐽 − 𝐷pos)2 + (1 − 𝑙) ⋅ max(0, 𝐷neg − 𝐷𝐼, 𝐽 )2\n𝑙 =\n⎧\n⎪\n⎨\n⎪⎩\n1 if I and J come from the same fragment, (positive pair)\n0 if I and J come from coexisting fragments (negative pair)\nHere 𝐷𝐼, 𝐽 is the Euclidean distance between the embeddings of images 𝐼 and 𝐽 . 𝐷pos is the maxi-278\nmum allowed distance between the two images of a positive pair, and 𝐷neg, the minimum allowed279\ndistance between the two images in negative pair.280\nTraining281\nResNet18 is trained using Adam optimizer with the hyperparameters described in Kingma and282\nBa (2017). The learning rate is set at the value of 0.001 using training batches of 1600 images (400283\npositive pairs and 400 negative pairs of images). See Appendix 2 for details.284\nPair selection285\nThe selection of pairs was done by combining two sampling strategies:286\n1. Sampling fragments according to their size so that fragments containing more images are287\nsampled more often.288\n2. Sampling fragments according to the loss function by increasing the sampling probability289\nof pairs of fragments from whom the corresponding images had positive loss, and decreasing290\nthe sampling probability of pairs of fragments from whom the corresponding images had loss291\nzero.292\nSee Appendix 2 for more details on the pair sampling strategy.293\nClustering and stopping criteria294\nFor clustering, we use the minibatch K-means clustering, which significantly reduces the computa-295\ntion time compared to a classical implementation Sculley (2010).296\nStopping of the training was done by computing the K-means clustering for a subset of (number297\nof animals times 1,000) images, and measuring the corresponding Silhouette score (SS) Rousseeuw298\n(1987) every number of animals times 5 batches. We stop training if there have been 30 consecutive299\nSS evaluations without any improvement (patience of 30), or if there have been 2 consecutive SS300\nevaluations without any improvement but the SS already achieved the target value 0.91. Check301\nAppendix 2 for more details on the criteria to stop the training of the network.302\nOcclusion tests303\nFor the occlusion tests, we took videos of freely behaving animals in a round arena (included in the304\nbenchmark) and occluded a sector of the circle between 0 and 𝜃 radians. For the tracking software,305\nanimals disappeared when entering this occluded section of the arena. The light gray area in Fig-306\nure 3 corresponds to a degree of occlusion that prevents the existence of global fragments. The307\n9 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\ndark gray area in Figure 3 corresponds to a degree of occlusion where there are less than0.25(𝑁 −1)308\ncoexisting individual fragments ( 𝑁 being the number of animals in the video). With these degrees309\nof occlusion, too few animals overlap at any given time and identification is expected to deteriorate310\nin this regime (Figure 3 , dark gray region). idtracker.ai flags when this condition is not met.311\nComputation of tracking accuracy312\nUsing the idtracker.ai Validator tool (see Methods), we manually generated ground-truth trajecto-313\nries based on v5 outputs. This ground-truth consists on the positions and identities of all animals314\nin each frame and their classification as either individual or crossing.315\nTo detect tracking errors, we analyze the video frame by frame, verifying whether the predicted316\nposition of each animal deviates from the ground-truth by more than a threshold 𝑇 . Errors are also317\nrecorded when the software loses the identity or fails to detect an animal in a given frame.318\nTracking accuracy is then defined as one minus the proportion of errors in the trajectory. For319\naccuracy with crossings , we consider all trajectory points, whereas for accuracy without cross-320\nings, we exclude points corresponding to crossing events in the ground-truth.321\nWe present all results using a threshold 𝑇 = 1BL with BL being a body length. We also verified322\nthat accuracy remains largely unaffected by the value of this threshold. For instance, reducing it323\nto 𝑇 = 0.5BL results in a very small change of the mean accuracy (without crossings) across the324\nbenchmark in the new idtracker.ai from 99.92% to 99.90%.325\nBenchmark of accuracy and tracking time326\nTo evaluate the tracking time and accuracy of different versions of idtracker.ai and version 1.1.9327\nof TRex, we used a set of 33 videos with their corresponding human-validated ground-truth tra-328\njectories. Each video is 10 minutes long and features one of three species: mice, drosophila, or329\nzebrafish, with the number of individuals ranging from 2 to 100 (see Methods).330\nPrevious versions of idtracker.ai (v4 and v5) can resort to protocol 3 for tracking, a method that331\ncan take days to process more complex videos but is necessary when protocol 2 fails. Similarly,332\nTRex, lacking an equivalent of protocol 3, can fail to track certain videos, leading to missing accuracy333\noutputs (Figure 2—figure Supplement 2).334\nTo estimate the accuracy and tracking time that a standard user might experience, we simulate335\na realistic user workflow. This simulation accounts for the possibility that the software may fail to336\ntrack the video, prompting the user to try again with a slightly different parameter configuration,337\nup to a certain number of attempts.338\nThe user is given up to 5 attempts to successfully track a video. Attempts are sampled from a339\nprecomputed dataset of tracking runs. Accuracy is taken from the first successful run. The reported340\ntracking time is the sum of the time taken by that successful run and all preceding failed attempts.341\nIn cases where all attempts fail, accuracy is determined by protocol 3 (in v4 and v5 of idtracker.ai),342\nand tracking time includes the time required for protocol 3 plus the total time of all failed attempts.343\nThis sampling process is repeated 10,000 times per software and video to obtain statistically robust344\nestimates of the tracking times and accuracies. Figure 2 and Figure 2—figure Supplement 1 report345\nthe median accuracies, without and with crossings, respectively, and tracking times. Supplemen-346\ntary Table 1, Supplementary Table 2, Supplementary Table 3, and Supplementary Table 4 present347\nthe median, mean, and the 20 and 80 percentiles in v4, v5, v6 and TRex respectively.348\nDataset of tracking runs349\nTo build the dataset of tracking runs we used for the benchmark of accuracies and times, we de-350\nfine input parameters through each software’s graphical interface. Fixed parameters (e.g., num-351\nber of animals, regions of interest) are held constant, while those with multiple valid values are352\ntreated as variable, with their ranges annotated. In idtracker.ai, the variable parameter is the353\nintensity_threshold, whereas in TRex, the variable parameters arethreshold and track_max_speed.354\n10 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\nTracking is repeated for each video and software until either 5 successful runs or 35 total runs355\nare reached. For the original version of idtracker.ai, this is limited to 3 successful runs or 7 total runs356\ndue to significantly longer tracking times. In successful runs, both accuracy and tracking time are357\nrecorded. In failed runs, when idtracker.ai defaults to protocol 3 or TRex fails to output identities358\n(see Figure 2—figure Supplement 2 ), only the time until failure is recorded. For previous idtracker.ai359\nversions (v4 and v5), failure time corresponds to the time until the software switched to protocol360\n3.361\nEach tracking run is conducted by randomly sampling values for the variable parameters from362\nthe annotated ranges and executing the full tracking process. To ensure a fair comparison, TGrabs363\nis included when running TRex, graphical interfaces are always disabled at runtime to maximize364\nperformance, and output_interpolate_positions is enabled in TRex.365\nReferences366\nAwasthi P, Dikkala N, Kamath P, Do More Negative Samples Necessarily Hurt in Contrastive Learning?; 2022.367\nhttps://arxiv.org/abs/2205.01789.368\nBengio Y, Courville A, Vincent P. Representation Learning: A Review and New Perspectives. IEEE369\nTrans Pattern Anal Mach Intell. 2013 Aug; 35(8):1798–1828. https://doi.org/10.1109/TPAMI.2013.50, doi:370\n10.1109/TPAMI.2013.50.371\nBernardes RC, Lima MAP, Guedes RNC, da Silva CB, Martins GF. Ethoflow: Computer Vision and Artificial372\nIntelligence-Based Software for Automatic Behavior Analysis. Sensors. 2021; 21(9). https://www.mdpi.com/373\n1424-8220/21/9/3237, doi: 10.3390/s21093237.374\nBiderman D, Whiteway MR, Hurwitz C, Greenspan N, Lee RS, Vishnubhotla A, Warren R, Pedraja F, Noone D,375\nSchartner MM, Huntenburg JM, Khanal A, Meijer GT, Noel JP, Pan-Vazquez A, Socha KZ, Urai AE, Cunningham376\nJP, Sawtell NB, Paninski L. Lightning Pose: improved animal pose estimation via semi-supervised learning,377\nBayesian ensembling and cloud-native open-source tools. Nature Methods. 2024 July; 21(7):1316–1328. doi:378\n10.1038/s41592-024-02319-1.379\nBranson K, Robie AA, Bender J, Perona P, Dickinson MH. High-throughput ethomics in large groups380\nof Drosophila. Nature Methods. 2009; 6(6):451–457. https://doi.org/10.1038/nmeth.1328, doi:381\n10.1038/nmeth.1328.382\nChen T, Kornblith S, Norouzi M, Hinton G. A simple framework for contrastive learning of visual representations.383\nIn: Proceedings of the 37th International Conference on Machine Learning ICML’20, JMLR.org; 2020. .384\nChen T, Kornblith S, Swersky K, Norouzi M, Hinton G. Big self-supervised models are strong semi-supervised385\nlearners. In: Proceedings of the 34th International Conference on Neural Information Processing Systems NIPS386\n’20, Red Hook, NY, USA: Curran Associates Inc.; 2020. .387\nChen Z, Zhang R, Fang HS, Zhang YE, Bal A, Zhou H, Rock RR, Padilla-Coreano N, Keyes LR, Zhu H, Li YL, Komiyama388\nT, Tye KM, Lu C. AlphaTracker: a multi-animal tracking and behavioral analysis tool. Frontiers in Behavioral389\nNeuroscience. 2023; 17. https://www.frontiersin.org/journals/behavioral-neuroscience/articles/10.3389/fnbeh.390\n2023.1111908, doi: 10.3389/fnbeh.2023.1111908.391\nChiara V , Kim SY. AnimalTA: A highly flexible and easy-to-use program for tracking and analysing animal392\nmovement in different environments. Methods in Ecology and Evolution. 2023; 14(7):1699–1707. https:393\n//besjournals.onlinelibrary.wiley.com/doi/abs/10.1111/2041-210X.14115, doi: https://doi.org/10.1111/2041-394\n210X.14115.395\nChopra S, Hadsell R, LeCun Y. Learning a similarity metric discriminatively, with application to face verification.396\nIn: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 1; 2005. p.397\n539–546 vol. 1. doi: 10.1109/CVPR.2005.202.398\nDong X, Shen J. Triplet Loss in Siamese Network for Object Tracking. In: Ferrari V, Hebert M, Sminchisescu C,399\nWeiss Y, editors. Computer Vision – ECCV 2018 Cham: Springer International Publishing; 2018. p. 472–488.400\nEricsson L, Gouk H, Loy CC, Hospedales TM. Self-Supervised Representation Learning: Introduction, advances,401\nand challenges. IEEE Signal Processing Magazine. 2022; 39(3):42–62. doi: 10.1109/MSP.2021.3134634.402\n11 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\nGuo S, Xu P, Miao Q, Shao G, Chapman CA, Chen X, He G, Fang D, Zhang H, Sun Y, Shi Z, Li B. Automatic Identi-403\nfication of Individual Primates with Deep Learning Techniques. iScience. 2020; 23(8):101412. https://www.404\nsciencedirect.com/science/article/pii/S2589004220306027, doi: https://doi.org/10.1016/j.isci.2020.101412.405\nHe K, Zhang X, Ren S, Sun J. Deep Residual Learning for Image Recognition. In: 2016 IEEE Conference on Computer406\nVision and Pattern Recognition (CVPR); 2016. p. 770–778. doi: 10.1109/CVPR.2016.90.407\nKAYA M, BİLGE HS. Deep Metric Learning: A Survey. Symmetry. 2019; 11(9). https://www.mdpi.com/2073-8994/408\n11/9/1066, doi: 10.3390/sym11091066.409\nKhosla P, Teterwak P, Wang C, Sarna A, Tian Y, Isola P, Maschinot A, Liu C, Krishnan D, Supervised Contrastive410\nLearning; 2021. https://arxiv.org/abs/2004.11362.411\nKingma DP, Ba J, Adam: A Method for Stochastic Optimization; 2017. https://arxiv.org/abs/1412.6980.412\nLauer J, Zhou M, Ye S, Menegas W, Schneider S, Nath T, Rahman MM, Santo VD, Soberanes D, Feng G, Murthy413\nVN, Lauder G, Dulac C, Mathis MW, Mathis A. Multi-animal pose estimation, identification and tracking with414\nDeepLabCut. Nature Methods. 2022 April; 19(4):496–504. https://doi.org/10.1038/s41592-022-01443-0 , doi:415\n10.1038/s41592-022-01443-0.416\nLiu S, Han L, Liu X, Ren J, Wang F, YingLiu, Lin Y, FishMOT: A Simple and Effective Method for Fish Tracking Based417\non IoU Matching; 2023. https://arxiv.org/abs/2309.02975.418\nvan der Maaten L, Hinton G. Visualizing Data using t-SNE. Journal of Machine Learning Research. 2008;419\n9(86):2579–2605. http://jmlr.org/papers/v9/vandermaaten08a.html.420\nPereira TD, Tabris N, Matsliah A, Turner DM, Li J, Ravindranath S, Papadoyannis ES, Normand E, Deutsch DS,421\nWang ZY, McKenzie-Smith GC, Mitelut CC, Castro MD, D’Uva J, Kislin M, Sanes DH, Kocher SD, Wang SSH,422\nFalkner AL, Shaevitz JW, et al. SLEAP: A deep learning system for multi-animal pose tracking. Nature Methods.423\n2022 April; 19(4):486–495. https://doi.org/10.1038/s41592-022-01426-1, doi: 10.1038/s41592-022-01426-1.424\nPérez-Escudero A, Vicente-Page J, Hinz RC, Arganda S, De Polavieja GG. idTracker: tracking individuals in a425\ngroup by automatic identification of unmarked animals. Nature methods. 2014; 11(7):743–748.426\nPlum F. OmniTrax: A deep learning-driven multi-animal tracking and pose-estimation add-on for427\nBlender. Journal of Open Source Software. 2024; 9(95):5549. https://doi.org/10.21105/joss.05549, doi:428\n10.21105/joss.05549.429\nRomero-Ferrero F , Bergomi MG, Hinz RC, Heras FJ, De Polavieja GG. Idtracker. ai: tracking all individuals in430\nsmall or large collectives of unmarked animals. Nature methods. 2019; 16(2):179–182.431\nRomero-Ferrero F, Heras FJ, Rance D, de Polavieja GG. A study of transfer of information in animal collectives432\nusing deep learning tools. Philosophical Transactions of the Royal Society B. 2023; 378(1874):20220073.433\nRousseeuw PJ. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of434\nComputational and Applied Mathematics. 1987; 20:53–65. https://www.sciencedirect.com/science/article/pii/435\n0377042787901257, doi: https://doi.org/10.1016/0377-0427(87)90125-7.436\nRösch PJ, Oswald N, Geierhos M, Libovický J, Enhancing Conceptual Understanding in Multimodal Contrastive437\nLearning through Hard Negative Samples; 2024. https://arxiv.org/abs/2403.02875.438\nSchroff F, Kalenichenko D, Philbin J. FaceNet: A unified embedding for face recognition and clustering. In: 2015439\nIEEE Conference on Computer Vision and Pattern Recognition (CVPR) IEEE; 2015.http://dx.doi.org/10.1109/CVPR.440\n2015.7298682, doi: 10.1109/cvpr.2015.7298682.441\nSculley D. Web-scale k-means clustering. In: Proceedings of the 19th International Conference on World Wide Web442\nWWW ’10, New York, NY, USA: Association for Computing Machinery; 2010. p. 1177–1178. https://doi.org/10.443\n1145/1772690.1772862, doi: 10.1145/1772690.1772862.444\nSegalin C, Williams J, Karigo T, Hui M, Zelikowsky M, Sun JJ, Perona P, Anderson DJ, Kennedy A. The Mouse Action445\nRecognition System (MARS) software pipeline for automated analysis of social behaviors in mice. eLife. 2021446\nnov; 10:e63720. https://doi.org/10.7554/eLife.63720, doi: 10.7554/eLife.63720.447\nTang G, Han Y, Sun X, Zhang R, Han M, Liu Q, Wei P. Anti-drift pose tracker (ADPT): A transformer-based network448\nfor robust animal pose estimation cross-species. eLife. 2025 Mar; http://dx.doi.org/10.7554/eLife.95709.2,449\ndoi: 10.7554/elife.95709.2.450\n12 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\nTeh CH, Chin RT. On the detection of dominant points on digital curve. Pattern Analysis and Machine Intelli-451\ngence, IEEE Transactions on. 1989 09; 11:859 – 872. doi: 10.1109/34.31447.452\nWalter T, Couzin ID. TRex, a fast multi-animal tracking system with markerless identification, and 2D esti-453\nmation of posture and visual fields. eLife. 2021 feb; 10:e64000. https://doi.org/10.7554/eLife.64000, doi:454\n10.7554/eLife.64000.455\nWang Z, Zheng L, Liu Y, Li Y, Wang S. Towards Real-Time Multi-Object Tracking. In: Computer Vision – ECCV 2020:456\n16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI Berlin, Heidelberg: Springer-457\nVerlag; 2020. p. 107–122. https://doi.org/10.1007/978-3-030-58621-8_7, doi: 10.1007/978-3-030-58621-8_7.458\nXing E , Jordan M, Russell SJ, Ng A. Distance Metric Learning with Application to Clustering with459\nSide-Information. In: Becker S, Thrun S, Obermayer K, editors. Advances in Neural Information460\nProcessing Systems , vol. 15 MIT Press; 2002. https://proceedings.neurips.cc/paper_files/paper/2002/file/461\nc3e4035af2a1cde9f21e1ae1951ac80b-Paper.pdf .462\nYang F , Chang X, Dang C, Zheng Z, Sakti S, Nakamura S, Wu Y. ReMOTS: Self-Supervised Refining Multi-463\nObject Tracking and Segmentation. ArXiv. 2020; abs/2007.03200. https://api.semanticscholar.org/CorpusID:464\n220404263.465\nSupplementary tables466\n13 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\nAccuracy\nwithout crossings (%) Accuracy with crossings (%) Tracking time\nName Median\nMean 20-80 percentiles Median\nMean 20-80 percentiles Median\nMean 20-80 percentiles\ndrosophila_6 99.135\n99.135 98.536 - 99.734 99.055\n99.055 98.514 - 99.596 0:22:27\n0:22:27 0:21:39 - 0:23:15\ndrosophila_8 98.955\n98.663 98.369 - 98.955 98.603\n98.310 98.015 - 98.603 0:44:47\n0:48:50 0:30:51 - 1:15:10\ndrosophila_10 99.003\n99.003 one point only 99.004\n99.004 one point only 6:28:25\n6:28:25 6:18:34 - 6:38:12\ndrosophila_38 99.822\n99.756 99.688 - 99.822 99.699\n99.618 99.535 - 99.699 6:55:15\n6:15:08 4:16:03 - 7:23:33\ndrosophila_59 99.489\n99.489 99.489 - 99.489 99.441\n99.441 99.441 - 99.441 1\nday, 3h 1 day, 3h 9:41:00 - 1 day, 20h\ndrosophila_60 99.914\n99.914 one point only 99.848\n99.848 one point only 4\ndays, 18h 4 days, 18h 4 days, 16h - 4 days, 21h\ndrosophila_72 99.980\n99.965 99.949 - 99.980 99.960\n99.948 99.934 - 99.960 11:35:20\n13:00:53 7:00:26 - 16:42:43\ndrosophila_80 99.319\n99.319 one point only 99.220\n99.220 one point only 6\ndays, 15h 6 days, 15h 6 days, 15h - 6 days, 16h\ndrosophila_100_1 96.605\n96.605 one point only 96.344\n96.344 one point only 8\ndays, 1h 8 days, 1h 8 days, 1h - 8 days, 2h\ndrosophila_100_2 95.358\n95.358 one point only 95.314\n95.314 one point only 9\ndays, 22h 9 days, 22h 9 days, 21h - 9 days, 22h\ndrosophila_100_3 54.021\n54.021 one point only 53.758\n53.758 one point only 13\ndays, 14h 13 days, 14h 13 days, 13h - 13 days, 15h\nmice_2_1 98.858\n98.851 98.845 - 98.858 97.646\n97.328 97.006 - 97.646 0:08:36\n0:07:12 0:05:47 - 0:08:36\nmice_2_2 99.039\n98.980 98.919 - 99.039 97.998\n97.999 97.998 - 98.000 0:08:08\n0:07:19 0:06:30 - 0:08:08\nmice_2_3 95.140\n97.528 95.140 - 99.953 94.695\n96.737 94.695 - 98.810 0:07:03\n0:06:01 0:04:58 - 0:07:03\nmice_2_4 99.924\n99.947 99.924 - 99.971 99.919\n99.942 99.919 - 99.966 0:06:19\n0:05:24 0:04:28 - 0:06:19\nmice_4_1 98.940\n98.711 98.480 - 98.940 98.944\n98.613 98.278 - 98.944 0:14:27\n0:12:37 0:10:45 - 0:14:27\nmice_4_2 92.977\n90.780 88.553 - 92.977 93.046\n90.865 88.654 - 93.046 0:13:58\n0:14:56 0:13:58 - 0:15:55\nzebrafish_5 99.922\n99.652 99.375 - 99.922 99.910\n99.617 99.317 - 99.910 0:14:36\n0:11:40 0:08:40 - 0:14:36\nzebrafish_7 99.987\n99.976 99.965 - 99.987 99.946\n99.939 99.932 - 99.946 0:13:59\n0:15:38 0:13:59 - 0:17:22\nzebrafish_10_1 99.999\n99.999 99.999 - 99.999 99.994\n99.996 99.994 - 99.998 0:44:31\n0:44:12 0:42:42 - 0:45:24\nzebrafish_10_2 99.975\n99.975 99.975 - 99.976 99.953\n99.959 99.953 - 99.965 0:47:27\n0:47:23 0:47:18 - 0:47:27\nzebrafish_10_3 99.983\n99.984 99.983 - 99.984 99.982\n99.976 99.971 - 99.982 0:45:08\n0:44:05 0:43:02 - 0:45:08\nzebrafish_10_4 99.930 99.955 99.930 - 99.980 99.929 99.954 99.929 - 99.979 0:20:05 0:19:19 0:18:32 - 0:20:05\nzebrafish_20 99.994\n99.996 99.994 - 99.997 99.963\n99.960 99.957 - 99.963 1:01:06\n0:56:15 0:51:24 - 1:01:06\nzebrafish_60_1 98.571\n98.622 98.571 - 98.673 98.575\n98.626 98.575 - 98.676 2:44:27\n2:54:04 2:44:27 - 3:03:42\nzebrafish_60_2 99.809\n99.881 99.809 - 99.955 99.783\n99.857 99.783 - 99.934 3:58:33\n3:02:13 2:04:41 - 3:58:33\nzebrafish_60_3 99.982\n99.980 99.979 - 99.982 99.976\n99.976 99.975 - 99.976 2:27:01\n2:34:41 2:27:01 - 2:42:29\nzebrafish_80_1 99.770\n99.848 99.703 - 99.997 99.720\n99.818 99.661 - 99.983 8:34:31\n12:25:34 6:29:28 - 11:47:52\nzebrafish_80_2 99.995\n99.949 99.901 - 99.995 99.988\n99.943 99.896 - 99.988 4:21:16\n4:19:10 4:17:00 - 4:21:16\nzebrafish_80_3 99.998\n99.946 99.894 - 99.998 99.983\n99.933 99.882 - 99.983 3:37:08\n4:21:29 3:37:08 - 5:05:52\nzebrafish_100_1 93.929\n96.881 93.929 - 99.862 93.467\n96.631 93.467 - 99.825 11:55:32\n12:03:34 10:12:42 - 13:06:53\nzebrafish_100_2 99.962\n99.965 99.962 - 99.969 99.953\n99.955 99.953 - 99.958 5:44:14\n5:35:00 5:25:35 - 5:44:14\nzebrafish_100_3 99.933\n99.906 99.880 - 99.933 99.922\n99.897 99.870 - 99.922 6:20:55\n6:01:34 5:41:37 - 6:20:55\nSupplementary\nTable 1. Performance of original idtracker.ai (v4) in the benchmark.\n14 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\nAccuracy\nwithout crossings (%) Accuracy with crossings (%) Tracking time\nName Median\nMean 20-80 percentiles Median\nMean 20-80 percentiles Median\nMean 20-80 percentiles\ndrosophila_6 98.392\n98.238 97.285 - 98.402 98.111\n98.109 97.331 - 98.127 0:04:28\n0:04:37 0:03:37 - 0:05:13\ndrosophila_8 98.955\n99.165 98.953 - 98.999 98.603\n98.861 98.585 - 98.688 0:04:23\n0:04:59 0:03:39 - 0:05:14\ndrosophila_10 99.903\n98.618 96.103 - 99.903 99.903\n98.618 96.103 - 99.903 1:36:38\n1:15:26 0:30:41 - 1:42:51\ndrosophila_38 99.994\n99.974 99.987 - 99.995 99.932\n99.909 99.916 - 99.952 0:30:22\n0:31:23 0:23:22 - 0:37:52\ndrosophila_59 99.994\n99.867 99.724 - 100.000 99.971\n99.855 99.716 - 99.995 1:43:19\n2:01:38 1:12:12 - 2:29:09\ndrosophila_60 100.000\n99.932 99.774 - 100.000 100.000\n99.908 99.654 - 100.000 1\nday, 14h 1 day, 5h 3:51:31 - 1 day, 15h\ndrosophila_72 99.980\n99.985 99.979 - 99.993 99.964\n99.969 99.961 - 99.980 1:06:10\n1:23:35 0:46:16 - 1:42:32\ndrosophila_80 99.897\n99.904 99.877 - 99.925 99.726\n99.724 99.715 - 99.741 2:11:25\n4:56:26 1:24:11 - 3:23:58\ndrosophila_100_1 99.895\n99.830 99.647 - 99.945 99.723\n99.659 99.492 - 99.749 2:52:21\n10:45:30 1:45:07 - 1 day, 2h\ndrosophila_100_2 98.070\n98.070 one point only 98.030\n98.030 one point only 1\nday, 18h 1 day, 18h 1 day, 17h - 1 day, 19h\ndrosophila_100_3 99.760\n99.735 99.599 - 99.770 99.641\n99.613 99.471 - 99.645 2:45:22\n8:04:32 1:39:37 - 4:11:32\nmice_2_1 99.640\n99.639 99.579 - 99.724 98.250\n98.225 97.735 - 98.541 0:02:15\n0:02:21 0:02:14 - 0:02:29\nmice_2_2 99.176\n99.162 99.053 - 99.246 97.881\n97.970 97.687 - 98.493 0:02:50\n0:02:48 0:02:28 - 0:03:01\nmice_2_3 99.883\n98.795 94.339 - 100.000 98.633\n97.669 93.397 - 99.124 0:03:05\n0:03:08 0:02:54 - 0:03:39\nmice_2_4 99.937\n99.935 99.863 - 100.000 99.871\n99.868 99.828 - 99.904 0:02:04\n0:02:07 0:02:01 - 0:02:25\nmice_4_1 99.765\n99.593 99.291 - 99.969 99.640\n99.324 98.873 - 99.756 0:04:36\n0:04:58 0:04:34 - 0:06:53\nmice_4_2 93.117\n92.985 92.294 - 93.128 93.058\n92.906 92.287 - 93.087 0:07:12\n0:08:38 0:04:37 - 0:12:06\nzebrafish_5 99.998\n99.997 99.998 - 99.999 99.984\n99.984 99.983 - 99.984 0:01:51\n0:01:49 0:01:40 - 0:02:03\nzebrafish_7 99.963\n99.539 98.800 - 99.967 99.909\n99.505 98.776 - 99.938 0:02:29\n0:03:11 0:02:23 - 0:05:05\nzebrafish_10_1 100.000\n100.000 100.000 - 100.000 100.000\n100.000 99.999 - 100.000 0:08:50\n0:08:47 0:07:53 - 0:09:30\nzebrafish_10_2 99.999\n99.952 99.763 - 100.000 99.989\n99.941 99.747 - 99.991 0:09:24\n0:09:51 0:09:21 - 0:11:48\nzebrafish_10_3 100.000\n99.713 99.999 - 100.000 99.994\n99.712 99.993 - 99.996 0:08:54\n0:09:15 0:08:32 - 0:08:54\nzebrafish_10_4 99.999 99.996 99.986 - 99.999 99.997 99.994 99.984 - 99.999 0:02:52 0:02:55 0:02:44 - 0:02:58\nzebrafish_20 99.997\n99.992 99.992 - 99.999 99.901\n99.906 99.898 - 99.932 0:08:29\n0:08:38 0:07:42 - 0:10:46\nzebrafish_60_1 99.999\n99.999 99.997 - 100.000 99.994\n99.994 99.992 - 99.994 0:21:14\n0:21:33 0:20:34 - 0:24:17\nzebrafish_60_2 99.978\n99.894 99.947 - 99.999 99.957\n99.878 99.924 - 99.992 0:37:40\n0:33:28 0:20:44 - 0:43:23\nzebrafish_60_3 99.967\n99.965 99.924 - 99.998 99.929\n99.934 99.888 - 99.960 0:31:47\n0:35:43 0:30:53 - 0:43:33\nzebrafish_80_1 99.999\n99.993 99.999 - 99.999 99.982\n99.977 99.982 - 99.984 1:01:10\n1:27:36 0:47:00 - 1:22:16\nzebrafish_80_2 99.978\n99.970 99.922 - 99.998 99.970\n99.962 99.915 - 99.989 0:30:23\n0:29:19 0:27:24 - 0:30:27\nzebrafish_80_3 100.000\n100.000 100.000 - 100.000 99.985\n99.985 99.978 - 99.989 0:30:52\n0:54:43 0:27:00 - 1:29:25\nzebrafish_100_1 99.996\n99.988 99.966 - 99.996 99.956\n99.953 99.938 - 99.958 1:46:04\n1:48:59 1:30:47 - 2:09:56\nzebrafish_100_2 99.998\n99.976 99.909 - 99.998 99.985\n99.967 99.902 - 99.990 0:45:27\n0:46:22 0:36:17 - 0:50:22\nzebrafish_100_3 99.999\n99.984 99.999 - 100.000 99.989\n99.974 99.986 - 99.991 0:59:47\n1:03:35 0:33:26 - 1:06:14\nSupplementary\nTable 2. Performance of optimized v4 (v5) in the benchmark.\n15 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\nAccuracy\nwithout crossings (%) Accuracy with crossings (%) Tracking time\nName Median\nMean 20-80 percentiles Median\nMean 20-80 percentiles Median\nMean 20-80 percentiles\ndrosophila_6 99.988\n99.914 99.976 - 99.994 99.950\n99.871 99.941 - 99.972 0:02:12\n0:02:15 0:02:10 - 0:02:17\ndrosophila_8 100.000\n99.999 99.995 - 100.000 99.922\n99.864 99.638 - 99.927 0:01:53\n0:01:50 0:01:39 - 0:01:54\ndrosophila_10 98.968\n99.167 98.968 - 98.969 98.969\n99.168 98.969 - 98.970 0:10:34\n0:10:48 0:10:32 - 0:11:45\ndrosophila_38 99.992\n99.975 99.992 - 99.993 99.876\n99.871 99.823 - 99.920 0:12:04\n0:12:01 0:11:46 - 0:12:09\ndrosophila_59 99.989\n99.940 99.985 - 100.000 99.961\n99.915 99.942 - 99.991 0:36:35\n0:44:24 0:27:39 - 0:45:42\ndrosophila_60 99.964\n99.152 99.963 - 99.998 99.896\n99.076 99.878 - 99.897 0:22:37\n0:27:02 0:13:14 - 0:23:23\ndrosophila_72 99.996\n99.997 99.995 - 99.999 99.984\n99.982 99.980 - 99.984 0:33:35\n0:27:40 0:18:45 - 0:33:43\ndrosophila_80 99.975\n99.972 99.967 - 99.978 99.878\n99.879 99.877 - 99.879 1:54:15\n1:51:19 1:48:36 - 1:54:33\ndrosophila_100_1 99.895\n99.911 99.867 - 99.940 99.752\n99.781 99.731 - 99.764 1:08:56\n1:01:28 0:48:35 - 1:10:18\ndrosophila_100_2 99.998\n99.558 97.800 - 100.000 99.971\n99.530 97.775 - 99.976 0:32:32\n0:41:57 0:31:19 - 1:19:35\ndrosophila_100_3 99.740\n99.376 98.688 - 99.787 99.618\n99.236 98.492 - 99.686 2:17:32\n2:16:44 1:52:39 - 2:29:40\nmice_2_1 99.850\n99.853 99.837 - 99.883 99.100\n99.145 99.012 - 99.255 0:03:15\n0:03:13 0:03:10 - 0:03:17\nmice_2_2 99.886\n99.853 99.818 - 99.922 98.680\n98.536 98.406 - 98.995 0:03:11\n0:03:14 0:03:09 - 0:03:13\nmice_2_3 100.000\n100.000 100.000 - 100.000 98.932\n98.951 98.805 - 99.071 0:03:03\n0:03:02 0:02:47 - 0:03:33\nmice_2_4 100.000\n100.000 100.000 - 100.000 99.945\n99.953 99.938 - 99.971 0:02:53\n0:02:51 0:02:34 - 0:03:03\nmice_4_1 99.893\n99.850 99.640 - 99.940 99.716\n99.685 99.488 - 99.770 0:03:19\n0:03:11 0:03:03 - 0:03:26\nmice_4_2 99.495\n99.538 99.333 - 99.832 99.241\n99.318 99.170 - 99.700 0:04:09\n0:04:16 0:03:39 - 0:04:58\nzebrafish_5 99.998\n99.997 99.997 - 99.998 99.984\n99.980 99.968 - 99.985 0:01:23\n0:01:23 0:01:21 - 0:01:30\nzebrafish_7 99.965\n99.965 99.963 - 99.966 99.916\n99.914 99.903 - 99.921 0:02:02\n0:02:05 0:01:56 - 0:02:13\nzebrafish_10_1 100.000\n100.000 100.000 - 100.000 100.000\n100.000 100.000 - 100.000 0:08:37\n0:08:50 0:08:32 - 0:09:14\nzebrafish_10_2 100.000\n100.000 100.000 - 100.000 99.992\n99.991 99.992 - 99.994 0:09:47\n0:09:48 0:09:40 - 0:10:03\nzebrafish_10_3 100.000\n99.999 99.998 - 100.000 99.997\n99.996 99.991 - 99.999 0:11:32\n0:11:34 0:11:25 - 0:12:02\nzebrafish_10_4 99.998 99.993 99.993 - 99.999 99.990 99.990 99.987 - 99.996 0:02:08 0:02:06 0:01:56 - 0:02:09\nzebrafish_20 99.999\n99.995 99.997 - 99.999 99.914\n99.913 99.880 - 99.933 0:03:47\n0:03:42 0:03:38 - 0:03:50\nzebrafish_60_1 99.963\n99.978 99.963 - 100.000 99.960\n99.974 99.960 - 99.997 0:20:40\n0:21:16 0:20:19 - 0:22:24\nzebrafish_60_2 99.994\n99.973 99.978 - 99.999 99.975\n99.957 99.956 - 99.992 0:34:05\n0:31:39 0:31:36 - 0:34:07\nzebrafish_60_3 99.999\n99.984 99.998 - 99.999 99.965\n99.950 99.963 - 99.965 0:32:51\n0:33:24 0:32:45 - 0:36:21\nzebrafish_80_1 99.998\n99.998 99.997 - 99.999 99.987\n99.988 99.987 - 99.989 0:30:15\n0:34:42 0:29:02 - 0:42:19\nzebrafish_80_2 99.978\n99.981 99.955 - 99.996 99.974\n99.978 99.952 - 99.994 0:29:50\n0:29:29 0:27:30 - 0:31:02\nzebrafish_80_3 99.998\n99.967 99.994 - 100.000 99.983\n99.956 99.983 - 99.990 0:30:39\n0:34:28 0:28:39 - 0:53:04\nzebrafish_100_1 99.986\n99.982 99.966 - 99.997 99.960\n99.951 99.938 - 99.960 1:31:06\n1:31:54 1:30:21 - 1:38:11\nzebrafish_100_2 99.910\n99.930 99.832 - 99.997 99.905\n99.924 99.825 - 99.991 0:37:33\n0:41:29 0:35:35 - 0:59:00\nzebrafish_100_3 99.975\n99.956 99.842 - 99.991 99.969\n99.947 99.831 - 99.982 0:37:47\n0:46:11 0:36:05 - 1:05:10\nSupplementary\nTable 3. Performance of new idtracker.ai (v6) in the benchmark.\n16 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\nAccuracy\nwithout crossings (%) Accuracy with crossings (%) Tracking time\nName Median\nMean 20-80 percentiles Median\nMean 20-80 percentiles Median\nMean 20-80 percentiles\ndrosophila_6 99.940\n99.917 99.831 - 99.973 99.874\n99.844 99.682 - 99.962 0:26:55\n0:27:18 0:21:45 - 0:32:45\ndrosophila_8 94.665\n94.253 88.727 - 99.994 94.680\n94.128 88.392 - 99.988 0:31:08\n0:31:25 0:28:55 - 0:32:19\ndrosophila_10 97.934\n98.330 97.899 - 97.940 97.936\n98.331 97.901 - 97.942 0:37:17\n0:37:05 0:34:00 - 0:37:23\ndrosophila_38 99.915\n99.839 99.666 - 99.916 99.807\n99.680 99.371 - 99.824 1:34:29\n1:20:44 1:25:16 - 1:42:35\ndrosophila_59 99.862\n99.357 97.554 - 99.990 99.823\n99.342 97.556 - 99.982 0:52:17\n0:59:53 0:22:28 - 1:27:04\ndrosophila_60 99.998\n99.987 99.943 - 99.998 99.998\n99.987 99.943 - 99.998 1:05:04\n1:10:01 0:58:33 - 1:14:41\ndrosophila_72 97.793\n96.412 91.004 - 99.971 97.789\n96.405 90.994 - 99.968 1:02:38\n1:16:24 0:49:20 - 1:49:13\ndrosophila_80 97.643\n97.742 95.802 - 99.230 97.618\n97.702 95.735 - 99.183 1:36:49\n1:38:19 1:20:36 - 1:49:03\ndrosophila_100_1 99.955\n97.355 94.311 - 99.959 99.954\n97.336 94.256 - 99.958 2:08:46\n1:48:20 0:55:32 - 2:21:53\ndrosophila_100_2 74.957\n73.651 60.365 - 85.480 74.930\n73.642 60.371 - 85.469 0:49:40\n0:50:59 0:44:09 - 0:53:05\ndrosophila_100_3 94.200\n93.884 92.066 - 96.935 94.123\n93.796 92.021 - 96.836 1:02:03\n1:13:05 1:01:52 - 1:19:57\nmice_2_1 99.683\n99.419 99.611 - 99.687 98.515\n98.122 97.791 - 98.628 0:05:46\n0:05:40 0:05:15 - 0:05:52\nmice_2_2 99.118\n98.332 97.894 - 99.735 96.376\n94.921 92.830 - 96.941 0:04:06\n0:04:05 0:04:01 - 0:04:22\nmice_2_3 99.775\n99.301 98.821 - 99.801 97.601\n96.930 95.688 - 97.788 0:04:03\n0:04:04 0:03:55 - 0:04:08\nmice_2_4 95.925\n95.711 95.793 - 96.043 95.075\n94.813 94.757 - 95.403 0:02:45\n0:03:25 0:02:03 - 0:04:52\nmice_4_1 99.964\n99.959 99.940 - 99.965 99.577\n99.574 99.559 - 99.581 0:18:11\n0:18:10 0:17:39 - 0:18:31\nmice_4_2 93.100\n92.228 88.715 - 93.329 92.680\n91.778 88.091 - 93.024 0:20:34\n0:18:34 0:07:47 - 0:23:14\nzebrafish_5 100.000\n100.000 100.000 - 100.000 100.000\n100.000 100.000 - 100.000 0:09:57\n0:09:48 0:09:17 - 0:10:39\nzebrafish_7 99.982\n99.987 99.981 - 99.996 99.981\n99.986 99.979 - 99.996 0:08:37\n0:09:02 0:06:41 - 0:11:34\nzebrafish_10_1 99.864\n99.778 99.858 - 99.912 99.852\n99.772 99.848 - 99.910 0:13:57\n0:13:45 0:12:26 - 0:14:43\nzebrafish_10_2 99.972\n99.926 99.774 - 99.985 99.968\n99.923 99.773 - 99.984 0:22:07\n0:21:59 0:21:03 - 0:23:23\nzebrafish_10_3 99.869\n99.643 98.680 - 99.998 99.861\n99.636 98.679 - 99.997 0:37:51\n0:31:47 0:19:06 - 0:41:44\nzebrafish_10_4 99.998 99.996 99.998 - 99.998 99.996 99.994 99.995 - 99.998 0:14:12 0:14:23 0:12:36 - 0:15:16\nzebrafish_20 99.942\n99.872 99.717 - 99.988 99.845\n99.842 99.717 - 99.967 0:43:02\n0:48:25 0:36:57 - 1:03:46\nzebrafish_60_1 99.887\n99.919 99.885 - 99.964 99.881\n99.914 99.878 - 99.963 1:43:58\n1:43:10 1:37:48 - 1:46:02\nzebrafish_60_2 99.541\n98.727 95.295 - 99.674 99.520\n98.710 95.268 - 99.657 1:30:15\n1:20:03 0:42:10 - 1:38:18\nzebrafish_60_3 99.229\n99.356 99.091 - 99.795 99.223\n99.352 99.089 - 99.790 1:39:17\n1:31:01 0:59:17 - 1:45:48\nzebrafish_80_1 99.655\n99.697 99.628 - 99.657 99.644\n99.686 99.619 - 99.645 1:23:48\n1:26:30 1:20:59 - 1:30:03\nzebrafish_80_2 99.689\n99.688 99.683 - 99.746 99.688\n99.685 99.681 - 99.744 1:38:38\n1:36:18 1:26:50 - 1:42:06\nzebrafish_80_3 99.789\n99.653 99.719 - 99.977 99.781\n99.643 99.712 - 99.971 1:36:15\n1:34:16 1:33:04 - 1:39:45\nzebrafish_100_1 99.565\n99.059 98.559 - 99.731 99.547\n99.028 98.540 - 99.703 1:36:42\n1:19:48 0:59:43 - 1:41:22\nzebrafish_100_2 97.987\n98.316 97.750 - 98.179 97.975\n98.306 97.734 - 98.169 1:06:55\n1:10:20 1:04:29 - 1:16:39\nzebrafish_100_3 99.293\n98.700 99.178 - 99.836 99.285\n98.689 99.170 - 99.832 1:15:39\n1:13:43 1:11:46 - 1:42:34\nSupplementary\nTable 4. Performance of TRex in the benchmark.\n17 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\nAppendix 1467\nPreliminary concepts468\nImage-based tracking relies on identifying individuals through their visual features. The\nprocess begins by distinguishing the pixels corresponding to animals from those of the\nbackground. Let 𝑏 represent a blob that is distinct from the background. For each blob\n𝑏 segmented from a video, an identification image 𝐼𝑏 is generated by first taking the mini-\nmal bounding box image around 𝑏 and then converting all pixels in 𝐼𝑏 that do not belong to\n𝑏 to black. The blob within 𝐼𝑏 is then rotated so that its first principal component is aligned\nat a 𝜋\n4 angle to the x-axis and, finally, the image is cropped to a specified square size suitable\nfor batch processing.\n469\n470\n471\n472\n473\n474\n475\n476\nEach image 𝐼𝑏 is classified as either an individual or a crossing of individuals. For more\ndetails on the background subtraction and individual-crossing classification process, please\nrefer to Appendix D1-2 of the Supplementary Information of Romero-Ferrero et al. (2019).\n477\n478\n479\nA Fragment 𝐹 is defined as a sequence of blobs that maintain a one-to-one spatial over-\nlap, meaning they share pixels in each pair of consecutive frames over time. If two blobs\nmerge into a single blob in the subsequent frame, or if a single blob splits into two in the next\nframe, each of these three blobs will terminate or initiate a new Fragment. Fragments are\nclassified as either individual or crossing Fragments based on the classification of the blobs\nthey contain. Blobs of different classifications are not permitted within the same Fragment.\nSince crossings are solved as a post-processing step after identification, from now on we will\nnot take into consideration crossing Fragments, and we will refer to individual Fragments\nas Fragments.\n480\n481\n482\n483\n484\n485\n486\n487\n488\nA pair of Fragments is said to coexist if they both contain blobs from the same frames\nin the video. Moreover, being 𝑁 the number of individuals in a video, a Global Fragment is\ndefined as a collection of 𝑁 Fragments all sharing a common frame.\n489\n490\n491\nBy construction, we can assume that all blobs in a Fragment correspond to the same\nidentity, this is the Fragment’s identity. From this, coexisting Fragments will have different\nidentities and Global Fragments will have all identities, one per Fragment.\n492\n493\n494\nFrom now on, we will denote 𝐹𝑖 as the fragment with some arbitrary unique identifier 𝑖\nand 𝐼𝑖𝑘 will correspond to the identification image with the unique arbitrary identifier 𝑘 in\nthe fragment 𝑖.\n495\n496\n497\nGeneral overview of Identification Protocols in the original idtracker.ai498\nIn this section we will give a brief and high level overview on the algorithm idtracker.ai uses\nto assign identities to the different fragments. Please refer to Romero-Ferrero et al. (2019)\nfor a more complete description of the algorithm.\n499\n500\n501\nCascade of Training and Identification Protocols502\nThe identification process begins with three sequential protocols that incrementally refine\nthe identification network’s ability to label individuals. The protocols leverage segments of\nthe video where individuals appear distinctly, called global fragments, to construct a labeled\ndataset for the training of the network.\n503\n504\n505\n506\nProtocol 1: Basic Accumulation of Global Fragments In Protocol 1, the algorithm\nsearches for global fragments. The initial set of labeled images from these fragments forms\nthe base dataset to train the identification network. This trained network is then used to la-\nbel additional global fragments throughout the video. If Protocol is not able to accumulate\nat least 99.95% of all images in the global fragments, the algorithm proceeds to Protocol 2.\n507\n508\n509\n510\n511\n18 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\nProtocol 2: Iterative Expansion with High-Quality Fragments Protocol 2 builds on\nthe initial training by iteratively alternating between accumulating new global fragments\nand using them to further train the identification network. With each iteration, the network\nlabels more fragments, adding only those that pass strict quality checks (explained in the\nsection below). This process continues until either 99.95% of the images in the global frag-\nments are labeled with high certainty, or no more high-quality fragments are available.\n512\n513\n514\n515\n516\n517\nProtocol 3: Pretraining and Fine-Tuning for Complex Scenarios Protocol 2 might fail\nfor videos with high visual complexity (accumulating less than 90% of the images). In those\ncases, idtracker.ai proceeds to Protocol 3. Protocol 3 pretrains the convolutional layers\nof the identification network on a large sample of global fragments, using the same con-\nvolutional layers for each global fragment while changing only the last classification layer.\nAlthough this protocol is effective in tracking videos that cannot be tracked with Protocol 2,\nit is very slow and may take days for some videos.\n518\n519\n520\n521\n522\n523\n524\nLabeling and Accumulating Images in Global Fragments525\nThe process of labeling and accumulating images from global fragments involves the follow-\ning steps:\n526\n527\n1. Selection of Global Fragments: The algorithm identifies global fragments where all\nanimals are visually distinct, ensuring unambiguous initial identity assignments.\n528\n529\n2. Labeling with the Trained Network: The identification network, trained on an initial\nset of global fragments, predicts identities across additional fragments belong to the\nother global fragments. Each fragment is assigned an identity based on the network’s\nclassification probabilities of its corresponding images, denoted 𝑃 1(𝐹 , 𝑖).\n530\n531\n532\n533\n3. Quality Checks: Labeled fragments are subjected to a series of quality checks to en-\nsure the reliability of their identity assignments. For each global fragment these checks\ninclude:\n534\n535\n536\n• Certainty: Each fragment 𝐹 must have a high certainty score, defined by the\ndistinction between the highest and second-highest identity probabilities:\ncert(𝐹 ) = median(𝑆𝑎) ⋅ 𝑃 1(𝐹 , 𝑎) − median(𝑆𝑏) ⋅ 𝑃 1(𝐹 , 𝑏)\n𝑃 1(𝐹 , 𝑎) + 𝑃 1(𝐹 , 𝑏)\nwhere 𝑃 1(𝐹 , 𝑖) represents the probability of fragment 𝐹 being assigned identity\n𝑖. Here, 𝑎 and 𝑏 represent the identity predictions with the highest and second\nhighest 𝑃 1 values for fragment for 𝐹 , with 𝑆𝑎 and 𝑆𝑏 being the vectors of soft-\nmax values of all the images in the fragment 𝐹 assigned to the identities 𝑎 and 𝑏\nrespectively.\n537\n538\n539\n540\n541\n542\n543\n544\n545\n546\n• Consistency: The identity assignment for each fragment must remain consistent\nacross frames, preventing arbitrary changes in identity due to minor variations\nin appearance. This is reflected on the value of 𝑃 1.\n547\n548\n549\n• Uniqueness: Within a single global fragment, each assigned identity must be\nunique, ensuring that no two animals share the same identity label within that\nfragment.\n550\n551\n552\n4. Accumulation into the Training Set: Fragments that pass the quality checks are\nadded to the training dataset, allowing the network to improve its accuracy iteratively.\nThis accumulation process continues, increasing the network’s generalization ability\nacross the video.\n553\n554\n555\n556\n19 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\nResidual Identification557\nAfter the cascade protocols, residual identification is applied to label any fragments that re-\nmain unlabeled or have low-certainty assignments. This step uses a probabilistic approach\nthat accounts for temporal coexistence constraints, refining identity assignments. For each\nunlabeled fragment 𝐹 , an adjusted probability 𝑃 2(𝐹 , 𝑖) is computed for assigning identity 𝑖,\nconsidering neighboring fragments 𝛾(𝐹 ) that overlap in time:\n𝑃 2(𝐹 , 𝑖) =\n𝑃 1(𝐹 , 𝑖)∏\n𝐺∈𝛾(𝐹 )(1 − 𝑃 1(𝐺, 𝑖))\n∑\n𝑗 𝑃 1(𝐹 , 𝑗) ∏\n𝐺∈𝛾(𝐹 )(1 − 𝑃 1(𝐺, 𝑗))\nwhere 𝑃 1(𝐹 , 𝑖) represents the initial probability of 𝐹 being identity 𝑖.\n558\n559\n560\n561\n562\n563\n564\n565\n566\nAfterwards a new measure of identification certainty is defined as\ncert(𝐹 ) = 𝑃 2(𝐹 , 𝑎)\n𝑃 2(𝐹 , 𝑏\nin which 𝑎 and 𝑏 again represent the identity predictions with the highest and second highest\n𝑃 1 values for fragment for 𝐹 . Fragments then are assigned identities in descending order\nof certainty, with the highest-confidence fragments labeled first.\n567\n568\n569\n570\n571\n572\n573\nIn this work, the primary advancement was the replacement of protocols in idtracker.ai\nwith an identification method based on deep metric learning. Additionally, several smaller\nbut significant technical improvements were implemented, enhancing feature set, tracking\ntime, and memory usage efficiency.\n574\n575\n576\n577\n20 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\nAppendix 2578\nContrastive protocol579\nContrastive learning is a type of self-supervised learning that aims to learn useful data rep-\nresentations by contrasting positive and negative pairs of examples. The fundamental idea\nis to bring similar (positive) pairs closer in the representation space while pushing dissimilar\n(negative) pairs farther apart. This approach leverages the inherent structure of the data,\nallowing the model to learn without labeled examples.\n580\n581\n582\n583\n584\nThe representation space or embedding in contrastive learning is a high-dimensional\nenvironment where data points are mapped to vectors, capturing essential features and\npatterns of the original data. This space can be conceptualized as a vast, multidimensional\nenvironment in which each data point is represented as a vector. The primary objective is to\nposition similar data points in close proximity while ensuring that dissimilar data points are\nsituated at a considerable distance from one another. Positive pairs are typically created by\napplying different transformations or augmentations to the same data point, such as crop-\nping, rotating, or color jittering an image, preserving the inherent semantics of the original\ndata point. These augmentations ensure that the model learns robust features invariant\nto such transformations. Conversely, negative pairs are composed of different data points\nexpected to be dissimilar, such as two distinct images.\n585\n586\n587\n588\n589\n590\n591\n592\n593\n594\n595\nAs the model undergoes training, the representation space becomes increasingly struc-\ntured, with similar types of data points forming coherent clusters. These clusters encapsu-\nlate the inherent similarities within the data, even if the specific instances differ, such as\ndifferent breeds of cats or different poses. By maximizing the agreement between positive\npairs and minimizing the agreement between negative pairs, the model learns to distin-\nguish subtle differences and similarities within the data. The contrastive loss minimizes the\ndistance between positive pairs and maximizes the distance between negative pairs in the\nrepresentation space. This contrastive objective ensures the learned representations cap-\nture essential features and discriminative patterns, facilitating downstream tasks such as\nclassification, clustering, and retrieval, even without labeled data. Thus, the representation\nspace serves as a learned map where the positions of data points reflect their semantic re-\nlationships, enabling the model to capture and utilize the underlying structure of the data\nfor various tasks.\n596\n597\n598\n599\n600\n601\n602\n603\n604\n605\n606\n607\n608\nWe apply the principles of contrastive learning to create an embedding of all the im-\nages in a video that reflects the fragmented structure of the video. Specifically, points in the\nembedding corresponding to images from coexisting fragments (different identities) are po-\nsitioned further apart than points corresponding to images from the same fragment (same\nidentity) (Figure 1a–c).\n609\n610\n611\n612\n613\n1. Segmentation and Fragmentation: The video is segmented and the blobs grouped\ninto fragments based on temporal or content-based criteria.\n614\n615\n2. Training ResNet18: ResNet18 is trained using positive pairs (images from the same\nfragment) and negative pairs (images from coexisting fragments). The network learns\na representation space where the distance between positive pairs is minimized, while\nthe distance between negative pairs is maximized.\n616\n617\n618\n619\n3. Clustering in the Representational Space: All images are passed through the net-\nwork. K-means clustering is then applied to the embedded images, assigning them to\ndifferent cluster labels.\n620\n621\n622\n4. Cluster based labeling of Single Image: Each cluster is labeled as a distinct animal\nidentity. Images are classified based on their assigned clusters, and a probability dis-\n21 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\ntribution for each identity prediction is computed based on the Euclidean distance\nto the center of each cluster. If global fragments are present, proceed to next step;\notherwise, proceed to Step 7.\n623\n624\n625\n626\n627\n5. Fragment Identification with Global Fragments : A thorough identification process\nis conducted to classify all images belonging to global fragments, correcting any errors\nfrom the initial classification. If 99.9% > of all the images in global fragments are suc-\ncessfully accumulated (pass the quality checks, see section 1), go to Step 7; otherwise,\ngo to next step.\n628\n629\n630\n631\n632\n6. Run Accumulation Protocol if Step 5 Fails: Run protocol 2 from idtracker.ai v5 but\nusing correctly identified images as the ground truth, as a sort of synthetic first Global\nFragment.\n633\n634\n635\n7. Residual Identification: A thorough identification process is conducted to classify all\nimages in the video, correcting any errors from the initial classification step.\n636\n637\nNetwork architecture638\nDeep metric learning often requires larger networks for classification tasks compared to\nstandard supervised learning. To identify the most suitable architecture, we evaluated sev-\neral state-of-the-art image classification networks, including the model used in the original\nidtracker.ai.\n639\n640\n641\n642\nThere were specific constraints in selecting the optimal architecture. The image size\nis automatically set during each tracking session to fit the average blob size, but it is typi-\ncally small, ranging from 20×20 to 100×100 pixels. This limited some architectures, such as\nAlexNet, which requires a fixed input size of 227×227, and DenseNet, which has a minimum\ninput size of 29×29. Additionally, the large training batches commonly associated with deep\nmetric learning necessitate a compact model that can be trained on a consumer-grade GPU.\nThis constraint excluded other architectures, including EfficientNet and the larger ResNet\nmodels (ResNet101 and ResNet152).\n643\n644\n645\n646\n647\n648\n649\n650\nAs shown in Figure 1—figure Supplement 1, ResNet18 offered the best balance between\ntraining speed and tracking accuracy.\n651\n652\nEmbedding dimension653\nAnother critical hyperparameter is the embedding dimension. Here, too, there is a trade-\noff between achieving a robust representation of subtle differences between animals—\ndifferences that may be minimal and even challenging to detect visually—and maintaining\na compact network size and efficient training speed. This parameter was empirically deter-\nmined to be 8 ( Figure 1—figure Supplement 2).\n654\n655\n656\n657\n658\nLoss function659\nThe contrastive loss function operates on pairs of data points, aiming to minimize the dis-\ntance between positive pairs and maximize the distance for negative pairs. Mathematically\nfor our case, the contrastive loss 𝐿 for a pair of images (𝐼𝑖𝑘, 𝐼𝑗𝑙 ) is defined as:\n660\n661\n662\n(𝐼𝑖𝑘, 𝐼𝑗𝑙 , 𝑙𝑖𝑘, 𝑗𝑙\n)=𝑙𝑖𝑘, 𝑗𝑙 ⋅ max(0, 𝐷𝑖𝑘, 𝑗𝑙 − 𝐷pos)2\n+ (1 − 𝑙𝑖𝑘, 𝑗𝑙) ⋅ max(0, 𝐷neg − 𝐷𝑖𝑘, 𝑗𝑙)2\n𝑙𝑖𝑘, 𝑗𝑙 =\n⎧\n⎪\n⎨\n⎪⎩\n1 if 𝑖 = 𝑗 (positive pair)\n0 Otherwise (negative pair)\n(1)\n22 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\nwhere 𝐷𝑖𝑘, 𝑗𝑙 is the Euclidean distance between the embedding of 𝐼𝑖𝑘 and 𝐼𝑗𝑙 , 𝐷neg is the min-\nimum allowed distance in a negative pair of images (images coming from coexisting frag-\nments), and 𝐷pos is the maximum allowed distance in a positive pair of images (images from\nthe same fragment). It is important to emphasize that the network processes one image at\na time, obtaining a single independent point in the representational space for each image.\nThe Euclidean distance between the embeddings for the corresponding pairs of images is\ncomputed only afterwards.\n663\n664\n665\n666\n667\n668\n669\n670\n671\n672\n673\n𝐷neg and 𝐷pos serve as thresholds to regulate distances in the embedding space. 𝐷neg\nprevents images from negative pairs from being pushed indefinitely far apart, while 𝐷pos\nprevents the collapse of images from positive pairs into a single point. These thresholds are\ncrucial in our problem, where we aim to embed individuals of the same identity in similar\nregions of the representational space. However, we face the restriction of not being able to\ncompare all possible pairs of images and are instead limited to the fragment structure of\nthe video to obtain the labels 𝑙𝑖𝑘, 𝑗𝑙.\n674\n675\n676\n677\n678\n679\n680\nThis limitation means that the loss function does not directly pull together embeddings\nof the same identity, but rather images from the same fragment. Similarly, the loss does\nnot push apart embeddings of different identities but images from coexisting fragments.\n𝐷pos helps prevent the collapse of all images from the same fragment to a single point,\nallowing for the creation of a diffuse region in the representational space where fragments\nfrom the same identity are clustered together. 𝐷neg prevents excessive scattering, ensuring\nbetter compression of the representational space and maintaining the integrity of clusters\nof images from the same identity.\n681\n682\n683\n684\n685\n686\n687\n688\nIn the contrastive protocol, we used 𝐷pos = 1 and 𝐷neg = 10. These values were deter-\nmined empirically and provide effective embeddings and were robust for tracking multiple\nvideos across various species and different numbers of animals ( Figure 1—figure Supple-\nment 3).\n689\n690\n691\n692\nClustering and assignment693\nAfter training the network using contrastive loss, we pass all images through the network\nto generate their corresponding embeddings in the learned representational space. These\nembeddings are then grouped using K-means clustering. Each cluster ideally represents im-\nages of the same identity, as the training process has encouraged the network to place sim-\nilar images close together and dissimilar ones farther apart in the embedding space. Next,\nwe perform single-image classification, assigning each image a label based on the cluster to\nwhich its embedding belongs. Afterwards, the assignment method follows two conditions.\nIf global fragments are present, follow the procedure mentioned in the subsection 1. If on\nthe contrary there are no global fragments we move straight to residual identification as\nexplained in section 1\n694\n695\n696\n697\n698\n699\n700\n701\n702\n703\nIn order to identify fragments we, not only need an identity prediction for each image\nbut also a probability distribution over all the identities. Let 𝑑𝑗(𝐼𝑖𝑘) be the distance of image\n𝐼𝑖𝑘 to the center of cluster 𝑗. We define the probability of image 𝐼𝑖𝑘 belonging to identity 𝑗\nby\n𝑃 (𝐼𝑖𝑘 belongs to identity 𝑗)=\n𝑑𝑗(𝐼𝑖𝑘)7\n∑\n𝑗 𝑑𝑗(𝐼𝑖𝑘)7 (2)\n704\n705\n706\n707\n708\n709\n710\nEquation (2) is used to emphasize differences in distances between points and clusters,\ncreating a more peaked probability distribution that clearly distinguishes closer clusters\nfrom farther ones. The exponent of 7 smooths the probability distribution and reduces\nthe influence of distant clusters, making the assignment more discriminative. In higher-\ndimensional spaces like the 8-dimensional space in the paper, distances are more spread\n23 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\nout, and using a high power helps to counteract this dispersion, resulting in more confident\ncluster assignments.\n711\n712\n713\n714\n715\n716\n717\nIf we are in a scenario where global fragments exist, we use them for K-means initializa-\ntion: we use the embeddings from the first global fragment as initial cluster centers, choos-\ning the one where the minimum fragment is the largest. This approach provides a strong\ninitialization for the K-means algorithm, aligning it with the different identities and mitigat-\ning issues related to random initialization. It also allows us to better compare clusters as\ntraining progresses.\n718\n719\n720\n721\n722\n723\nStopping criteria724\nStopping network training using the loss function directly can be highly variable, as differ-\nent video conditions, the number of individuals and the sampling method significantly influ-\nence this value. To circumvent this we use the silhouette score (SS) Rousseeuw (1987) of the\nclusters of the embedded images. Let 𝑑(𝐼, 𝐽 ) be the Euclidean distance between the embed-\ndings of image 𝐼 and 𝐽 , for each image 𝐼, in cluster 𝐶𝑎 we compute the mean intra-cluster\ndistance\n𝑎(𝐼) = 1\n|𝐶𝑎| − 1\n∑\n𝐽 ∈𝐶𝑎,𝐽 ≠𝐼\n𝑑(𝐼, 𝐽 ),\nand the mean nearest-cluster distance\n𝑏(𝐼) = min\n𝑎≠𝑏\n1\n|𝐶𝑏|\n∑\n𝐽 ∈𝐶𝑏\n𝑑(𝐼, 𝐽 ).\nThe SS is given by\n𝑆𝑆 = 1\nnumber of images\n∑\n𝐼\n𝑏(𝐼) − 𝑎(𝐼)\nmax{𝑏(𝐼), 𝑎(𝐼)}\n725\n726\n727\n728\n729\n730\n731\n732\n733\n734\n735\n736\n737\n738\n739\n740\n741\nTo determine when to stop training, every 𝑚 batches we compute the SS by clustering\nthe embeddings of a random sample of the images in the video, generating also a check-\npoint of the model. 𝑚 was set to be the maximum between 100 and number of animals in\na video times 5. We stop training if: 1) there have been 30 consecutive SS evaluations with-\nout any improvement (patience of 30), or 2) there have been 2 consecutive SS evaluations\nwithout any improvement but the SS already achieved a value of 0.91. After stopping the\ntraining, the model with the highest SS is chosen. A threshold of 0.91 was validated empir-\nically (Figure 1d and Figure 1e). The number of images used for the computation of the SS\nis 1000 times the number of animals.\n742\n743\n744\n745\n746\n747\n748\n749\n750\nPairs selection751\nIdeally, we would create two datasets of image pairs: one containing negative pairs and an-\nother containing positive pairs. However, the challenge with this approach is that very long\nvideos or those containing a large number of animals can yield trillions of pairs of images,\nmaking the process computationally prohibitive. Therefore, we approach the problem with\na hierarchical sampling method: first, we randomly select a pair of coexisting fragments,\nand then we sample an image from each fragment. For a positive pair, we sample two\nimages from the same fragment.\n752\n753\n754\n755\n756\n757\n758\nFollowing this idea, we start by creating two datasets. The first consists of a list of all the\nfragments in the video, from which we will sample the positive pairs. The second dataset\ncontains all possible pairs of coexisting fragments in the video. From these lists we exclude\nall fragments smaller than 4 images to reduce possible noisy blobs.\n759\n760\n761\n762\nEmpirical testing has revealed that large and balanced batches, with an equal number of\npositive and negative pairs, are ideal for our setting of contrastive learning. More concretely,\n24 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\nwe choose batches consisting of 400 positive pairs of images and 400 negative pairs of im-\nages (1600 images in total), as it was the smaller batch size that didn’t compromise training\nspeed/accuracy ( Figure 1 —figure Supplement 4). Intuitively, large batch sizes allow for a\ngood spread of pairs from a significant proportion of the video, thereby forcing the net-\nwork to learn a global embedding of the video. Since positive pairs tend to diminish the size\nof the representational space while negative pairs tend to increase it, a good balance be-\ntween the two forces the network to compress the representational space while respecting\nthe negative relationships Chen et al. (2020a). This balance between positive and negative\npairs is somewhat surprising, given that several works emphasize the importance of nega-\ntive examples over positive ones Awasthi et al. (2022); Khosla et al. (2021). While we do not\nyet have an explanation for why this balance appears to perform better in our case, we note\nthat it is not possible to compare all images from one class against those of another, as neg-\native pairs of images can only be sampled from coexisting fragments. Additionally, positive\npairs that compress the space can only be sampled from the same fragment and not the\nsame identity. Since we cannot compare images freely and are constrained by the fragment\nstructure of the video, we might need more positive pairs to ensure a higher degree of com-\npression of the representational space, such that not only images from the same fragment\nare close together, but also images from the same identity.\n763\n764\n765\n766\n767\n768\n769\n770\n771\n772\n773\n774\n775\n776\n777\n778\n779\n780\n781\n782\nThe hierarchical sampling allows us to address the question of how to select pairs of frag-\nments to optimize the training speed of the network. Since we sample pairs of fragments\nrather than directly sampling pairs of images, we need to skew the probability of a pair of\nfragments being sampled to reflect the number of images they contain. More concretely,\nlet 𝑓𝑖 be the number of images in fragment 𝐹𝑖. For negative relations we define 𝑓𝑖,𝑗 = 𝑓𝑖 + 𝑓𝑗\nand set the probability of sampling the pair 𝐹𝑖, 𝐹𝑗, by their size as:\n𝑃𝑠(𝐹𝑖, 𝐹𝑗) =\n𝑓𝑖,𝑗\n∑𝑁−1\n𝑘=1\n∑𝑁\n𝑙=𝑘+1 𝑓𝑘,𝑙\n.\nFor positive pairs, the probability of sampling a given fragment 𝑓𝑖 is:\n𝑃𝑠(𝐹𝑖) = 𝑓𝑖\n∑𝑁\n𝑗=1 𝑓𝑗\n.\n783\n784\n785\n786\n787\n788\n789\n790\n791\n792\n793\n794\n795\nBy examining the evolution of the clusters during training ( Figure 1c ) it becomes clear\nthat the learning process is not uniform; some identities become separated sooner than\nothers. Figure 1c top row second and third columns give us a nice illustration of this phe-\nnomenon. The images embedded in the red rectangle of the representational space already\nsatisfy the loss function, meaning that the negative pairwise relationships are already em-\nbedded further away than 𝐷neg, and images that form positive pairwise relationships are\nalready embedded closer than 𝐷pos. Consequently, the loss function for these pairs is ef-\nfectively zero, and passing them through the network will not alter the weights, merely pro-\nlonging the training process. In contrast, the separation of clusters in the green rectangle\nis incomplete, indicating that image pairs in this region still contribute to the loss function.\nThese pairs are more pertinent, as they contain information that the network has yet to\nlearn. To bias the sampling of image pairs towards those that still contribute to the loss\nfunction, each pair of fragments is assigned a loss score. When a pair of images is sampled\nfor training, if the loss for that pair is not zero, the loss score for the corresponding pair of\nfragments is incremented by one. This score then undergoes an exponential decay of 2%\nper batch. More specifically, let 𝑙𝑠(𝑖, 𝑗) be the loss score of the pair of fragments 𝐹𝑖 and 𝐹𝑗,\nand (𝐼𝑖𝑙, 𝐼𝑖𝑘\n)the loss of the images 𝐼𝑖𝑙 and 𝐼𝑖𝑘. If the pair 𝐼𝑖𝑙 and 𝐼𝑖𝑘 is sampled the loss score\n25 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\nis updated by\n𝑙𝑠(𝑖, 𝑗) ⟵\n⎧\n⎪\n⎨\n⎪⎩\n(𝑙𝑠(𝑖, 𝑗) + 1)(1 − 0.02), if (𝐼𝑖𝑙, 𝐼𝑖𝑘\n)> 0\n𝑙𝑠(𝑖, 𝑗)(1 − 0.02), otherwise\n(3)\nThe exponential decay is always applied independently to every pair of fragments, regard-\nless of whether the pairs of images were sampled from those fragments in the previous\nbatch of images or not. The loss score is converted into a probably distribution over all\npairs of fragments by\n𝑃𝑙𝑠 (𝐹𝑖, 𝐹𝑗) =\n⎧\n⎪\n⎨\n⎪⎩\n𝑙𝑠(𝑖,𝑗)\n∑\n𝑖≠𝑗 𝑙𝑠(𝑖,𝑗) , if 𝑖 ≠ 𝑗\n𝑙𝑠(𝑖,𝑖)\n∑\n𝑖 𝑙𝑠(𝑖,𝑖) , otherwise\n(4)\n796\n797\n798\n799\n800\n801\n802\n803\n804\n805\n806\n807\n808\n809\n810\n811\n812\n813\n814\n815\n816\n817\n818\n819\n820\n821\n822\n823\nThe final probability of sampling pairs of fragments is given by\n𝑃 (𝐹𝑖, 𝐹𝑗) = 𝛼𝑃𝑠(𝐹𝑖, 𝐹𝑗) + (1 − 𝛼)𝑃𝑙𝑠 (𝐹𝑖, 𝐹𝑗) (5)\nThis balance between these two probabilities can be seen as an exploitation versus explo-\nration paradigm. 𝑃𝑠(𝐹𝑖, 𝐹𝑗) enforces constant exploration, while 𝑃𝑙𝑠 (𝐹𝑖, 𝐹𝑗) exploits the cur-\nrent state of learning by dynamically updating the sampling probability. This ensures that\npairs of fragments containing unlearned knowledge are sampled more frequently, while\nmaintaining a baseline of exploration based on fragment size. We tried several values for 𝛼\nand saw that a value of 𝛼 around 1\n2 produced the best decrease the time required to train the\nnetwork across a large collection of videos ( Figure 1—figure Supplement 5). It is notewor-\nthy that the failure of the 𝛼 = 0 case renders the contrastive protocol ineffective in solving\nthe tracking problem. This failure occurs because the sampling becomes highly biased to-\nwards specific regions of the representational space, leading to only local solutions for the\nseparation of negative pairs and the compression of positive pairs. In effect, the network\nexperiences catastrophic forgetting by focusing excessively on small groups of fragments\nat a time, thereby compromising the embeddings of other images.\n824\n825\n826\n827\n828\n829\n830\n831\n832\n833\n834\n835\n836\n837\n838\n839\n840\n26 of 26\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\n1%\n10%\n100%Error\nzebrafish_20\nModel MACs Params\nSqueezeNet 22G 0.7M\nidtracker.ai CNN 45G 1.2M\nMobileNet 25G 2.2M\nMnasNet 29G 3.1M\nShuffleNet 51G 5.4M\nDenseNet 130G 7.0M\nResNet18 131G 11.2M\nResNet34 265G 21.3M\nResNet50 296G 23.5M\ndrosophila_100_1 zebrafish_100_1\n0 10 20\nTraining time (minutes)\n1%\n10%\n100%Error\ndrosophila_59\n0 10 20\nTraining time (minutes)\nzebrafish_60_1\n0 10 20\nTraining time (minutes)\nzebrafish_100_2\nFigure 1—figure supplement 1. Models comparison. Error in image identification as a function\nof training time for different deep learning models in 6 test videos. For each network we report the\nmultiply-accumulate operations (MAC) in giga operations (G) and the number of parameters in the\nunits of million parameters (M). Every 100 training batches, we perform k-means clustering on a\nrandomly selected set of 20,000 images, assigning identities based on clusters. We then compute\nthe Silhouette Score and ground-truth error on the same set. The reported error corresponds to\nthe model with the best Silhouette Score observed up to that point.\n841\n1%\n10%\n100%Error\nzebrafish_20\nDimension\n2\n4\n8\n16\n32\n64\ndrosophila_100_1 zebrafish_100_1\n0 5 10 15 20\nTraining time (minutes)\n1%\n10%\n100%Error\ndrosophila_59\n0 5 10 15 20\nTraining time (minutes)\nzebrafish_60_1\n0 5 10 15 20\nTraining time (minutes)\nzebrafish_100_2\nFigure 1—figure supplement 2. Embedding dimensions comparison. Error in image identifi-\ncation as a function of training time for different embedding dimensions in 6 test videos. Every\n100 training batches, we perform k-means clustering on a randomly selected set of 20,000 images,\nassigning identities based on clusters. We then compute the Silhouette Score and ground-truth er-\nror on the same set. The reported error corresponds to the model with the best Silhouette Score\nobserved up to that point.\n842\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\n1%\n10%\n100%Error\nzebrafish_20\nDneg/Dpos\n2\n4\n7\n10\n13\n16\n19\n24\n30\ndrosophila_100_1 zebrafish_100_1\n0 5 10 15 20\nTraining time (minutes)\n1%\n10%\n100%Error\ndrosophila_59\n0 5 10 15 20\nTraining time (minutes)\nzebrafish_60_1\n0 5 10 15 20\nTraining time (minutes)\nzebrafish_100_2\nFigure 1—figure supplement 3. 𝐷neg over 𝐷pos comparison. Error in image identification as a\nfunction of training time for different ratios of 𝐷neg∕𝐷pos in 6 test videos. Every 100 training batches,\nwe perform k-means clustering on a randomly selected set of 20,000 images, assigning identities\nbased on clusters. We then compute the Silhouette Score and ground-truth error on the same set.\nThe reported error corresponds to the model with the best Silhouette Score observed up to that\npoint.\n843\n1%\n10%\n100%Error\nzebrafish_20\nBatch size\n100\n200\n400\n600\n800\n1000\ndrosophila_100_1 zebrafish_100_1\n0 5 10 15 20\nTraining time (minutes)\n1%\n10%\n100%Error\ndrosophila_59\n0 5 10 15 20\nTraining time (minutes)\nzebrafish_60_1\n0 5 10 15 20\nTraining time (minutes)\nzebrafish_100_2\nFigure 1—figure supplement 4. Batch size comparison. Error in image identification as a func-\ntion of training time for different batch sizes of pairs of images in 6 test videos. Every 100 training\nbatches, we perform k-means clustering on a randomly selected set of 20,000 images, assigning\nidentities based on clusters. We then compute the Silhouette Score and ground-truth error on the\nsame set. The reported error corresponds to the model with the best Silhouette Score observed\nup to that point.\n844\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\n1%\n10%\n100%Error\nzebrafish_20\nExploitation\n0\n0.1\n0.2\n0.3\n0.4\n0.5\n0.6\n0.7\n0.8\n0.9\n1\nExploration\ndrosophila_100_1 zebrafish_100_1\n0 5 10 15 20\nTraining time (minutes)\n1%\n10%\n100%Error\ndrosophila_59\n0 5 10 15 20\nTraining time (minutes)\nzebrafish_60_1\n0 5 10 15 20\nTraining time (minutes)\nzebrafish_100_2\nFigure 1—figure supplement 5. Exploration and exploitation comparison. Error in image iden-\ntification as a function of training time for different exploration/exploitation weights 𝛼 in 6 test\nvideos. Every 100 training batches, we perform k-means clustering on a randomly selected set of\n20,000 images, assigning identities based on clusters. We then compute the Silhouette Score and\nground-truth error on the same set. The reported error corresponds to the model with the best\nSilhouette Score observed up to that point.\n845\nz_10_1z_80_3z_80_2\nz_20z_7\nz_10_3z_60_3d_72z_10_2z_100_2z_100_3z_10_4m_2_4\nz_5d_60d_38z_60_2z_80_1d_59d_80d_6\nm_2_2d_10d_8\nm_4_1m_2_1z_60_1d_100_1d_100_2m_2_3z_100_1m_4_2d_100_3\n92%\n93%\n94%\n95%\n96%\n97%\n98%\n99%\n100%accuracy with crossings\nTRex\noriginal idtracker.ai (v4)\noptimized v4 (v5)\nnew idtracker.ai (v6)\nm_2_4m_2_3m_2_2m_2_1m_4_2\nz_7\nm_4_1\nz_5\nz_10_4\nd_6\nz_10_1\nd_8\nz_10_3z_10_2\nz_20\nz_60_3z_60_1z_80_3z_60_2z_80_2z_100_2z_100_3\nd_10d_38z_80_1d_72\nz_100_1\nd_59d_60d_80\nd_100_1d_100_2d_100_3\n0.0\n0.5\n1.0\n1.5\n2.0\n2.5tracking time (hours)\na\nb\nFigure 2—figure supplement 1. Performance for the benchmark with full trajectories with\nanimal crossings . a. Median accuracy was computed using all images of animals in the videos\nincluding animal crossings. b. Median tracking times. Supplementary Table 1, Supplementary\nTable 2, Supplementary Table 3 and Supplementary Table 4 give more complete statistics (me-\ndian, mean and 20-80 percentiles) for the original idtracker.ai (version 4 of the software), optimized\nv4 (version 5), new idtracker.ai (version 6) and TRex, respectively.\n846\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\nd_100_2\nd_60d_10\nd_100_1\nd_80\nd_100_3z_80_1d_59d_72m_4_2d_38\nz_100_1\nd_8\nm_2_1z_10_2\nz_20\nz_10_1z_10_3z_100_2z_100_3\nd_6\nm_2_2m_2_3m_2_4m_4_1z_10_4\nz_5\nz_60_1z_60_2z_60_3\nz_7\nz_80_2z_80_3\n0%\n20%\n40%\n60%\n80%\n100%\noriginal idtracker.ai (v4)\noptimized v4 (v5)\nTRex\nFigure 2—figure supplement 2. Protocol 2 failure rate. Probability for the different tracking sys-\ntems of not tracking the video with Protocol 2 in idtracker.ai (v4 and v5) and in TRex the probability\nthat it fails without generating trajectories.\n847\n0.0 0.5 1.0 1.5 2.0 2.5 3.0\nnumber of blobs in the video (millions)\n0\n10\n20\n30\n40\n50memory peak (GB)\nold idtracker.ai (v4) P2\nold idtracker.ai (v4) P3\nTRex\noptimized v4 (v5) P2\noptimized v4 (v5) P3\nnew idtracker.ai (v6)\nFigure 2—figure supplement 3. Memory usage across the different softwares. The solid line\nis a logarithmic fit to the memory peak as a function of the number of blobs in a video. Disclaimer:\nBoth software programs include automatic optimizations that adjust based on machine resources,\nso results may vary on systems with less available memory. These results were measured on com-\nputers with the specifications in Methods\n848\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint \n\nAccuracy = 99.975%Accuracy = 99.963±0.01%\nAccuracy = 99.963±0.01%\nAccuracy = 99.957%\nOriginal light conditions\n(all lights on)\nManipulated light conditions\n(bottom and right lights off)\nBlurred with std=1px, rescaled to 40% the original\nresolution, and compressed with MJPG codec\nOriginal zebrafish_60_1\nFigure 2—figure supplement 4. Robustness to blurring and light conditions. First column:\nUnmodified video zebrafish_60_1. Second column: zebrafish_60_1 with a gaussian blurring of\nsigma=1 pixel plus a resolution reduction to 40% of the original plus MJPG video compression.\nThird column: Videos of 60 zebrafish with manipulated light conditions (same test as in id-\ntracker.ai Romero-Ferrero et al. (2019)). First row: Uniform light conditions across the arena (ze-\nbrafish_60_1). Second row: Similar setup but with lights off in the bottom and right side of the\narena.\n849\n.CC-BY 4.0 International licenseavailable under a \nwas not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprint (whichthis version posted June 2, 2025. ; https://doi.org/10.1101/2025.05.30.657023doi: bioRxiv preprint","source_license":"CC-BY-4.0","license_restricted":false}