A computer vision toolkit for the dynamic study of air sacs in Siamang with a general application to the study of elastic kinematics in other animals

Biological structures are defined by elements like bones and cartilage, and elastic elements like muscles and membranes. Computer vision advances have enabled automatic tracking of moving animal skeletal poses. Such developments put us on the verge of gaining insights in complex dynamics otherwise studied in more static terms (e.g., images). However, the elastic soft-tissues of organisms, like the nose of Elephant seals, or the buccal sac of frogs, have been poorly studiedand no computer vision methods have been proposed. This leaves major gaps in different areas in biology. In the area of primatology, most critically, the function of air sacs is widely debated and many questions exist about their role in communication and human language evolution. Moving towards the dynamic study of soft-tissue elastic structures, we present a toolkit for the automated tracking of semi-circular elastic structures in biological video data. The toolkit containsunsupervised computer vision tools (using Hough transform) and supervised deep learning (by adapting Python’s Deeplabcut) methodology to track inflation of laryngeal air sacs or otherbiological spherical objects (e.g., gular cavities). Confirming the value of elastic kinematic analysis we show that air sac inflation correlates with acoustic markers that likely inform aboutbody size. Finally, we present a pre-processed audiovisual-kinematic dataset of 7+ hours of closeup audiovisual recordings of Siamang (Symphalangus syndactylus) singing. This toolkitrevitalizes the study of non-skeletal morphological structures in a wide range of animals.