Prof. Alexander Mathis

Estimating the pose of multiple animals is a challenging computer vision problem: frequent interactions cause occlusions and complicate the association of detected keypoints to the correct individuals, as well as having highly similar looking animals that interact more closely than in typical multi-human scenarios. To take up this challenge, we build on DeepLabCut, an open-source pose estimation toolbox, and provide high-performance animal assembly and tracking—features required for multi-animal scenarios. Firstly, we developed tailored ConvNets together with data-driven approaches for body plan agnostic pose estimation. Secondly, we integrate the ability to predict an animal’s identity to assist tracking (in case of occlusions). This can be achieved both in a supervised and unsupervised fashion using transformers.. In an effort to simplify this pipeline, I will also discuss an end-to-end trainable approach for multi-instance pose estimation, called POET (POse Estimation Transformer). Combining a ConvNet with a transformer encoder-decoder architecture, we formulate multi-instance pose estimation from images as a direct set prediction problem.