CIS – “Get to know your neighbors” Seminar Series
“Multi-individual pose estimation, identification and tracking“
Prof. Alexander Mathis, head of the Prof. Alexander Mathis Group
Monday, Jan 31, 2022 3:15 – 4:15pm (CET)
Estimating the pose of multiple animals is a challenging computer vision problem: frequent interactions cause occlusions and complicate the association of detected keypoints to the correct individuals, as well as having highly similar looking animals that interact more closely than in typical multi-human scenarios. To take up this challenge, we build on DeepLabCut, an open-source pose estimation toolbox, and provide high-performance animal assembly and tracking—features required for multi-animal scenarios. Firstly, we developed tailored ConvNets together with data-driven approaches for body plan agnostic pose estimation. Secondly, we integrate the ability to predict an animal’s identity to assist tracking (in case of occlusions). This can be achieved both in a supervised and unsupervised fashion using transformers.. In an effort to simplify this pipeline, I will also discuss an end-to-end trainable approach for multi-instance pose estimation, called POET (POse Estimation Transformer). Combining a ConvNet with a transformer encoder-decoder architecture, we formulate multi-instance pose estimation from images as a direct set prediction problem.
Alexander Mathis is an Assistant Professor École polytechnique fédérale de Lausanne.
He studied pure mathematics with a minor in logic & theory of science at the Ludwig Maximilians University in Munich. Subsequently, he pursued a PhD with Prof. Andreas Herz in Munich and worked on optimal coding approaches to elucidate how grid cells represent space. As a postdoctoral fellow with Prof. Venkatesh N. Murthy at Harvard University and Prof. Matthias Bethge at Tuebingen AI, he studied olfactory behaviors such as odor-guided navigation, social behaviors and the cocktail party problem in mice. During this time, he increasingly got interested sensorimotor behaviors beyond olfaction and started working on proprioception, motor adaption, as well as computer vision tools for measuring animal behavior. In 2020 he joined EPFL.
His group is interested in elucidating how the brain gives rise to behavior. For those purposes, he develops algorithms and systems to analyze animal behavior, neural data, as well as creates experimentally testable computational models.