Tracking Multiple People in a Multi-Camera Environment

Tracking People using Multiple Cameras

Reliably tracking multiple people using ordinary cameras is challenging, mostly due to the severe occlusions that occur when many people are involved. We tackle this problem by using several cameras, observing the scene from different viewpoints.

To exploit the resulting images, we developed a people detection algorithm called POM and a Deep-Learning based improvement, that use a generative model and Bayesian reasoning to estimate people’s positions  in each individual time frame.

Given these estimates we then rely on a global optimization method known as K-Shortest Path (KSP) to generate trajectories.

Combining these two algorithms let us track people reliably and in real-time in spite of very significant occlusions. This technology now forms the basis for a video-based commercial product that tracks team-sport players.

Results

Our Deep-Occlusion Reasoning method, which combines discriminative and generative models, allows us to perform 3D tracking, even in very crowded scenes.

This video shows our recent tracking results using global appearance constraints – Basketball sequence.

This video shows our recent tracking results using global appearance constraints – Soccer sequence.

This video shows our recent tracking results using global appearance constraints.

Technology Transfer

This technology was initially commercialized by PlayfulVision, an EPFL spinoff, which as acquired by SecondSpectrum in 2016. It is now being used to track NBA players during basketball games.

Source Code

  • The C++ source code of our POM people detector is available under a GPL license on the Software POM page of our web site.
  • The Theano-Python code of the Deep-Learning based version of POM is available on the Deep-Occlusion Reasoning Github .
  • The C++ source code for the K-shortest path multiple object tracker used to generate the results shown on this page is available upon request for academic purposes. More information is available on the Software KSP page of our web site.
  • The Multi-Tracked Paths (MTP) is another light weight implementation of the K shortest-paths algorithm for multi-target tracking, and is available under the GPL3 license.

Data Set

Some of the multi-camera video sequences that we acquired for this project are available for download on the Data part of our web site.

More annotations available from third-partie https://bitbucket.org/merayxu/multiview-object-tracking-dataset

References

Tracking Interacting Objects Using Intertwined Flows

X. Wang; E. Türetken; F. Fleuret; P. Fua 

IEEE Transactions on Pattern Analysis and Machine Intelligence. 2016. Vol. 38, num. 11, p. 2312–2326. DOI : 10.1109/Tpami.2015.2513406.

Detailed record

Full text – View at publisher

Multi-Commodity Network Flow for Tracking Multiple People

H. Ben Shitrit; J. Berclaz; F. Fleuret; P. Fua 

IEEE Transactions on Pattern Analysis and Machine Intelligence. 2014. Vol. 36, num. 8, p. 1614-1627. DOI : 10.1109/Tpami.2013.210.

Detailed record

Full text – View at publisher

Multiple Object Tracking using K-Shortest Paths Optimization

J. Berclaz; E. Turetken; F. Fleuret; P. Fua 

IEEE Transactions on Pattern Analysis and Machine Intelligence. 2011. Vol. 33, p. 1806–1819. DOI : 10.1109/TPAMI.2011.21.

Detailed record

Full text – View at publisher

Multi-Camera People Tracking with a Probabilistic Occupancy Map

F. Fleuret; J. Berclaz; R. Lengagne; P. Fua 

IEEE Transactions on Pattern Analysis and Machine Intelligence. 2008. Vol. 30, num. 2, p. 267-282. DOI : 10.1109/TPAMI.2007.1174.

Detailed record

Full text – View at publisher