Full title

Multi-view detection of unusual events for smart video surveillance

Project website

Duration of project

24 months extended until 31.1.2009

Funding source

COST Switzerland

Funding number

SER No. C06.0117


This project was taking place in the frame of COST Action 292 Semantic Multimodal Analysis of Digital Media, with the purpose of strengthening the scientific networking with all partners on the one hand, while specifically emphasizing the cooperation with Working Group 7 (JPSearch and TrecVid), Working Group 6 (Framework evaluation) and the already completed Working Groups 1 to 3 (Definition of test data and framework specifications and low-level analysis; Low-level analysis) through direct contact with concerned members.

The detection in video surveillance sequences of so-called “unusual events” (also labeled “unusual activities”) that are potentially representing a risk is a critical operation when performed by humans. Examples of such unusual events are slips and falls, vehicles driving on the wrong side of the road, potential theft, and left luggage or contra-flow motion of individuals in airports or railway stations. Indeed, humans are known to be unreliable in detecting unusual activities when screening / reviewing either on-line video sequences or archived data. This limitation is due to the quantity of information needing to be analyzed per video channel, to the number of separate video channels concurrently handled on a given site, to the possibly strongly varying illumination conditions and to possibly hindering occlusion phenomena, whereas on the other hand the nature of the human being renders long-lasting concentration of human operators very difficult. Therefore, the recourse to automatic systems enabling the reliable and robust detection of unusual events in video surveillance sequences is imperative.

The purpose of this project was to develop an unusual event detection algorithm and system architecture for video surveillance. One of the main drawbacks of the state-of-the-art solutions is the lack of generic usage, namely that the algorithms are valid only for a specific scenario. The novelties of the project consisted in using multiple cameras to improve the object extraction and tracking steps, as well as the development of a generic event representation to detect anomalies. In order to exploit the additional information provided by the different cameras, a geometric approach was investigated. This approach allowed us to establish the correspondence between the different views, which enabled in particular the proper handling of occlusion phenomena. Hidden Markov Models and motion patterns were studied to develop a generic event representation. The developed algorithm and system architecture were evaluated on several datasets representing different scenarios, and they were also practically experimented and tested.

Research activities

  • System design, test material procurement and evaluation methodology, to establish a design framework of the unusual event detection algorithm and system architecture, including organization of appropriate test material (e.g. images and video sequences) and the organization of a corresponding assessment methodology.
  • Algorithms development and implementation, encompassing the realization of the constitutive algorithms handling single camera object extraction and tracking, multi-view based occlusion processing, metadata generation, and unusual event detection.
  • Evaluation and testing, to systematically assess the performance of the algorithms and system architecture developed.
  • Ongoing technology survey and dissemination. Finally, a relevant effort was deployed for the ongoing survey of the state-of-the-art on the one hand, and for the publication of the scientific results on the other hand, including contribution to the organization/promotion of special sessions and special journal issues.
  • G. Mohammadi, F. Dufaux, T. Ha Minh and T. Ebrahimi, Multi-view video segmentation and tracking for video surveillance, SPIE Proc. Mobile Multimedia/Image Processing, Security and Applications, Orlando, FL, USA, April 2009
  • I. Ivanov, F. Dufaux, T. M. Ha, T. Ebrahimi, Towards Generic Detection of Unusual Events in Video Surveillance, 6th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS’09), Genoa, Italy, June 2009