Krista Kappeler (master project)

MSc Semester Project


Affective Multimedia Content Analysis




Krista Kappeler


Prof. Dr. Touradj Ebrahimi

Assistant: Ashkan Yazdani


June 26,  2011






With the increasing popularity of video-on-demand (VOD) and personalized movie recommendation services, the generation of automatic content descriptions for indexing, retrieval and personalization purposes is a key issue. The technology required to achieve this is referred to as multimedia content analysis (MCA) and aims at “bridging the semantic gap”, that is, to develop models of the relationship between low level features and the semantics conveyed by audiovisual (AV) content. To this end, two different approaches have been adopted for MCA so far: cognitive and emotional. The cognitive approach analyses a piece of AV content in terms of the semantics of a scene: location, characters and events. In the past decade, most of the MCA-related research effort focused on these methods. The emotional approach, on the other hand attempts to characterize video content by the emotions it elicits in viewers. This approach is often referred to as affective MCA or affective content analysis. and predicts or (partially) infers viewers’ emotional reactions when perceiving video content. The latter has been less thoroughly investigated when compared to cognitive approach, but its importance has been rapidly increasing with the growing awareness of the role that the emotional load of multimedia and viewers’ reactions to it, play in VOD concepts and personalized video recommendation.


  •   Study of the state of the art

  1.             Emotion and affect psycho-physiology and assessment techniques

  2.             Complete literature review on Affective Computing (AC)

  3.             Studying different modalities for AC  >> select a model 

  4.             Affective Multimedia Content Analysis (AMCA):

                      -Video  -Audio  -Fusion

  •   AMCA for music video clips (Data will be provided by the lab)

  1.             Feature extraction and classification for video

  2.             Feature extraction and classification for audio (music)

  3.             Fusion

  • Database collection, annotation and analysis

                  1.             Selection of new test material (video from YouTube)

                  2.             (Non) Web-based annotation of these data (subjective test)

                  3.             Analysis of correlation or fuzzy rules generation for mapping of dimensional model of affect (arousal, valence.      etc.)  to categorical model of affect.

                  4.             AMCA for this database.