Informed Democracy: Voting-based Novelty Detection for Action Recognition

Authors: Alina Roitberg*, Ziad Al-Halah*, Rainer Stiefelhagen

BMVC 2018
[paper] [poster] [bib]



Motivation

  • Action Recognition in Open-Set scenario
  • Goal: Identifying classes not previously seen by the classifier (novelty detection)



Overview

Contributions
  • New model for detecting previously unseen action classes
  • Generic framework for zero-shot action recognition in generalized case (GZS)
Main Idea
  • Leverage the predictive uncertainty of the classifiers
  • Two Concepts: the Leader and it’s Council
  • Leader: the classifier with the highest confidence score (→ votes for the predicted “known” category)
  • Council: a selected subset of the classifiers validates the leader’s decision
  • Informed Voting: voting for novelty based on the classifiers uncertainty is privileged to the council
Abstract
Novelty detection is crucial for real-life applications. While it is common in activity recognition to assume a closed-set setting, i.e. test samples are always of training categories, this assumption is impractical in a real-world scenario. Test samples can be of various categories including those never seen before during training. Thus, being able to know what we know and what we don’t know is decisive for the model to avoid what can be catastrophic consequences. We present in this work a novel approach for identifying samples of activity classes that are not previously seen by the classifier. Our model employs a voting-based scheme that leverages the estimated uncertainty of the individual classifiers in their predictions to measure the novelty of a new input sample. Furthermore, the voting is privileged to a subset of informed classifiers that can best estimate whether a sample is novel or not when it is classified to a certain known category. In a thorough evaluation on UCF-101 and HMDB-51, we show that our model consistently outperforms state-of-the-art in novelty detection. Additionally, by combining our model with off-the-shelf zero-shot learning (ZSL) approaches, our model leads to a significant improvement in action classification accuracy for the generalized ZSL setting.

Datasets & Features

Note: Please contact Alina Roitberg for further information regarding e.g. the dataset splits and features for the tasks of novelty detection and generalized zero-shot action recognition.