Story-based Video Retrieval

---

Paper and Presentation

Sep 2014 - NEW! We have the extended version of the ICMR paper in IJMIR. Check it out.

Story-based Video Retrieval in TV series using Plot Synopses
Makarand Tapaswi, Martin Baeuml and Rainer Stiefelhagen
ACM International Conference on Multimedia Retrieval (ICMR Oral, Full paper) Glasgow, Scotland, Apr 2014
[paper] [presentation] [demo poster] [HQ video]

---

Demo

A demo of the proposed retrieval system was presented at CVPR 2014 and can also be found here. Please contact me to obtain a login. Below is a video of the retrieval in action

---

Contributions

• Story-based video retrieval in TV series
• Alignment of Plot Synopsis (such as from Wikipedia) to shots in the video
• Person identification and subtitles used as guidelines for alignment
• Annotation and evaluation on the entire Season 5 of Buffy the Vampire Slayer (TV series)

---

Dataset

Buffy the Vampire Slayer (Season 5, all Episodes 1 to 22)
The dataset package contains the plot synopses used in this work (as they might have changed on Wiki) and the annotations from one user indicating which shot is assigned to which sentence.
PRACTICAL NOTE: The frame numbers and time-stamps are for Region 2 DVDs (PAL) and have a 25 fps frame rate.

• Video Events videvents.tar.gz
  Contains a list of automatically detected video events: shots, title song, credits.
  Format: start_frame, start_time, TYPE, [end_frame,] [end_time]
  
• Plot Synopsis synopses.tar.gz
  Contains plot synopsis extracted from Wikipedia and stored in simple text format
  
• Shot to Sentence annotation annots.tar.gz
  Contains annotations for plot synopsis matching them to shots in the video
  The files are encoded in JSON format and contain the sentence, start/end shot number, and the list of characters that appear in that sentence as inferred by reading the plot synopsis only
  

Links