Abstract:

This paper presents algorithms for finding the meanings of the audio-visual video patterns obtained in the unsupervised discovery process. This problem is interesting in domains where neither perceptual patterns nor semantic concepts have simple structures. The patterns in the video are modeled with hierarchical hidden Markov models, with efficient algorithms to jointly learn the model parameters, the optimal model complexity, as well as the relevant feature subsets. The meanings are contained in words of the speech transcript of the video. The pattern-word association is obtained via co-occurrence analysis and machine translation models. Promising results are obtained on TRECVID news videos: video patterns that associate with distinct topics such as el-nino and politics are itentified; a temporal structure model compares favorably to a non-temporal clustering algorithm.