Automatic visual theme discovery from joint image and text corpora

This paper presents an unsupervised visual theme discovery framework as a better (more compact and effective) alternative for semantic representation of visual contents. Firstly, a tag filtering algorithm was proposed focusing on the tag's ability of visual content description. Then a spectral clustering algorithm is applied to cluster tags into visual themes based on their visual similarity and semantic similarity measures. User studies have been conducted to evaluate the effectiveness and rationality of the discovered visual themes and obtain promising results. Additionally, two common computer vision tasks, example based image search and keyword based image search to explore potential applications of the proposed framework. The experimental results show that visual themes significantly outperform tags on semantic image understanding and achieve state-of-art performance in these two tasks.