Semantic Navigation of PowerPoint-Based Lecture Video for AutoNote Generation

Chengpei Xu, Wenjing Jia, Ruomei Wang, Xiangjian He, Baoquan Zhao, Yuanfang Zhang

Research output: Journal PublicationArticlepeer-review


With the increasing popularity of open educational resources in the past few decades, more and more users watch online videos to gain knowledge. However, most educational videos only provide monotonous navigation tools and lack elaborating annotations. This makes the task of locating interesting contents time consuming. To address this limitation, in this article, we propose a slide-based video navigation tool that is able to extract the hierarchical structure and semantic relationship of visual entities in videos, by integrating multichannel information. Features of visual entities are first extracted from the presentation slides by a novel deep learning framework. Then, we propose a clustering approach to extract hierarchical relationships between visual entities (e.g., formulas, texts, or graphs appearing in educational slides). We use this information to associate visual entities with their corresponding audio speech text, by evaluating their semantic relationship. We present two cases where we use the structured data produced by this tool to generate a multilevel table of contents and notes to provide additional navigation materials for learning. The evaluation experiments demonstrate the effectiveness of our proposed solutions for visual entity extraction, hierarchical relationship extraction, as well as corresponding speech text matching. The user study also shows promising improvement in the autogenerated table of contents and notes for facilitating learning.

Original languageEnglish
Pages (from-to)1-17
Number of pages17
JournalIEEE Transactions on Learning Technologies
Issue number1
Publication statusPublished - 1 Feb 2023


  • AutoNote generation
  • educational videos
  • hierarchical relationship extraction
  • video navigation
  • visual entity segmentation

ASJC Scopus subject areas

  • General Engineering
  • Education
  • Computer Science Applications


Dive into the research topics of 'Semantic Navigation of PowerPoint-Based Lecture Video for AutoNote Generation'. Together they form a unique fingerprint.

Cite this