Enhancing Unsupervised Semantic Segmentation Through Context-Aware Clustering

Wei Zhuo, Yuan Wang, Junliang Chen, Songhe Deng, Zhi Wang, Linlin Shen, Wenwu Zhu

Research output: Journal PublicationArticlepeer-review


Despite the great progress of semantic segmentation with supervised learning, annotating large amounts of pixel-wise labels is, however, very expensive and time-consuming. To this end, Unsupervised Semantic Segmentation(USS) has been proposed to learn semantic segmentation, without any form of annotations. This approach involves dense prediction of semantics which is however challenging due to the unreliable nature of local representations. To solve this problem, we propose a newly context-aware unsupervised semantic segmentation framework, which aims to enhance the unsupervised semantic segmentation by leveraging contextual knowledge within and across images. In particular, we introduce a training strategy based on our Pyramid Semantic Guidance (PSG), which utilizes holistic semantics on pyramid views to guide pixel clustering with a siamese network-based framework. Additionally, we introduce a Context-Aware Embedding (CAE) module to fuse global features with low-level geometrical and appearance representations. We evaluate our method on the COCO-Stuff dataset and achieved competitive results compared to both the convolutional and ViT-based USS methods. Specifically, we attain significant improvements of +4.5% and +5% mIoU for Stuff and all class segmentation respectively, compared to previous approaches that employ unsupervised convolutional backbones.

Original languageEnglish
Pages (from-to)1-13
Number of pages13
JournalIEEE Transactions on Multimedia
Publication statusAccepted/In press - 2024
Externally publishedYes


  • Annotations
  • Computer science
  • context-aware embedding
  • Convolutional neural networks
  • pseudo labeling
  • self-supervised learning
  • semantic clustering
  • Semantic segmentation
  • Semantics
  • Training
  • Unsupervised learning
  • Unsupervised semantic segmentation

ASJC Scopus subject areas

  • Signal Processing
  • Media Technology
  • Computer Science Applications
  • Electrical and Electronic Engineering


Dive into the research topics of 'Enhancing Unsupervised Semantic Segmentation Through Context-Aware Clustering'. Together they form a unique fingerprint.

Cite this