MaskDiffuse: Text-Guided Face Mask Removal Based on Diffusion Models

Jingxia Lu, Xianxu Hou, Hao Li, Zhibin Peng, Linlin Shen, Lixin Fan

Research output: Chapter in Book/Conference proceedingConference contributionpeer-review

Abstract

As masked face images can significantly degrade the performance of face-related tasks, face mask removal remains an important and challenging task. In this paper, we propose a novel learning framework, called MaskDiffuse, to remove face masks based on Denoising Diffusion Probabilistic Model (DDPM). In particular, we leverage CLIP to fill the missing parts by guiding the reverse process of pretrained diffusion model with text prompts. Furthermore, we propose a multi-stage blending strategy to preserve the unmasked areas and a conditional resampling approach to make the generated contents consistent with the unmasked regions. Thus, our method achieves interactive user-controllable and identity-preserving masking removal with high quality. Both qualitative and quantitative experimental results demonstrate the superiority of our method for mask removal over alternative methods.

Original languageEnglish
Title of host publicationPattern Recognition and Computer Vision - 6th Chinese Conference, PRCV 2023, Proceedings
EditorsQingshan Liu, Hanzi Wang, Rongrong Ji, Zhanyu Ma, Weishi Zheng, Hongbin Zha, Xilin Chen, Liang Wang
PublisherSpringer Science and Business Media Deutschland GmbH
Pages435-446
Number of pages12
ISBN (Print)9789819985364
DOIs
Publication statusPublished - 2024
Externally publishedYes
Event6th Chinese Conference on Pattern Recognition and Computer Vision, PRCV 2023 - Xiamen, China
Duration: 13 Oct 202315 Oct 2023

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume14430 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference6th Chinese Conference on Pattern Recognition and Computer Vision, PRCV 2023
Country/TerritoryChina
CityXiamen
Period13/10/2315/10/23

Keywords

  • Diffusion models
  • Mask removal
  • Text-to-image

ASJC Scopus subject areas

  • Theoretical Computer Science
  • General Computer Science

Fingerprint

Dive into the research topics of 'MaskDiffuse: Text-Guided Face Mask Removal Based on Diffusion Models'. Together they form a unique fingerprint.

Cite this