PLIMSeg: Pathology language–image matching for weakly supervised semantic segmentation of histopathology images

Semantic segmentation of tissues is crucial in aiding clinical diagnosis by quantitatively and objectively linking morphological characteristics to clinical outcomes. More and more tissue segmentation methods rely on weakly-supervised methods due to the time-consuming and labor-intensive burden of manual pixel-level annotations. Although current weakly supervised semantic segmentation (WSSS) methods achieve significant performance by Class Activation Maps (CAM), these methods cannot perform well on histopathological images due to the homogeneous features of different tissue types. Moreover, some pathology Contrastive Language–Image Pretraining (CLIP) models have great representation capability for histopathology, but they have not been fully used to capture homogeneous features in histopathology. To solve these challenges, we propose a novel framework named PLIMSeg (Pathology Language–Image Matching for Weakly Supervised Semantic Segmentation), which aims to leverage Contrastive Language–Image Pretraining into WSSS. Specifically, PLIMSeg utilizes pathology CLIP as the feature extractor, aiming to utilize the strong representation capability of pre-trained language–image models to represent features. Then we design three losses based on pathology language–image matching (PLIM), to constrain the CAMs generated by the original image encoder. With these constraints, PLIMSeg can generate a more complete and precise pseudo mask for segmentation. Our PLIMSeg has a better performance compared with other weakly supervised segmentation methods of pathology on LUAD-HistoSeg and BCSS-WSSS datasets, setting a new state-of-the-art for WSSS of histopathology images.