FProtoSeg: Fine-grained prototype alignment for Weakly Supervised Semantic Segmentation of histopathology images

Weakly Supervised Semantic Segmentation (WSSS) for histopathology tissues has significantly improved to reduce the burden of annotation through class activation maps (CAMs). Nevertheless, accurate segmentation remains challenging due to the high intra-class variability across patients and the subtle inter-class differences, as early-stage abnormal cells often resemble normal ones. Moreover, WSSS methods tend to emphasize the most discriminative features, often neglecting outlier features that are from less common or more subtle morphological variations within a class. Despite progress in recent approaches, the reliance on a coarse, one-to-many mapping hampers their capacity to capture subtle, pixel-level distinctions. Motivated by this limitation, we hypothesize that adopting a fine-grained, one-to-one alignment will yield more accurate and complete segmentation outcomes. Therefore, we propose a novel fine-grained prototype alignment framework named FProtoSeg, with structure-aware prototype modeling and text-aware prototype alignment to extract more specific features and activate more complete CAMs. Specifically, structure-aware prototype modeling captures class characteristics by employing prototypes, thereby adapting to the semantic attributes of different instances. Text-aware prototype alignment aligns visual and textual features to enhance prototype awareness, ensuring that instance feature distributions are in harmony with text features. Experimental results demonstrate that FProtoSeg achieves state-of-the-art performance, attaining a mean Intersection over Union (mIoU) of 71.21% on the BCSS-WSSS dataset and 76.64% on the LUAD-HistoSeg dataset, significantly outperforming existing methods.