Fine-grained facial description generation with retrieval augmentation

With the rapid development of Visual Large Language Models (VLLMs), significant improvements have been made in vision tasks such as image description generation and visual question answering. However, for the facial description task, the direct application of VLLMs poses additional challenges due to the lack of fine-grained facial description data and tailored prompts for focusing on facial features. In this work, we first construct a Question–Answer Facial Description dataset (QA-FaceDesc) specifically designed for facial description tasks. Then, to leverage QA-FaceDesc with VLLMs, we propose a Fine-grained Facial Description framework (FFD) that consists of a Facial Detector Module (FDM), a QA-FaceDesc Retrieval Augmented Module (QA-FaceRAM), and a Retrieval Module (RM). In addition, to comprehensively assess performance, we construct three test sets based on existing datasets for fine-grained facial descriptions. Through extensive experiments, our approach outperforms all the baseline models even without VLLMs fine-tuning.