Zero-shot referring expression comprehension via guidance of Multimodal Large Language Models

Zero-shot Referring Expression Comprehension (REC) aims to locate the referring object in an image that corresponds to the given natural language description (Referring Expression). This demands the ability of fine-grained instance recognition within complex visual scenes and textual descriptions, as well as comprehension of the relationships between different instances. Existing zero-shot REC methods based on large-scale Visual Language Alignment (VLA) model perform poorly when encountering ambiguous referring expression. To address this issue, we propose a zero-shot referring expression comprehension paradigm guided by Multimodal Large Language Model (MLLM), which introduced a text expansion, and a new activation map aggregation method for VLA model. We leverages the image-to-text generation capabilities of MLLM to refine the referring expression and predict the category of the referring object. In addition, we present an effective activation map aggregation method for VLA model, which extracts higher-quality activation map related to the referring expression from VLA model and, matches the map with the candidate boxes generated by an open-vocabulary detection model to locate the referring object. Extensive experiments demonstrate that our method outperforms the most advanced state-of-the-art methods on the RefCOCO/+/g datasets, with a performance improvement of up to 11.07%.