In a controlled comparison, removing descriptions of prominent image regions from long queries hurt retrieval more than deleting the same amount of text at random, the researchers report.
Qiuyu Kong, Zanxi Ruan, Marco Cristani and Yiming Wang call this pattern “saliency collapse” and propose RoSA to help models use contextual cues.
The paper attributes the problem to image-text models relying heavily on prominent cues. RoSA creates prominent and contextual views of each image-caption pair, deriving image regions with a visual-saliency tool and independently splitting the caption with a language model.
The framework trains models using both parts of the caption and adds objectives that align image and text information. The authors reported higher retrieval scores in the long-description benchmarks they tested, including gains when RoSA’s auxiliary objectives were added to other fine-tuning approaches. They also identified additional preprocessing and training as costs of the approach.
