Decoupled Global-Local Alignment for Improving Compositional Understanding

Motivation

Prior studies have demonstrated that CLIP lacks compositional understanding—particularly in comprehending relational and attributive concepts. While existing approaches attempt to address this limitation through fine-tuning with hard negative samples, they exhibit a critical drawback: the improvement in compositional understanding often comes at the expense of significant degradation in general performance.

This presents a crucial question: how can we enhance CLIP's compositional understanding while preserving its general capabilities to maintain a good trade-off between compositional understanding and general performance?

LLM-driven Hard Negative Generation

Unlike previous rule-based or unmasking-based hard negative generation methods, we propose a hard negative generation method leveraging large language models (LLMs). Specifically, we first use ChatGPT to generate high-quality rewritten examples for each type of negative sample. From these, we manually select 50 examples to serve as templates for large-scale rewrites. Subsequently, we harness the context learning capabilities of the LLama3.1-8B-instruct model to conduct large-scale rewrites, generating high-quality hard negative samples for subsequent fine-tuning.

DeGLA training framework

To address the challenge of the degradation in general performance and improve the compositional understanding, we propose a DeGLA training framework, which decouples the global and local alignment during the training process.
• Global Alignment: Based on the NegCLIP loss, we integrate the self-distillation of the fine-tuned model and the frozen EMA model to constrain subtle adjustments in the pre-trained embedding space.。
• Local Alignment: We propose two local alignment losses, Image-Grounded Contrast (IGC) and Text-Grounded Contrast (TGC), to further enhance the model's understanding of compositional concepts..

BiBTex

If you find this work useful, please consider citing our paper:

@misc{hu2025decoupledgloballocalalignmentimproving,
  title={Decoupled Global-Local Alignment for Improving Compositional Understanding}, 
  author={Xiaoxing Hu and Kaicheng Yang and Jun Wang and Haoran Xu and Ziyong Feng and Yupei Wang},
  year={2025},
  eprint={2504.16801},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2504.16801}, 
}