CoAPT: Context attribute words for prompt tuning

Authors: Gun Lee, Subin An, Sungyoung Baik, Soochahn Lee

TL;DR: CoAPT improves prompt tuning for vision-language models by incorporating context-aware attribute words, leading to better generalization across visual recognition tasks.

What this paper is about

Prompt tuning has become a popular way to adapt large vision-language models (like CLIP) to downstream tasks without full fine-tuning. However, learned prompts often overfit to seen classes and fail to generalize, especially in few-shot or zero-shot settings.

Comparative overview of CoAPT. Existing soft prompt tuning methods, such as CoOp, do not fully utilize the text encoder input. The empty slots in the text query can be enhanced by integrating additional hard prompts. For the sample classes from the OxfordPets dataset, CoAPT achieves better classification accuracy for both base and new classes.

Key idea

CoAPT introduces context attribute words into the prompt tuning process, enriching the learned prompts with semantic attributes that capture meaningful visual and contextual properties. This provides more structured and transferable prompts compared to purely learned token embeddings.

Overview of CoAPT method with baseline prompt learning model. The CoAPT method consists of two steps. First, attribute words are generated using a language model, which is a one-time process. Second, during prompt learning, these words are combined with the soft prompt and class token. Inputs generate queries processed by a Meta-network, adding a bias term to the text queries. The combined image–text queries are then used to maximize the score for the ground-truth class.

Why it matters

This work pushes prompt tuning closer to practical deployment by improving generalization, which is critical for real-world applications where models encounter unseen categories.