Abstract
Studies on self-supervised visual representation learning (SSL) improve encoder backbones to discriminate training samples without labels. While CNN encoders via SSL achieve comparable recognition performance to those via supervised learning, their network attention is under-explored for further improvement. Motivated by the transformers that explore visual attention effectively in recognition scenarios, we propose a CNN Attention REvitalization (CARE) framework to train attentive CNN encoders guided by transformers in SSL. The proposed CARE framework consists of a CNN stream (C-stream) and a transformer stream (T-stream), where each stream contains two branches. C-stream follows an existing SSL framework with two CNN encoders, two projectors, and a predictor. T-stream contains two transformers, two projectors, and a predictor. T-stream connects to CNN encoders and is in parallel to the remaining C-Stream. During training, we perform SSL in both streams simultaneously and use the T-stream output to supervise C-stream. The features from CNN encoders are modulated in T-stream for visual attention enhancement and become suitable for the SSL scenario. We use these modulated features to supervise C-stream for learning attentive CNN encoders. To this end, we revitalize CNN attention by using transformers as guidance. Experiments on several standard visual recognition benchmarks, including image classification, object detection, and semantic segmentation, show that the proposed CARE framework improves CNN encoder backbones to the state-of-the-art performance.
Original language | English |
---|---|
Title of host publication | Advances in Neural Information Processing Systems 34 (NeurIPS 2021) |
Editors | Marc'Aurelio Ranzato, Alina Beygelzimer, Yann Dauphin, Percy S. Liang, Jenn Wortman Vaughan |
Publisher | NeurIPS |
Pages | 4193-4206 |
Number of pages | 14 |
ISBN (Print) | 9781713845393 |
Publication status | Published - 11 Oct 2021 |
Event | Thirty-fifth Conference on Neural Information Processing Systems - Virtual Duration: 6 Dec 2021 → 14 Dec 2021 |
Publication series
Name | Advances in Neural Information Processing Systems |
---|---|
Volume | 34 |
ISSN (Print) | 1049-5258 |
Conference
Conference | Thirty-fifth Conference on Neural Information Processing Systems |
---|---|
Abbreviated title | NeurIPS 2021 |
Period | 6/12/21 → 14/12/21 |
Bibliographical note
Funding Information:Acknowledgement. This work is supported by CCF-Tencent Open Fund, the General Research Fund of Hong Kong No.27208720 and the EPSRC Programme Grant Visual AI EP/T028572/1.
Publisher Copyright:
© 2021 Neural information processing systems foundation. All rights reserved.
ASJC Scopus subject areas
- Computer Networks and Communications
- Information Systems
- Signal Processing