Differential Attention for Visual Question Answering

Badri Patro, Vinay P. Namboodiri

Research output: Chapter in Book/Report/Conference proceedingConference contribution

15 Citations (Scopus)

Abstract

In this paper we aim to answer questions based on images when provided with a dataset of question-answer pairs for a number of images during training. A number of methods have focused on solving this problem by using image based attention. This is done by focusing on a specific part of the image while answering the question. Humans also do so when solving this problem. However, the regions that the previous systems focus on are not correlated with the regions that humans focus on. The accuracy is limited due to this drawback. In this paper, we propose to solve this problem by using an exemplar based method. We obtain one or more supporting and opposing exemplars to obtain a differential attention region. This differential attention is closer to human attention than other image based attention methods. It also helps in obtaining improved accuracy when answering questions. The method is evaluated on challenging benchmark datasets. We perform better than other image based attention methods and are competitive with other state of the art methods that focus on both image and questions.

Original languageEnglish
Title of host publicationProceedings - 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018
PublisherIEEE
Pages7680-7688
Number of pages9
ISBN (Electronic)9781538664209
DOIs
Publication statusPublished - 14 Dec 2018
Event31st Meeting of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018 - Salt Lake City, USA United States
Duration: 18 Jun 201822 Jun 2018

Publication series

NameProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
ISSN (Print)1063-6919

Conference

Conference31st Meeting of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018
CountryUSA United States
CitySalt Lake City
Period18/06/1822/06/18

ASJC Scopus subject areas

  • Software
  • Computer Vision and Pattern Recognition

Cite this

Patro, B., & Namboodiri, V. P. (2018). Differential Attention for Visual Question Answering. In Proceedings - 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2018 (pp. 7680-7688). [8578899] (Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition). IEEE. https://doi.org/10.1109/CVPR.2018.00801