MUMC: Minimizing uncertainty of mixture of cues

Badri N. Patro, Vinod K. Kurmi, Sandeep Kumar, Vinay P. Namboodiri

Research output: Contribution to journalArticlepeer-review

1 Citation (SciVal)


Generating natural questions from an image is a semantic task that requires using vision and language modalities to learn multimodal representations. Images can have multiple visual and language cues such as places, captions, and tags. In this paper, we propose a principled deep Bayesian learning framework that combines these cues to produce natural questions. We observe that with the addition of more cues and by minimizing uncertainty in the among cues, the Bayesian network becomes more confident. We propose a Minimizing Uncertainty of Mixture of Cues (MUMC), that minimizes uncertainty present in a mixture of cues experts for generating probabilistic questions. This is a Bayesian framework and the results show a remarkable similarity to natural questions as validated by a human study. Ablation studies of our model indicate that a subset of cues is inferior at this task and hence the principled fusion of cues is preferred. Further, we observe that the proposed approach substantially improves over state-of-the-art benchmarks on the quantitative metrics (BLEU-n, METEOR, ROUGE, and CIDEr). Here, we provide project link for Deep Bayesian VQG:

Original languageEnglish
Article number104280
JournalImage and Vision Computing
Early online date24 Aug 2021
Publication statusPublished - 1 Nov 2021

Bibliographical note

Publisher Copyright:
© 2021 Elsevier B.V.


  • CNN
  • Encoder-decoder
  • LSTM
  • Mixture of cues
  • Paraphrase
  • Uncertainty estimation
  • Visual Question Answering
  • Visual Question Generation

ASJC Scopus subject areas

  • Signal Processing
  • Computer Vision and Pattern Recognition


Dive into the research topics of 'MUMC: Minimizing uncertainty of mixture of cues'. Together they form a unique fingerprint.

Cite this