Visually Precise Query

Riddhiman Dasgupta; Francis Tom; Sudhir Kumar; Mithun Das Gupta; Yokesh Kumar; Badri N. Patro; Vinay P. Namboodiri

doi:10.1145/3394171.3413558

Visually Precise Query

Riddhiman Dasgupta, Francis Tom, Sudhir Kumar, Mithun Das Gupta, Yokesh Kumar, Badri N. Patro, Vinay P. Namboodiri

Department of Computer Science

Research output: Chapter or section in a book/report/conference proceeding › Chapter in a published conference proceeding

1 Citation (SciVal)

Abstract

We present the problem of Visually Precise Query (VPQ) generation which enables a more intuitive match between a user's information need and an e-commerce site's product description. Given an image of a fashion item, what is the most optimum search query that will retrieve the exact same or closely related product(s) with high probability. In this paper we introduce the task of VPQ generation which takes a product image and its title as its input and provides aword level extractive summary of the title, containing a list of salient attributes, which can now be used as a query to search for similar products. We collect a large dataset of fashion images and their titles and merge it with an existing research dataset which was created for a different task. Given the image and title pair, VPQ problem is posed as identifying a non-contiguous collection of spans within the title. We provide a dataset of around 400K image, title and corresponding VPQ entries and release it to the research community. We provide a detailed description of the data collection process as well as discuss the future direction of research for the problem introduced in this work. We provide the standard text as well as visual domain baseline comparisons and also provide multi-modal baseline models to analyze the task introduced in this work. Finally, we propose a hybrid fusion model which promises to be the direction of research in the multi-modal community.

Original language	English
Title of host publication	MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia
Place of Publication	U. S. A.
Publisher	Association for Computing Machinery
Pages	3550-3558
Number of pages	9
ISBN (Electronic)	9781450379885
DOIs	https://doi.org/10.1145/3394171.3413558
Publication status	Published - 12 Oct 2020
Event	28th ACM International Conference on Multimedia, MM 2020 - Virtual, Online, USA United States Duration: 12 Oct 2020 → 16 Oct 2020

Publication series

Name	MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia

Conference

Conference	28th ACM International Conference on Multimedia, MM 2020
Country/Territory	USA United States
City	Virtual, Online
Period	12/10/20 → 16/10/20

Keywords

e-commerce
hybrid fusion
multi-modal information retrieval
visual attributes

ASJC Scopus subject areas

Software
Computer Graphics and Computer-Aided Design
Human-Computer Interaction

Access to Document

10.1145/3394171.3413558

Cite this

Dasgupta, R., Tom, F., Kumar, S., Das Gupta, M., Kumar, Y., Patro, B. N., & Namboodiri, V. P. (2020). Visually Precise Query. In MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia (pp. 3550-3558). (MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia). Association for Computing Machinery. https://doi.org/10.1145/3394171.3413558

Visually Precise Query. / Dasgupta, Riddhiman; Tom, Francis; Kumar, Sudhir et al.
MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia. U. S. A.: Association for Computing Machinery, 2020. p. 3550-3558 (MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia).

Research output: Chapter or section in a book/report/conference proceeding › Chapter in a published conference proceeding

Dasgupta, R, Tom, F, Kumar, S, Das Gupta, M, Kumar, Y, Patro, BN & Namboodiri, VP 2020, Visually Precise Query. in MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia. MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia, Association for Computing Machinery, U. S. A., pp. 3550-3558, 28th ACM International Conference on Multimedia, MM 2020, Virtual, Online, USA United States, 12/10/20. https://doi.org/10.1145/3394171.3413558

@inproceedings{7d73a2f9c9d14c148deeb64c15b3295c,

title = "Visually Precise Query",

abstract = "We present the problem of Visually Precise Query (VPQ) generation which enables a more intuitive match between a user's information need and an e-commerce site's product description. Given an image of a fashion item, what is the most optimum search query that will retrieve the exact same or closely related product(s) with high probability. In this paper we introduce the task of VPQ generation which takes a product image and its title as its input and provides aword level extractive summary of the title, containing a list of salient attributes, which can now be used as a query to search for similar products. We collect a large dataset of fashion images and their titles and merge it with an existing research dataset which was created for a different task. Given the image and title pair, VPQ problem is posed as identifying a non-contiguous collection of spans within the title. We provide a dataset of around 400K image, title and corresponding VPQ entries and release it to the research community. We provide a detailed description of the data collection process as well as discuss the future direction of research for the problem introduced in this work. We provide the standard text as well as visual domain baseline comparisons and also provide multi-modal baseline models to analyze the task introduced in this work. Finally, we propose a hybrid fusion model which promises to be the direction of research in the multi-modal community.",

keywords = "e-commerce, hybrid fusion, multi-modal information retrieval, visual attributes",

author = "Riddhiman Dasgupta and Francis Tom and Sudhir Kumar and {Das Gupta}, Mithun and Yokesh Kumar and Patro, {Badri N.} and Namboodiri, {Vinay P.}",

year = "2020",

month = oct,

day = "12",

doi = "10.1145/3394171.3413558",

language = "English",

series = "MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia",

publisher = "Association for Computing Machinery",

pages = "3550--3558",

booktitle = "MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia",

address = "USA United States",

note = "28th ACM International Conference on Multimedia, MM 2020 ; Conference date: 12-10-2020 Through 16-10-2020",

}

TY - GEN

T1 - Visually Precise Query

AU - Dasgupta, Riddhiman

AU - Tom, Francis

AU - Kumar, Sudhir

AU - Das Gupta, Mithun

AU - Kumar, Yokesh

AU - Patro, Badri N.

AU - Namboodiri, Vinay P.

PY - 2020/10/12

Y1 - 2020/10/12

N2 - We present the problem of Visually Precise Query (VPQ) generation which enables a more intuitive match between a user's information need and an e-commerce site's product description. Given an image of a fashion item, what is the most optimum search query that will retrieve the exact same or closely related product(s) with high probability. In this paper we introduce the task of VPQ generation which takes a product image and its title as its input and provides aword level extractive summary of the title, containing a list of salient attributes, which can now be used as a query to search for similar products. We collect a large dataset of fashion images and their titles and merge it with an existing research dataset which was created for a different task. Given the image and title pair, VPQ problem is posed as identifying a non-contiguous collection of spans within the title. We provide a dataset of around 400K image, title and corresponding VPQ entries and release it to the research community. We provide a detailed description of the data collection process as well as discuss the future direction of research for the problem introduced in this work. We provide the standard text as well as visual domain baseline comparisons and also provide multi-modal baseline models to analyze the task introduced in this work. Finally, we propose a hybrid fusion model which promises to be the direction of research in the multi-modal community.

AB - We present the problem of Visually Precise Query (VPQ) generation which enables a more intuitive match between a user's information need and an e-commerce site's product description. Given an image of a fashion item, what is the most optimum search query that will retrieve the exact same or closely related product(s) with high probability. In this paper we introduce the task of VPQ generation which takes a product image and its title as its input and provides aword level extractive summary of the title, containing a list of salient attributes, which can now be used as a query to search for similar products. We collect a large dataset of fashion images and their titles and merge it with an existing research dataset which was created for a different task. Given the image and title pair, VPQ problem is posed as identifying a non-contiguous collection of spans within the title. We provide a dataset of around 400K image, title and corresponding VPQ entries and release it to the research community. We provide a detailed description of the data collection process as well as discuss the future direction of research for the problem introduced in this work. We provide the standard text as well as visual domain baseline comparisons and also provide multi-modal baseline models to analyze the task introduced in this work. Finally, we propose a hybrid fusion model which promises to be the direction of research in the multi-modal community.

KW - e-commerce

KW - hybrid fusion

KW - multi-modal information retrieval

KW - visual attributes

UR - http://www.scopus.com/inward/record.url?scp=85105153924&partnerID=8YFLogxK

U2 - 10.1145/3394171.3413558

DO - 10.1145/3394171.3413558

M3 - Chapter in a published conference proceeding

AN - SCOPUS:85105153924

T3 - MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia

SP - 3550

EP - 3558

BT - MM 2020 - Proceedings of the 28th ACM International Conference on Multimedia

PB - Association for Computing Machinery

CY - U. S. A.

T2 - 28th ACM International Conference on Multimedia, MM 2020

Y2 - 12 October 2020 through 16 October 2020

ER -

Visually Precise Query

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this