CascadeXML: Rethinking Transformers for End-to-end Multi-resolution Training in Extreme Multi-label Classification

Siddhant Kharbanda; Atmadeep Banerjee; Erik Schultheis; Rohit Babbar

CascadeXML: Rethinking Transformers for End-to-end Multi-resolution Training in Extreme Multi-label Classification

Siddhant Kharbanda, Atmadeep Banerjee, Erik Schultheis, Rohit Babbar

Department of Computer Science

Aalto University

Research output: Chapter or section in a book/report/conference proceeding › Chapter in a published conference proceeding

17 Citations (SciVal)

Abstract

Extreme Multi-label Text Classification (XMC) involves learning a classifier that can assign an input with a subset of most relevant labels from millions of label choices. Recent approaches, such as XR-Transformer and LightXML, leverage a transformer instance to achieve state-of-the-art performance. However, in this process, these approaches need to make various trade-offs between performance and computational requirements. A major shortcoming, as compared to the Bi-LSTM based AttentionXML, is that they fail to keep separate feature representations for each resolution in a label tree. We thus propose CascadeXML, an end-to-end multi-resolution learning pipeline, which can harness the multi-layered architecture of a transformer model for attending to different label resolutions with separate feature representations. CascadeXML significantly outperforms all existing approaches with non-trivial gains obtained on benchmark datasets consisting of up to three million labels. Code for CascadeXML will be made publicly available at https://github.com/xmc-aalto/cascadexml.

Original language	English
Title of host publication	Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022
Editors	S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, A. Oh
Publisher	Neural Information Processing Systems Foundation, Inc.
ISBN (Electronic)	9781713871088
Publication status	Published - 9 Dec 2022
Event	36th Conference on Neural Information Processing Systems, NeurIPS 2022 - New Orleans, USA United States Duration: 28 Nov 2022 → 9 Dec 2022

Publication series

Name	Advances in Neural Information Processing Systems
Volume	35
ISSN (Print)	1049-5258

Conference

Conference	36th Conference on Neural Information Processing Systems, NeurIPS 2022
Country/Territory	USA United States
City	New Orleans
Period	28/11/22 → 9/12/22

Bibliographical note

Funding Information:
The authors would like to thank Devaansh Gupta and Mohammadreza Qaraei for useful discussions. They also acknowledge the support of CSC – IT Center for Science, Finland, as well as the Aalto Science-IT project, for providing the required computational resources. This research is supported in part by Academy of Finland grants : Decision No. 348215 and 347707.

Publisher Copyright:
© 2022 Neural information processing systems foundation. All rights reserved.

Funding

The authors would like to thank Devaansh Gupta and Mohammadreza Qaraei for useful discussions. They also acknowledge the support of CSC – IT Center for Science, Finland, as well as the Aalto Science-IT project, for providing the required computational resources. This research is supported in part by Academy of Finland grants : Decision No. 348215 and 347707.

ASJC Scopus subject areas

Computer Networks and Communications
Information Systems
Signal Processing

Access to Document

https://proceedings.neurips.cc/paper_files/paper/2022/hash/0e0157ce5ea15831072be4744cbd5334-Abstract-Conference.html

Cite this

Kharbanda, S., Banerjee, A., Schultheis, E., & Babbar, R. (2022). CascadeXML: Rethinking Transformers for End-to-end Multi-resolution Training in Extreme Multi-label Classification. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, & A. Oh (Eds.), Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022 (Advances in Neural Information Processing Systems; Vol. 35). Neural Information Processing Systems Foundation, Inc.. https://proceedings.neurips.cc/paper_files/paper/2022/hash/0e0157ce5ea15831072be4744cbd5334-Abstract-Conference.html

CascadeXML: Rethinking Transformers for End-to-end Multi-resolution Training in Extreme Multi-label Classification. / Kharbanda, Siddhant; Banerjee, Atmadeep; Schultheis, Erik et al.
Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022. ed. / S. Koyejo; S. Mohamed; A. Agarwal; D. Belgrave; K. Cho; A. Oh. Neural Information Processing Systems Foundation, Inc., 2022. (Advances in Neural Information Processing Systems; Vol. 35).

Research output: Chapter or section in a book/report/conference proceeding › Chapter in a published conference proceeding

Kharbanda, S, Banerjee, A, Schultheis, E & Babbar, R 2022, CascadeXML: Rethinking Transformers for End-to-end Multi-resolution Training in Extreme Multi-label Classification. in S Koyejo, S Mohamed, A Agarwal, D Belgrave, K Cho & A Oh (eds), Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022. Advances in Neural Information Processing Systems, vol. 35, Neural Information Processing Systems Foundation, Inc., 36th Conference on Neural Information Processing Systems, NeurIPS 2022, New Orleans, USA United States, 28/11/22. <https://proceedings.neurips.cc/paper_files/paper/2022/hash/0e0157ce5ea15831072be4744cbd5334-Abstract-Conference.html>

Kharbanda S, Banerjee A, Schultheis E, Babbar R. CascadeXML: Rethinking Transformers for End-to-end Multi-resolution Training in Extreme Multi-label Classification. In Koyejo S, Mohamed S, Agarwal A, Belgrave D, Cho K, Oh A, editors, Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022. Neural Information Processing Systems Foundation, Inc. 2022. (Advances in Neural Information Processing Systems).

Kharbanda, Siddhant ; Banerjee, Atmadeep ; Schultheis, Erik et al. / CascadeXML : Rethinking Transformers for End-to-end Multi-resolution Training in Extreme Multi-label Classification. Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022. editor / S. Koyejo ; S. Mohamed ; A. Agarwal ; D. Belgrave ; K. Cho ; A. Oh. Neural Information Processing Systems Foundation, Inc., 2022. (Advances in Neural Information Processing Systems).

@inproceedings{9e9f9fa47512406bbe2ba09b5541ec5f,

title = "CascadeXML: Rethinking Transformers for End-to-end Multi-resolution Training in Extreme Multi-label Classification",

abstract = "Extreme Multi-label Text Classification (XMC) involves learning a classifier that can assign an input with a subset of most relevant labels from millions of label choices. Recent approaches, such as XR-Transformer and LightXML, leverage a transformer instance to achieve state-of-the-art performance. However, in this process, these approaches need to make various trade-offs between performance and computational requirements. A major shortcoming, as compared to the Bi-LSTM based AttentionXML, is that they fail to keep separate feature representations for each resolution in a label tree. We thus propose CascadeXML, an end-to-end multi-resolution learning pipeline, which can harness the multi-layered architecture of a transformer model for attending to different label resolutions with separate feature representations. CascadeXML significantly outperforms all existing approaches with non-trivial gains obtained on benchmark datasets consisting of up to three million labels. Code for CascadeXML will be made publicly available at https://github.com/xmc-aalto/cascadexml.",

author = "Siddhant Kharbanda and Atmadeep Banerjee and Erik Schultheis and Rohit Babbar",

note = "Funding Information: The authors would like to thank Devaansh Gupta and Mohammadreza Qaraei for useful discussions. They also acknowledge the support of CSC – IT Center for Science, Finland, as well as the Aalto Science-IT project, for providing the required computational resources. This research is supported in part by Academy of Finland grants : Decision No. 348215 and 347707. Publisher Copyright: {\textcopyright} 2022 Neural information processing systems foundation. All rights reserved.; 36th Conference on Neural Information Processing Systems, NeurIPS 2022 ; Conference date: 28-11-2022 Through 09-12-2022",

year = "2022",

month = dec,

day = "9",

language = "English",

series = "Advances in Neural Information Processing Systems",

publisher = "Neural Information Processing Systems Foundation, Inc.",

editor = "S. Koyejo and S. Mohamed and A. Agarwal and D. Belgrave and K. Cho and A. Oh",

booktitle = "Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022",

}

TY - GEN

T1 - CascadeXML

T2 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022

AU - Kharbanda, Siddhant

AU - Banerjee, Atmadeep

AU - Schultheis, Erik

AU - Babbar, Rohit

N1 - Funding Information: The authors would like to thank Devaansh Gupta and Mohammadreza Qaraei for useful discussions. They also acknowledge the support of CSC – IT Center for Science, Finland, as well as the Aalto Science-IT project, for providing the required computational resources. This research is supported in part by Academy of Finland grants : Decision No. 348215 and 347707. Publisher Copyright: © 2022 Neural information processing systems foundation. All rights reserved.

PY - 2022/12/9

Y1 - 2022/12/9

N2 - Extreme Multi-label Text Classification (XMC) involves learning a classifier that can assign an input with a subset of most relevant labels from millions of label choices. Recent approaches, such as XR-Transformer and LightXML, leverage a transformer instance to achieve state-of-the-art performance. However, in this process, these approaches need to make various trade-offs between performance and computational requirements. A major shortcoming, as compared to the Bi-LSTM based AttentionXML, is that they fail to keep separate feature representations for each resolution in a label tree. We thus propose CascadeXML, an end-to-end multi-resolution learning pipeline, which can harness the multi-layered architecture of a transformer model for attending to different label resolutions with separate feature representations. CascadeXML significantly outperforms all existing approaches with non-trivial gains obtained on benchmark datasets consisting of up to three million labels. Code for CascadeXML will be made publicly available at https://github.com/xmc-aalto/cascadexml.

AB - Extreme Multi-label Text Classification (XMC) involves learning a classifier that can assign an input with a subset of most relevant labels from millions of label choices. Recent approaches, such as XR-Transformer and LightXML, leverage a transformer instance to achieve state-of-the-art performance. However, in this process, these approaches need to make various trade-offs between performance and computational requirements. A major shortcoming, as compared to the Bi-LSTM based AttentionXML, is that they fail to keep separate feature representations for each resolution in a label tree. We thus propose CascadeXML, an end-to-end multi-resolution learning pipeline, which can harness the multi-layered architecture of a transformer model for attending to different label resolutions with separate feature representations. CascadeXML significantly outperforms all existing approaches with non-trivial gains obtained on benchmark datasets consisting of up to three million labels. Code for CascadeXML will be made publicly available at https://github.com/xmc-aalto/cascadexml.

UR - http://www.scopus.com/inward/record.url?scp=85163200364&partnerID=8YFLogxK

M3 - Chapter in a published conference proceeding

AN - SCOPUS:85163200364

T3 - Advances in Neural Information Processing Systems

BT - Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022

A2 - Koyejo, S.

A2 - Mohamed, S.

A2 - Agarwal, A.

A2 - Belgrave, D.

A2 - Cho, K.

A2 - Oh, A.

PB - Neural Information Processing Systems Foundation, Inc.

Y2 - 28 November 2022 through 9 December 2022

ER -

CascadeXML: Rethinking Transformers for End-to-end Multi-resolution Training in Extreme Multi-label Classification

Abstract

Publication series

Conference

Bibliographical note

Funding

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this