TQCompressor: Improving Tensor Decomposition Methods in Neural Networks Via Permutations

Vadim Abronin; Aleksei Naumov; Denis Mazur; Dmitriy Bystrov; Katerina Tsarova; Artem Melnikov; Sergey Dolgov; Reuben Brasher; Michael Perelshein

doi:10.1109/MIPR62202.2024.00086

TQCompressor: Improving Tensor Decomposition Methods in Neural Networks Via Permutations

Vadim Abronin, Aleksei Naumov, Denis Mazur, Dmitriy Bystrov, Katerina Tsarova, Artem Melnikov, Sergey Dolgov, Reuben Brasher, Michael Perelshein

Terra Quantum AG

Research output: Chapter or section in a book/report/conference proceeding › Chapter in a published conference proceeding

Abstract

We introduce TQCompressor a neural network model compression method using enhanced tensor decompositions. We propose a permutation-based improvement to Kronecker decomposition, reducing the loss in model expressivity typically associated with compression. Applied to GPT-2_small, this results in the TQCompressedGPT-2 model with 81 million parameters, down from 124 million. Enhanced through multi-step knowledge distillation on 3.1% of OpenWebText, TQCompressedGPT-2 outperforms DistilGPT-2 and KnGPT-2. We made TQCompressedGPT-2 publicly available.

Original language	English
Title of host publication	Proceedings - 2024 IEEE 7th International Conference on Multimedia Information Processing and Retrieval, MIPR 2024
Place of Publication	U. S. A.
Publisher	IEEE
Pages	503-506
Number of pages	4
ISBN (Electronic)	9798350351422
ISBN (Print)	9798350351439
DOIs	https://doi.org/10.1109/MIPR62202.2024.00086
Publication status	Published - 15 Oct 2024
Event	7th IEEE International Conference on Multimedia Information Processing and Retrieval, MIPR 2024 - San Jose, USA United States Duration: 7 Aug 2024 → 9 Aug 2024

Conference

Conference	7th IEEE International Conference on Multimedia Information Processing and Retrieval, MIPR 2024
Country/Territory	USA United States
City	San Jose
Period	7/08/24 → 9/08/24

Keywords

GPT-2
Knowledge distillation
Kronecker decomposition
Neural network compression
Tensor decomposition

ASJC Scopus subject areas

Artificial Intelligence
Computer Science Applications
Computer Vision and Pattern Recognition
Information Systems
Media Technology

Access to Document

10.1109/MIPR62202.2024.00086

Cite this

Abronin, V., Naumov, A., Mazur, D., Bystrov, D., Tsarova, K., Melnikov, A., Dolgov, S., Brasher, R., & Perelshein, M. (2024). TQCompressor: Improving Tensor Decomposition Methods in Neural Networks Via Permutations. In Proceedings - 2024 IEEE 7th International Conference on Multimedia Information Processing and Retrieval, MIPR 2024 (pp. 503-506). IEEE. https://doi.org/10.1109/MIPR62202.2024.00086

TQCompressor: Improving Tensor Decomposition Methods in Neural Networks Via Permutations. / Abronin, Vadim; Naumov, Aleksei; Mazur, Denis et al.
Proceedings - 2024 IEEE 7th International Conference on Multimedia Information Processing and Retrieval, MIPR 2024. U. S. A.: IEEE, 2024. p. 503-506.

Research output: Chapter or section in a book/report/conference proceeding › Chapter in a published conference proceeding

Abronin, V, Naumov, A, Mazur, D, Bystrov, D, Tsarova, K, Melnikov, A, Dolgov, S, Brasher, R & Perelshein, M 2024, TQCompressor: Improving Tensor Decomposition Methods in Neural Networks Via Permutations. in Proceedings - 2024 IEEE 7th International Conference on Multimedia Information Processing and Retrieval, MIPR 2024. IEEE, U. S. A., pp. 503-506, 7th IEEE International Conference on Multimedia Information Processing and Retrieval, MIPR 2024, San Jose, USA United States, 7/08/24. https://doi.org/10.1109/MIPR62202.2024.00086

@inproceedings{21cad3c4dc8949578695ecb49f636edf,

title = "TQCompressor: Improving Tensor Decomposition Methods in Neural Networks Via Permutations",

abstract = "We introduce TQCompressor a neural network model compression method using enhanced tensor decompositions. We propose a permutation-based improvement to Kronecker decomposition, reducing the loss in model expressivity typically associated with compression. Applied to GPT-2small, this results in the TQCompressedGPT-2 model with 81 million parameters, down from 124 million. Enhanced through multi-step knowledge distillation on 3.1% of OpenWebText, TQCompressedGPT-2 outperforms DistilGPT-2 and KnGPT-2. We made TQCompressedGPT-2 publicly available.",

keywords = "GPT-2, Knowledge distillation, Kronecker decomposition, Neural network compression, Tensor decomposition",

author = "Vadim Abronin and Aleksei Naumov and Denis Mazur and Dmitriy Bystrov and Katerina Tsarova and Artem Melnikov and Sergey Dolgov and Reuben Brasher and Michael Perelshein",

year = "2024",

month = oct,

day = "15",

doi = "10.1109/MIPR62202.2024.00086",

language = "English",

isbn = "9798350351439",

pages = "503--506",

booktitle = "Proceedings - 2024 IEEE 7th International Conference on Multimedia Information Processing and Retrieval, MIPR 2024",

publisher = "IEEE",

address = "USA United States",

note = "7th IEEE International Conference on Multimedia Information Processing and Retrieval, MIPR 2024 ; Conference date: 07-08-2024 Through 09-08-2024",

}

TY - GEN

T1 - TQCompressor

T2 - 7th IEEE International Conference on Multimedia Information Processing and Retrieval, MIPR 2024

AU - Abronin, Vadim

AU - Naumov, Aleksei

AU - Mazur, Denis

AU - Bystrov, Dmitriy

AU - Tsarova, Katerina

AU - Melnikov, Artem

AU - Dolgov, Sergey

AU - Brasher, Reuben

AU - Perelshein, Michael

PY - 2024/10/15

Y1 - 2024/10/15

N2 - We introduce TQCompressor a neural network model compression method using enhanced tensor decompositions. We propose a permutation-based improvement to Kronecker decomposition, reducing the loss in model expressivity typically associated with compression. Applied to GPT-2small, this results in the TQCompressedGPT-2 model with 81 million parameters, down from 124 million. Enhanced through multi-step knowledge distillation on 3.1% of OpenWebText, TQCompressedGPT-2 outperforms DistilGPT-2 and KnGPT-2. We made TQCompressedGPT-2 publicly available.

AB - We introduce TQCompressor a neural network model compression method using enhanced tensor decompositions. We propose a permutation-based improvement to Kronecker decomposition, reducing the loss in model expressivity typically associated with compression. Applied to GPT-2small, this results in the TQCompressedGPT-2 model with 81 million parameters, down from 124 million. Enhanced through multi-step knowledge distillation on 3.1% of OpenWebText, TQCompressedGPT-2 outperforms DistilGPT-2 and KnGPT-2. We made TQCompressedGPT-2 publicly available.

KW - GPT-2

KW - Knowledge distillation

KW - Kronecker decomposition

KW - Neural network compression

KW - Tensor decomposition

UR - http://www.scopus.com/inward/record.url?scp=85210090773&partnerID=8YFLogxK

U2 - 10.1109/MIPR62202.2024.00086

DO - 10.1109/MIPR62202.2024.00086

M3 - Chapter in a published conference proceeding

AN - SCOPUS:85210090773

SN - 9798350351439

SP - 503

EP - 506

BT - Proceedings - 2024 IEEE 7th International Conference on Multimedia Information Processing and Retrieval, MIPR 2024

PB - IEEE

CY - U. S. A.

Y2 - 7 August 2024 through 9 August 2024

ER -

TQCompressor: Improving Tensor Decomposition Methods in Neural Networks Via Permutations

Abstract

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this