Improving Neural Ordinary Differential Equations with Nesterov's Accelerated Gradient Method

Nghia H. Nguyen; Tan M. Nguyen; Huyen K. Vo; Stanley J. Osher; Thieu N. Vo

Improving Neural Ordinary Differential Equations with Nesterov's Accelerated Gradient Method

Nghia H. Nguyen, Tan M. Nguyen, Huyen K. Vo, Stanley J. Osher, Thieu N. Vo

Department of Computer Science

Research output: Chapter or section in a book/report/conference proceeding › Chapter in a published conference proceeding

15 Citations (SciVal)

Abstract

We propose the Nesterov neural ordinary differential equations (NesterovNODEs), whose layers solve the second-order ordinary differential equations (ODEs) limit of Nesterov's accelerated gradient (NAG) method, and a generalization called GNesterovNODEs. Taking the advantage of the convergence rate O(1/k²) of the NAG scheme, GNesterovNODEs speed up training and inference by reducing the number of function evaluations (NFEs) needed to solve the ODEs. We also prove that the adjoint state of a GNesterovNODEs also satisfies a GNesterovNODEs, thus accelerating both forward and backward ODE solvers and allowing the model to be scaled up for large-scale tasks. We empirically corroborate the advantage of GNesterovNODEs on a wide range of practical applications, including point cloud separation, image classification, and sequence modeling. Compared to NODEs, GNesterovNODEs require a significantly smaller number of NFEs while achieving better accuracy across our experiments.

Original language	English
Title of host publication	Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022
Editors	S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, A. Oh
Publisher	Neural information processing systems foundation
Number of pages	15
ISBN (Electronic)	9781713871088
Publication status	Published - 28 Nov 2022
Event	36th Conference on Neural Information Processing Systems, NeurIPS 2022 - New Orleans, USA United States Duration: 28 Nov 2022 → 9 Dec 2022

Publication series

Name	Advances in Neural Information Processing Systems
Volume	35
ISSN (Print)	1049-5258

Conference

Conference	36th Conference on Neural Information Processing Systems, NeurIPS 2022
Country/Territory	USA United States
City	New Orleans
Period	28/11/22 → 9/12/22

Bibliographical note

Publisher Copyright:
© 2022 Neural information processing systems foundation. All rights reserved.

ASJC Scopus subject areas

Signal Processing
Information Systems
Computer Networks and Communications

Access to Document

Cite this

Nguyen, N. H., Nguyen, T. M., Vo, H. K., Osher, S. J., & Vo, T. N. (2022). Improving Neural Ordinary Differential Equations with Nesterov's Accelerated Gradient Method. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, & A. Oh (Eds.), Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022 (Advances in Neural Information Processing Systems; Vol. 35). Neural information processing systems foundation. https://proceedings.neurips.cc/paper_files/paper/2022/hash/32cc61322f1e2f56f989d29ccc7cfbb7-Abstract-Conference.html

Improving Neural Ordinary Differential Equations with Nesterov's Accelerated Gradient Method. / Nguyen, Nghia H.; Nguyen, Tan M.; Vo, Huyen K. et al.
Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022. ed. / S. Koyejo; S. Mohamed; A. Agarwal; D. Belgrave; K. Cho; A. Oh. Neural information processing systems foundation, 2022. (Advances in Neural Information Processing Systems; Vol. 35).

Research output: Chapter or section in a book/report/conference proceeding › Chapter in a published conference proceeding

Nguyen, NH, Nguyen, TM, Vo, HK, Osher, SJ & Vo, TN 2022, Improving Neural Ordinary Differential Equations with Nesterov's Accelerated Gradient Method. in S Koyejo, S Mohamed, A Agarwal, D Belgrave, K Cho & A Oh (eds), Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022. Advances in Neural Information Processing Systems, vol. 35, Neural information processing systems foundation, 36th Conference on Neural Information Processing Systems, NeurIPS 2022, New Orleans, USA United States, 28/11/22. <https://proceedings.neurips.cc/paper_files/paper/2022/hash/32cc61322f1e2f56f989d29ccc7cfbb7-Abstract-Conference.html>

Nguyen NH, Nguyen TM, Vo HK, Osher SJ, Vo TN. Improving Neural Ordinary Differential Equations with Nesterov's Accelerated Gradient Method. In Koyejo S, Mohamed S, Agarwal A, Belgrave D, Cho K, Oh A, editors, Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022. Neural information processing systems foundation. 2022. (Advances in Neural Information Processing Systems). Epub 2022 Nov 28.

Nguyen, Nghia H. ; Nguyen, Tan M. ; Vo, Huyen K. et al. / Improving Neural Ordinary Differential Equations with Nesterov's Accelerated Gradient Method. Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022. editor / S. Koyejo ; S. Mohamed ; A. Agarwal ; D. Belgrave ; K. Cho ; A. Oh. Neural information processing systems foundation, 2022. (Advances in Neural Information Processing Systems).

@inproceedings{fdb722d31400440d847d1f306c70198a,

title = "Improving Neural Ordinary Differential Equations with Nesterov's Accelerated Gradient Method",

abstract = "We propose the Nesterov neural ordinary differential equations (NesterovNODEs), whose layers solve the second-order ordinary differential equations (ODEs) limit of Nesterov's accelerated gradient (NAG) method, and a generalization called GNesterovNODEs. Taking the advantage of the convergence rate O(1/k2) of the NAG scheme, GNesterovNODEs speed up training and inference by reducing the number of function evaluations (NFEs) needed to solve the ODEs. We also prove that the adjoint state of a GNesterovNODEs also satisfies a GNesterovNODEs, thus accelerating both forward and backward ODE solvers and allowing the model to be scaled up for large-scale tasks. We empirically corroborate the advantage of GNesterovNODEs on a wide range of practical applications, including point cloud separation, image classification, and sequence modeling. Compared to NODEs, GNesterovNODEs require a significantly smaller number of NFEs while achieving better accuracy across our experiments.",

author = "Nguyen, \{Nghia H.\} and Nguyen, \{Tan M.\} and Vo, \{Huyen K.\} and Osher, \{Stanley J.\} and Vo, \{Thieu N.\}",

note = "Publisher Copyright: {\textcopyright} 2022 Neural information processing systems foundation. All rights reserved.; 36th Conference on Neural Information Processing Systems, NeurIPS 2022 ; Conference date: 28-11-2022 Through 09-12-2022",

year = "2022",

month = nov,

day = "28",

language = "English",

series = "Advances in Neural Information Processing Systems",

publisher = "Neural information processing systems foundation",

editor = "S. Koyejo and S. Mohamed and A. Agarwal and D. Belgrave and K. Cho and A. Oh",

booktitle = "Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022",

address = "USA United States",

}

TY - GEN

T1 - Improving Neural Ordinary Differential Equations with Nesterov's Accelerated Gradient Method

AU - Nguyen, Nghia H.

AU - Nguyen, Tan M.

AU - Vo, Huyen K.

AU - Osher, Stanley J.

AU - Vo, Thieu N.

PY - 2022/11/28

Y1 - 2022/11/28

N2 - We propose the Nesterov neural ordinary differential equations (NesterovNODEs), whose layers solve the second-order ordinary differential equations (ODEs) limit of Nesterov's accelerated gradient (NAG) method, and a generalization called GNesterovNODEs. Taking the advantage of the convergence rate O(1/k2) of the NAG scheme, GNesterovNODEs speed up training and inference by reducing the number of function evaluations (NFEs) needed to solve the ODEs. We also prove that the adjoint state of a GNesterovNODEs also satisfies a GNesterovNODEs, thus accelerating both forward and backward ODE solvers and allowing the model to be scaled up for large-scale tasks. We empirically corroborate the advantage of GNesterovNODEs on a wide range of practical applications, including point cloud separation, image classification, and sequence modeling. Compared to NODEs, GNesterovNODEs require a significantly smaller number of NFEs while achieving better accuracy across our experiments.

AB - We propose the Nesterov neural ordinary differential equations (NesterovNODEs), whose layers solve the second-order ordinary differential equations (ODEs) limit of Nesterov's accelerated gradient (NAG) method, and a generalization called GNesterovNODEs. Taking the advantage of the convergence rate O(1/k2) of the NAG scheme, GNesterovNODEs speed up training and inference by reducing the number of function evaluations (NFEs) needed to solve the ODEs. We also prove that the adjoint state of a GNesterovNODEs also satisfies a GNesterovNODEs, thus accelerating both forward and backward ODE solvers and allowing the model to be scaled up for large-scale tasks. We empirically corroborate the advantage of GNesterovNODEs on a wide range of practical applications, including point cloud separation, image classification, and sequence modeling. Compared to NODEs, GNesterovNODEs require a significantly smaller number of NFEs while achieving better accuracy across our experiments.

UR - https://www.scopus.com/pages/publications/85163154376

M3 - Chapter in a published conference proceeding

AN - SCOPUS:85163154376

T3 - Advances in Neural Information Processing Systems

BT - Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022

A2 - Koyejo, S.

A2 - Mohamed, S.

A2 - Agarwal, A.

A2 - Belgrave, D.

A2 - Cho, K.

A2 - Oh, A.

PB - Neural information processing systems foundation

T2 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022

Y2 - 28 November 2022 through 9 December 2022

ER -

Improving Neural Ordinary Differential Equations with Nesterov's Accelerated Gradient Method

Abstract

Publication series

Conference

Bibliographical note

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this