Deep learning as optimal control problems

models and numerical methods

Martin Benning, Elena Celledoni, Matthias J. Ehrhardt, Brynjulf Owren, Carola-Bibiane Schönlieb

Research output: Working paper

24 Downloads (Pure)

Abstract

We consider recent work of Haber and Ruthotto 2017 and Chang et al. 2018, where deep learning neural networks have been interpreted as discretisations of an optimal control problem subject to an ordinary differential equation constraint. We review the first order conditions for optimality, and the conditions ensuring optimality after discretization. This leads to a class of algorithms for solving the discrete optimal control problem which guarantee that the corresponding discrete necessary conditions for optimality are fulfilled. We discuss two different deep learning algorithms and make a preliminary analysis of the ability of the algorithms to generalise.
Original languageEnglish
PublisherarXiv
Publication statusPublished - 11 Apr 2019

Keywords

  • math.OC
  • cs.LG
  • math.NA

Cite this

Benning, M., Celledoni, E., Ehrhardt, M. J., Owren, B., & Schönlieb, C-B. (2019). Deep learning as optimal control problems: models and numerical methods. arXiv.

Deep learning as optimal control problems : models and numerical methods. / Benning, Martin; Celledoni, Elena; Ehrhardt, Matthias J.; Owren, Brynjulf; Schönlieb, Carola-Bibiane.

arXiv, 2019.

Research output: Working paper

Benning, M, Celledoni, E, Ehrhardt, MJ, Owren, B & Schönlieb, C-B 2019 'Deep learning as optimal control problems: models and numerical methods' arXiv.
Benning M, Celledoni E, Ehrhardt MJ, Owren B, Schönlieb C-B. Deep learning as optimal control problems: models and numerical methods. arXiv. 2019 Apr 11.
Benning, Martin ; Celledoni, Elena ; Ehrhardt, Matthias J. ; Owren, Brynjulf ; Schönlieb, Carola-Bibiane. / Deep learning as optimal control problems : models and numerical methods. arXiv, 2019.
@techreport{8c5672172afc4be8b9c3bee16bdbdfa2,
title = "Deep learning as optimal control problems: models and numerical methods",
abstract = "We consider recent work of Haber and Ruthotto 2017 and Chang et al. 2018, where deep learning neural networks have been interpreted as discretisations of an optimal control problem subject to an ordinary differential equation constraint. We review the first order conditions for optimality, and the conditions ensuring optimality after discretization. This leads to a class of algorithms for solving the discrete optimal control problem which guarantee that the corresponding discrete necessary conditions for optimality are fulfilled. We discuss two different deep learning algorithms and make a preliminary analysis of the ability of the algorithms to generalise.",
keywords = "math.OC, cs.LG, math.NA",
author = "Martin Benning and Elena Celledoni and Ehrhardt, {Matthias J.} and Brynjulf Owren and Carola-Bibiane Sch{\"o}nlieb",
year = "2019",
month = "4",
day = "11",
language = "English",
publisher = "arXiv",
type = "WorkingPaper",
institution = "arXiv",

}

TY - UNPB

T1 - Deep learning as optimal control problems

T2 - models and numerical methods

AU - Benning, Martin

AU - Celledoni, Elena

AU - Ehrhardt, Matthias J.

AU - Owren, Brynjulf

AU - Schönlieb, Carola-Bibiane

PY - 2019/4/11

Y1 - 2019/4/11

N2 - We consider recent work of Haber and Ruthotto 2017 and Chang et al. 2018, where deep learning neural networks have been interpreted as discretisations of an optimal control problem subject to an ordinary differential equation constraint. We review the first order conditions for optimality, and the conditions ensuring optimality after discretization. This leads to a class of algorithms for solving the discrete optimal control problem which guarantee that the corresponding discrete necessary conditions for optimality are fulfilled. We discuss two different deep learning algorithms and make a preliminary analysis of the ability of the algorithms to generalise.

AB - We consider recent work of Haber and Ruthotto 2017 and Chang et al. 2018, where deep learning neural networks have been interpreted as discretisations of an optimal control problem subject to an ordinary differential equation constraint. We review the first order conditions for optimality, and the conditions ensuring optimality after discretization. This leads to a class of algorithms for solving the discrete optimal control problem which guarantee that the corresponding discrete necessary conditions for optimality are fulfilled. We discuss two different deep learning algorithms and make a preliminary analysis of the ability of the algorithms to generalise.

KW - math.OC

KW - cs.LG

KW - math.NA

M3 - Working paper

BT - Deep learning as optimal control problems

PB - arXiv

ER -