Deeply learned spectral total variation decomposition

Tamara G. Grossmann, Yury Korolev, Guy Gilboa, Carola Bibiane Schönlieb

Research output: Contribution to journalConference articlepeer-review

2 Citations (SciVal)

Abstract

Non-linear spectral decompositions of images based on one-homogeneous functionals such as total variation have gained considerable attention in the last few years. Due to their ability to extract spectral components corresponding to objects of different size and contrast, such decompositions enable filtering, feature transfer, image fusion and other applications. However, obtaining this decomposition involves solving multiple non-smooth optimisation problems and is therefore computationally highly intensive. In this paper, we present a neural network approximation of a non-linear spectral decomposition. We report up to four orders of magnitude (×10, 000) speedup in processing of mega-pixel size images, compared to classical GPU implementations. Our proposed network, TVSpecNET, is able to implicitly learn the underlying PDE and, despite being entirely data driven, inherits invariances of the model based transform. To the best of our knowledge, this is the first approach towards learning a non-linear spectral decomposition of images. Not only do we gain a staggering computational advantage, but this approach can also be seen as a step towards studying neural networks that can decompose an image into spectral components defined by a user rather than a handcrafted functional.

Original languageEnglish
JournalAdvances in Neural Information Processing Systems
Volume2020-December
Publication statusPublished - 2020
Event34th Conference on Neural Information Processing Systems, NeurIPS 2020 - Virtual, Online
Duration: 6 Dec 202012 Dec 2020

Bibliographical note

Funding Information:
This work was supported by the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 777826 (NoMADS). TG, YK and CBS acknowledge the support of the Cantab Capital Institute for the Mathematics of Information. TG additionally acknowledges support from the EPSRC National Productivity and Investment Fund grant Nr. EP/S515334/1 reference 2089694. YK acknowledges the support of the Royal Society (Newton International Fellowship NF170045 Quantifying Uncertainty in Model-Based Data Inference Using Partial Order) and the National Physical Laboratory. GG acknowledges support by the Israel Science Foundation (Grant No. 534/19) and by the Ollendorff Minerva Center. CBS acknowledges support from the Leverhulme Trust project on Breaking the non-convexity barrier, the Philip Leverhulme Prize, the EPSRC grants EP/S026045/1 and EP/T003553/1, the EPSRC Centre Nr. EP/N014588/1, European Union Horizon 2020 research and innovation programmes under the Marie Skłodowska-Curie grant agreement No. 777826 NoMADS and No. 691070 CHiPS, and the Alan Turing Institute.

Funding Information:
This work was supported by the European Union?s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No. 777826 (NoMADS). TG, YK and CBS acknowledge the support of the Cantab Capital Institute for the Mathematics of Information. TG additionally acknowledges support from the EPSRC National Productivity and Investment Fund grant Nr. EP/S515334/1 reference 2089694. YK acknowledges the support of the Royal Society (Newton International Fellowship NF170045 Quantifying Uncertainty in Model-Based Data Inference Using Partial Order) and the National Physical Laboratory. GG acknowledges support by the Israel Science Foundation (Grant No. 534/19) and by the Ollendorff Minerva Center. CBS acknowledges support from the Leverhulme Trust project on Breaking the non-convexity barrier, the Philip Leverhulme Prize, the EPSRC grants EP/S026045/1 and EP/T003553/1, the EPSRC Centre Nr. EP/N014588/1, European Union Horizon 2020 research and innovation programmes under the Marie Sklodowska-Curie grant agreement No. 777826 NoMADS and No. 691070 CHiPS, and the Alan Turing Institute.

Funding Information:
We also acknowledge the support of NVIDIA Corporation with the donation of two Quadro P6000, a Tesla K40c and a Titan Xp GPU used for this research.

Publisher Copyright:
© 2020 Neural information processing systems foundation. All rights reserved.

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems
  • Signal Processing

Fingerprint

Dive into the research topics of 'Deeply learned spectral total variation decomposition'. Together they form a unique fingerprint.

Cite this