A Stein variational Newton method

Gianluca Detommaso; Tiangang Cui; Youssef Marzouk; Alessio Spantini; Robert Scheichl

A Stein variational Newton method

Gianluca Detommaso, Tiangang Cui, Youssef Marzouk, Alessio Spantini, Robert Scheichl

Research output: Chapter or section in a book/report/conference proceeding › Chapter in a published conference proceeding

Abstract

Stein variational gradient descent (SVGD) was recently proposed as a general purpose nonparametric variational inference algorithm [Liu & Wang, NIPS 2016]: it minimizes the Kullback-Leibler divergence between the target distribution and its approximation by implementing a form of functional gradient descent on a reproducing kernel Hilbert space. In this paper, we accelerate and generalize the SVGD algorithm by including second-order information, thereby approximating a Newton-like iteration in function space. We also show how second-order information can lead to more effective choices of kernel. We observe significant computational gains over the original SVGD algorithm in multiple test cases.

Original language	English
Title of host publication	Advances in Neural Information Processing Systems (NIPS) 2018
Publication status	Acceptance date - 8 Jun 2018

Bibliographical note

18 pages, 7 figures

Keywords

stat.ML
cs.LG
cs.NA

Cite this

@inproceedings{caca4d46dbe64e8aa60125b747d38eb6,

title = "A Stein variational Newton method",

abstract = "Stein variational gradient descent (SVGD) was recently proposed as a general purpose nonparametric variational inference algorithm [Liu & Wang, NIPS 2016]: it minimizes the Kullback-Leibler divergence between the target distribution and its approximation by implementing a form of functional gradient descent on a reproducing kernel Hilbert space. In this paper, we accelerate and generalize the SVGD algorithm by including second-order information, thereby approximating a Newton-like iteration in function space. We also show how second-order information can lead to more effective choices of kernel. We observe significant computational gains over the original SVGD algorithm in multiple test cases.",

keywords = "stat.ML, cs.LG, cs.NA",

author = "Gianluca Detommaso and Tiangang Cui and Youssef Marzouk and Alessio Spantini and Robert Scheichl",

note = "18 pages, 7 figures",

year = "2018",

month = jun,

day = "8",

language = "English",

booktitle = "Advances in Neural Information Processing Systems (NIPS) 2018",

}

TY - GEN

T1 - A Stein variational Newton method

AU - Detommaso, Gianluca

AU - Cui, Tiangang

AU - Marzouk, Youssef

AU - Spantini, Alessio

AU - Scheichl, Robert

N1 - 18 pages, 7 figures

PY - 2018/6/8

Y1 - 2018/6/8

N2 - Stein variational gradient descent (SVGD) was recently proposed as a general purpose nonparametric variational inference algorithm [Liu & Wang, NIPS 2016]: it minimizes the Kullback-Leibler divergence between the target distribution and its approximation by implementing a form of functional gradient descent on a reproducing kernel Hilbert space. In this paper, we accelerate and generalize the SVGD algorithm by including second-order information, thereby approximating a Newton-like iteration in function space. We also show how second-order information can lead to more effective choices of kernel. We observe significant computational gains over the original SVGD algorithm in multiple test cases.

AB - Stein variational gradient descent (SVGD) was recently proposed as a general purpose nonparametric variational inference algorithm [Liu & Wang, NIPS 2016]: it minimizes the Kullback-Leibler divergence between the target distribution and its approximation by implementing a form of functional gradient descent on a reproducing kernel Hilbert space. In this paper, we accelerate and generalize the SVGD algorithm by including second-order information, thereby approximating a Newton-like iteration in function space. We also show how second-order information can lead to more effective choices of kernel. We observe significant computational gains over the original SVGD algorithm in multiple test cases.

KW - stat.ML

KW - cs.LG

KW - cs.NA

UR - https://papers.nips.cc

M3 - Chapter in a published conference proceeding

BT - Advances in Neural Information Processing Systems (NIPS) 2018

ER -

A Stein variational Newton method

Abstract

Bibliographical note

Keywords

Other files and links

Fingerprint

Cite this