Sparse Bayesian Learning and the Relevance Vector Machine

Research output: Contribution to journalArticle

  • 3230 Citations

Abstract

This paper introduces a general Bayesian framework for obtaining sparse solutions to regression and classification tasks utilising models linear in the parameters. Although this framework is fully general, we illustrate our approach with a particular specialisation that we denote the `relevance vector machine' (RVM), a model of identical functional form to the popular and state-of-the-art `support vector machine' (SVM). We demonstrate that by exploiting a probabilistic Bayesian learning framework, we can derive accurate prediction models which typically utilise dramatically fewer basis functions than a comparable SVM while offering a number of additional advantages. These include the benefits of probabilistic predictions, automatic estimation of `nuisance' parameters, and the facility to utilise arbitrary basis functions (e.g. non-`Mercer' kernels). We detail the Bayesian framework and associated learning algorithm for the RVM, and give some illustrative examples of its application along with some comparative benchmarks. We offer some explanation for the exceptional degree of sparsity obtained, and discuss and demonstrate some of the advantageous features, and potential extensions, of Bayesian relevance learning.
Original languageEnglish
Pages (from-to)211-244
Number of pages34
JournalJournal of Machine Learning Research
Volume1
StatePublished - Jun 2001

Fingerprint

Support vector machines
Learning algorithms

Cite this

Sparse Bayesian Learning and the Relevance Vector Machine. / Tipping, Michael E.

In: Journal of Machine Learning Research, Vol. 1, 06.2001, p. 211-244.

Research output: Contribution to journalArticle

Tipping, Michael E / Sparse Bayesian Learning and the Relevance Vector Machine.

In: Journal of Machine Learning Research, Vol. 1, 06.2001, p. 211-244.

Research output: Contribution to journalArticle

@article{14d945a420e14314ab581477f5a5ec4e,
title = "Sparse Bayesian Learning and the Relevance Vector Machine",
abstract = "This paper introduces a general Bayesian framework for obtaining sparse solutions to regression and classification tasks utilising models linear in the parameters. Although this framework is fully general, we illustrate our approach with a particular specialisation that we denote the `relevance vector machine' (RVM), a model of identical functional form to the popular and state-of-the-art `support vector machine' (SVM). We demonstrate that by exploiting a probabilistic Bayesian learning framework, we can derive accurate prediction models which typically utilise dramatically fewer basis functions than a comparable SVM while offering a number of additional advantages. These include the benefits of probabilistic predictions, automatic estimation of `nuisance' parameters, and the facility to utilise arbitrary basis functions (e.g. non-`Mercer' kernels). We detail the Bayesian framework and associated learning algorithm for the RVM, and give some illustrative examples of its application along with some comparative benchmarks. We offer some explanation for the exceptional degree of sparsity obtained, and discuss and demonstrate some of the advantageous features, and potential extensions, of Bayesian relevance learning.",
author = "Tipping, {Michael E}",
year = "2001",
month = "6",
volume = "1",
pages = "211--244",
journal = "Journal of Machine Learning Research",
issn = "1532-4435",
publisher = "Journal of Machine Learning Research",

}

TY - JOUR

T1 - Sparse Bayesian Learning and the Relevance Vector Machine

AU - Tipping,Michael E

PY - 2001/6

Y1 - 2001/6

N2 - This paper introduces a general Bayesian framework for obtaining sparse solutions to regression and classification tasks utilising models linear in the parameters. Although this framework is fully general, we illustrate our approach with a particular specialisation that we denote the `relevance vector machine' (RVM), a model of identical functional form to the popular and state-of-the-art `support vector machine' (SVM). We demonstrate that by exploiting a probabilistic Bayesian learning framework, we can derive accurate prediction models which typically utilise dramatically fewer basis functions than a comparable SVM while offering a number of additional advantages. These include the benefits of probabilistic predictions, automatic estimation of `nuisance' parameters, and the facility to utilise arbitrary basis functions (e.g. non-`Mercer' kernels). We detail the Bayesian framework and associated learning algorithm for the RVM, and give some illustrative examples of its application along with some comparative benchmarks. We offer some explanation for the exceptional degree of sparsity obtained, and discuss and demonstrate some of the advantageous features, and potential extensions, of Bayesian relevance learning.

AB - This paper introduces a general Bayesian framework for obtaining sparse solutions to regression and classification tasks utilising models linear in the parameters. Although this framework is fully general, we illustrate our approach with a particular specialisation that we denote the `relevance vector machine' (RVM), a model of identical functional form to the popular and state-of-the-art `support vector machine' (SVM). We demonstrate that by exploiting a probabilistic Bayesian learning framework, we can derive accurate prediction models which typically utilise dramatically fewer basis functions than a comparable SVM while offering a number of additional advantages. These include the benefits of probabilistic predictions, automatic estimation of `nuisance' parameters, and the facility to utilise arbitrary basis functions (e.g. non-`Mercer' kernels). We detail the Bayesian framework and associated learning algorithm for the RVM, and give some illustrative examples of its application along with some comparative benchmarks. We offer some explanation for the exceptional degree of sparsity obtained, and discuss and demonstrate some of the advantageous features, and potential extensions, of Bayesian relevance learning.

UR - http://www.jmlr.org/papers/volume1/tipping01a/tipping01a.pdf

UR - http://www.jmlr.org/papers/volume1/tipping01a/tipping01a.pdf

M3 - Article

VL - 1

SP - 211

EP - 244

JO - Journal of Machine Learning Research

T2 - Journal of Machine Learning Research

JF - Journal of Machine Learning Research

SN - 1532-4435

ER -