ONLINE INFERENCE WITH MULTI-MODAL LIKELIHOOD FUNCTIONS

Let (Yt)t≥1 be a sequence of i.i.d. observations and {fθ ,θ ∈ Rd }
be a parametric model. We introduce a new online algorithm for computing a sequence (θˆ
t)t≥1, which is shown to converge almost surely to
argmaxθ∈Rd E[log fθ (Y1)] at rate O(log(t)(1+ε)/2t−1/2), with ε > 0 a user
specified parameter. This convergence result is obtained under standard conditions on the statistical model and, most notably, we allow the mapping
θ → E[log fθ (Y1)] to be multi-modal. However, the computational cost to
process each observation grows exponentially with the dimension of θ, which
makes the proposed approach applicable to low or moderate dimensional
problems only. We also derive a version of the estimator θˆ
t , which is well
suited to student-t linear regression models that are popular tools for robust
linear regression. As shown by experiments on simulated and real data, the
corresponding estimator of the regression coefficients is, as expected, robust
to the presence of outliers and thus, as a by-product, we obtain a new adaptive
and robust online estimation procedure for linear regression models.