TY - JOUR

T1 - Echo State Networks Trained by Tikhonov Least Squares are L2 Approximators of Ergodic Dynamical Systems

AU - Hart, Allen

AU - Hook, James

AU - Dawes, Jonathan

PY - 2021/3/2

Y1 - 2021/3/2

N2 - Echo State Networks (ESNs) are a class of single-layer recurrent neural networks with randomly generated internal weights, and a single layer of tuneable outer weights, which are usually trained by regularised linear least squares regression. Remarkably, ESNs still enjoy the universal approximation property despite the training procedure being entirely linear. In this paper, we prove that an ESN trained on a sequence of observations from an ergodic dynamical system (with invariant measure ) using Tikhonov least squares regression against a set of targets, will approximate the target function in the norm. In the special case that the targets are future observations, the ESN is learning the next step map, which allows time series forecasting. We demonstrate the theory numerically by training an ESN using Tikhonov least squares on a sequence of scalar observations of the Lorenz system.

AB - Echo State Networks (ESNs) are a class of single-layer recurrent neural networks with randomly generated internal weights, and a single layer of tuneable outer weights, which are usually trained by regularised linear least squares regression. Remarkably, ESNs still enjoy the universal approximation property despite the training procedure being entirely linear. In this paper, we prove that an ESN trained on a sequence of observations from an ergodic dynamical system (with invariant measure ) using Tikhonov least squares regression against a set of targets, will approximate the target function in the norm. In the special case that the targets are future observations, the ESN is learning the next step map, which allows time series forecasting. We demonstrate the theory numerically by training an ESN using Tikhonov least squares on a sequence of scalar observations of the Lorenz system.

U2 - 10.1016/j.physd.2021.132882

DO - 10.1016/j.physd.2021.132882

M3 - Article

JO - Physica D: Nonlinear Phenomena

JF - Physica D: Nonlinear Phenomena

SN - 0167-2789

M1 - 132882

ER -