Abstract
Echo State Networks (ESNs) are a class of singlelayer recurrent neural networks that have enjoyed recent attention. In this paper we prove that a suitable ESN, trained on a series of measurements of an invertible dynamical system, induces a C^{1} map from the dynamical system's phase space to the ESN's reservoir space. We call this the Echo State Map. We then prove that the Echo State Map is generically an embedding with positive probability. Under additional mild assumptions, we further conjecture that the Echo State Map is almost surely an embedding. For sufficiently large, and specially structured, but still randomly generated ESNs, we prove that there exists a linear readout layer that allows the ESN to predict the next observation of a dynamical system arbitrarily well. Consequently, if the dynamical system under observation is structurally stable then the trained ESN will exhibit dynamics that are topologically conjugate to the future behaviour of the observed dynamical system. Our theoretical results connect the theory of ESNs to the delayembedding literature for dynamical systems, and are supported by numerical evidence from simulations of the traditional Lorenz equations. The simulations confirm that, from a one dimensional observation function, an ESN can accurately infer a range of geometric and topological features of the dynamics such as the eigenvalues of equilibrium points, Lyapunov exponents and homology groups.
Original language  English 

Pages (fromto)  234247 
Number of pages  14 
Journal  Neural Networks 
Volume  128 
Early online date  16 May 2020 
DOIs  
Publication status  Published  1 Aug 2020 
Keywords
 Delay embedding
 Dynamical system
 Lorenz equations
 Persistent homology
 Recurrent neural networks
 Reservoir computing
ASJC Scopus subject areas
 Cognitive Neuroscience
 Artificial Intelligence
 Applied Mathematics
Fingerprint Dive into the research topics of 'Embedding and approximation theorems for echo state networks'. Together they form a unique fingerprint.
Profiles

Jonathan Dawes
 Department of Mathematical Sciences  Professor
 Centre for Networks and Collective Behaviour
 EPSRC Centre for Doctoral Training in Statistical Applied Mathematics (SAMBa)
 Water Innovation and Research Centre (WIRC)
 Institute for Mathematical Innovation (IMI)
 Institute for Policy Research (IPR)
 Water Informatics: Science and Engineering Centre for Doctoral Training (WISE)
 Centre for Mathematical Biology
Person: Research & Teaching