Optimising Sensor Path Planning with Reinforcement Learning and Passive Sonar Modelling

Edward Clark, Alan Hunter, Olga Isupova, Marcus Donnelly

Research output: Chapter or section in a book/report/conference proceedingChapter or section

Abstract

This paper presents a solution for optimising sensor path planning in marine sensor management using reinforcement learning (RL). RL is a type of machine learning where an intelligent system, known as an agent, learns how to make effective decisions by interacting with its environment. Acoustic propagation modelling is integrated into the RL framework using a standard python RL library gym and using algorithms from the stablebaselines3 library. The observation space encompasses signal-to-noise (SNR) information, platform position, bathymetry, and sound speed data. The action space is discretised into 16 horizontal directions and 3 vertical levels, resulting in a 49-dimensional action space. The reward function combines penalisation for movement and rewards for navigating to high SNR regions. SNR is calculated using PyRAM, a Python implementation of the RAM parabolic equation solution to the Helmholtz Equation. The RL agent uses proximal policy optimization to learn the management policy. The learnt policy is compared against a gradient ascent policy and an ‘oracle’ policy which can use perfect knowledge of the source location for direct navigation. The learning process converges to a stable median between 2.5 and 3 million learning steps. The results demonstrate that the learnt policy closely matches the ‘oracle’ policy in both reward distribution and behaviour. It also outperforms the gradient ascent policy in a realistic environment.

Original languageEnglish
Title of host publicationProceedings of the 7th Underwater Acoustics Conference and Exhibition, UACE 2023
EditorsM. Taroudakis
PublisherI.A.C.M, Foundation for Research and Technology - Hellas
Pages411-418
Number of pages8
Publication statusPublished - 30 Jun 2023
Event7th Underwater Acoustics Conference and Exhibition, UACE 2023 - Kalamata, Greece
Duration: 25 Jun 202330 Jun 2023

Publication series

NameUnderwater Acoustic Conference and Exhibition Series
ISSN (Print)2408-0915

Conference

Conference7th Underwater Acoustics Conference and Exhibition, UACE 2023
Country/TerritoryGreece
CityKalamata
Period25/06/2330/06/23

Keywords

  • Acoustic Propagation Modelling
  • Passive Sonar
  • Reinforcement Learning
  • Sensor Management

ASJC Scopus subject areas

  • Geophysics
  • Oceanography
  • Environmental Engineering
  • Acoustics and Ultrasonics

Fingerprint

Dive into the research topics of 'Optimising Sensor Path Planning with Reinforcement Learning and Passive Sonar Modelling'. Together they form a unique fingerprint.

Cite this