Q-Learning-Based Optimal Control via Adaptive Critic Network for a Wankel Rotary Engine

Anthony Siming Chen; Guido Herrmann; Reza Islam; Chris Brace; James W.G. Turner; Stuart Burgess

doi:10.1109/TCST.2025.3533312

Q-Learning-Based Optimal Control via Adaptive Critic Network for a Wankel Rotary Engine

Anthony Siming Chen
, Guido Herrmann
, Reza Islam
, Chris Brace
, James W.G. Turner
, Stuart Burgess

University of Manchester
King Abdullah University of Science and Technology
University of Bristol

Research output: Contribution to journal › Article › peer-review

1 Citation (SciVal)

Abstract

We propose a new Q-learning-based air-fuel ratio (AFR) controller for a Wankel rotary engine. We first present a mean-value engine model (MVEM) that is modified based on the rotary engine dynamics. The AFR regulation problem is reformulated as an optimal proportional-integral (PI) controller for fuel tracking over the augmented error dynamics. Leveraging the generalized-Hamilton-Jacobi-Bellman (GHJB) equation, we propose a new definition of the Q-function with its arguments being the augmented error and the injected fuel flow rate. We then derive its Q-learning Bellman (QLB) equation based on the optimality principle. This allows online learning of a controller via an adaptive critic network for solving the QLB equation, of which the solution satisfies the GHJB equation. The proposed model-free Q-learning-based controller is implemented on an AIE 225CS Wankel engine, where the practical experiments validate the optimality and performance of the proposed controller.

Original language	English
Pages (from-to)	1101-1109
Number of pages	9
Journal	IEEE Transactions on Control Systems Technology
Volume	33
Issue number	3
Early online date	4 Feb 2025
DOIs	https://doi.org/10.1109/TCST.2025.3533312
Publication status	Published - 31 May 2025

Funding

Received 24 May 2024; revised 19 November 2024; accepted 19 December 2024. Date of publication 4 February 2025; date of current version 25 April 2025. This work was supported in part by the University of Bristol Ph.D. Scholarship, in part by the U.K. Research and Innovation (UKRI) Advanced Machinery and Productivity Initiative (AMPI) under Reference 84646, in part by the Engineering and Physical Sciences Research Council (EPSRC) RAIN+ Research Hub under Grant EP/W001128/1, and in part by the Innovate U.K. Advanced Propulsion Centre (APC) ADAPT-IPT under Reference 113127. Recommended by Associate Editor J. T. Gravdahl. (Corresponding author: Anthony Siming Chen.) Anthony Siming Chen and Guido Herrmann are with the Department of Electrical and Electronic Engineering, The University of Manchester, M13 9PL Manchester, U.K. (e-mail: [email protected]; [email protected]).

Funders	Funder number
Innovate U.K. Advanced Propulsion Centre
University of Bristol
UK Research and Innovation	84646
Engineering and Physical Sciences Research Council	EP/W001128/1
UCLA Asia Pacific Center	113127

Keywords

Adaptive critic
adaptive optimal control
air-fuel ratio (AFR) control
reinforcement learning
rotary engines

ASJC Scopus subject areas

Control and Systems Engineering
Electrical and Electronic Engineering

Access to Document

10.1109/TCST.2025.3533312

https://research.manchester.ac.uk/en/publications/q-learning-based-optimal-control-via-adaptive-critic-network-for-Licence: CC BY

Cite this

@article{1ba4de4c5d4b4f639b90ec8bb781dd09,

title = "Q-Learning-Based Optimal Control via Adaptive Critic Network for a Wankel Rotary Engine",

abstract = "We propose a new Q-learning-based air-fuel ratio (AFR) controller for a Wankel rotary engine. We first present a mean-value engine model (MVEM) that is modified based on the rotary engine dynamics. The AFR regulation problem is reformulated as an optimal proportional-integral (PI) controller for fuel tracking over the augmented error dynamics. Leveraging the generalized-Hamilton-Jacobi-Bellman (GHJB) equation, we propose a new definition of the Q-function with its arguments being the augmented error and the injected fuel flow rate. We then derive its Q-learning Bellman (QLB) equation based on the optimality principle. This allows online learning of a controller via an adaptive critic network for solving the QLB equation, of which the solution satisfies the GHJB equation. The proposed model-free Q-learning-based controller is implemented on an AIE 225CS Wankel engine, where the practical experiments validate the optimality and performance of the proposed controller.",

keywords = "Adaptive critic, adaptive optimal control, air-fuel ratio (AFR) control, reinforcement learning, rotary engines",

author = "Chen, \{Anthony Siming\} and Guido Herrmann and Reza Islam and Chris Brace and Turner, \{James W.G.\} and Stuart Burgess",

year = "2025",

month = may,

day = "31",

doi = "10.1109/TCST.2025.3533312",

language = "English",

volume = "33",

pages = "1101--1109",

journal = "IEEE Transactions on Control Systems Technology",

issn = "1063-6536",

publisher = "IEEE",

number = "3",

}

TY - JOUR

T1 - Q-Learning-Based Optimal Control via Adaptive Critic Network for a Wankel Rotary Engine

AU - Chen, Anthony Siming

AU - Herrmann, Guido

AU - Islam, Reza

AU - Brace, Chris

AU - Turner, James W.G.

AU - Burgess, Stuart

PY - 2025/5/31

Y1 - 2025/5/31

N2 - We propose a new Q-learning-based air-fuel ratio (AFR) controller for a Wankel rotary engine. We first present a mean-value engine model (MVEM) that is modified based on the rotary engine dynamics. The AFR regulation problem is reformulated as an optimal proportional-integral (PI) controller for fuel tracking over the augmented error dynamics. Leveraging the generalized-Hamilton-Jacobi-Bellman (GHJB) equation, we propose a new definition of the Q-function with its arguments being the augmented error and the injected fuel flow rate. We then derive its Q-learning Bellman (QLB) equation based on the optimality principle. This allows online learning of a controller via an adaptive critic network for solving the QLB equation, of which the solution satisfies the GHJB equation. The proposed model-free Q-learning-based controller is implemented on an AIE 225CS Wankel engine, where the practical experiments validate the optimality and performance of the proposed controller.

AB - We propose a new Q-learning-based air-fuel ratio (AFR) controller for a Wankel rotary engine. We first present a mean-value engine model (MVEM) that is modified based on the rotary engine dynamics. The AFR regulation problem is reformulated as an optimal proportional-integral (PI) controller for fuel tracking over the augmented error dynamics. Leveraging the generalized-Hamilton-Jacobi-Bellman (GHJB) equation, we propose a new definition of the Q-function with its arguments being the augmented error and the injected fuel flow rate. We then derive its Q-learning Bellman (QLB) equation based on the optimality principle. This allows online learning of a controller via an adaptive critic network for solving the QLB equation, of which the solution satisfies the GHJB equation. The proposed model-free Q-learning-based controller is implemented on an AIE 225CS Wankel engine, where the practical experiments validate the optimality and performance of the proposed controller.

KW - Adaptive critic

KW - adaptive optimal control

KW - air-fuel ratio (AFR) control

KW - reinforcement learning

KW - rotary engines

UR - https://www.scopus.com/pages/publications/85217706132

U2 - 10.1109/TCST.2025.3533312

DO - 10.1109/TCST.2025.3533312

M3 - Article

AN - SCOPUS:85217706132

SN - 1063-6536

VL - 33

SP - 1101

EP - 1109

JO - IEEE Transactions on Control Systems Technology

JF - IEEE Transactions on Control Systems Technology

IS - 3

ER -

Q-Learning-Based Optimal Control via Adaptive Critic Network for a Wankel Rotary Engine

Abstract

Funding

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this