Files and links (1)
Preprint VersionCC BY V4.0, Open Access
Preprint
Wavelet Predictive Representations for Non-Stationary Reinforcement Learning
arXiv, Vol.6 October 2025
Cornell University
2025
Abstract
The real world is inherently non-stationary, with ever-changing factors, such as weather conditions and traffic flows, making it challenging for agents to adapt to varying environmental dynamics. Non-Stationary Reinforcement Learning (NSRL) addresses this challenge by training agents to adapt rapidly to sequences of distinct Markov Decision Processes (MDPs). However, existing NSRL approaches often focus on tasks with regularly evolving patterns, leading to limited adaptability in highly dynamic settings. Inspired by the success of Wavelet analysis in time series modeling, specifically its ability to capture signal trends at multiple scales, we propose WISDOM to leverage wavelet-domain predictive task representations to enhance NSRL. WISDOM captures these multi-scale features in evolving MDP sequences by transforming task representation sequences into the wavelet domain, where wavelet coefficients represent both global trends and fine-grained variations of non-stationary changes. In addition to the auto-regressive modeling commonly employed in time series forecasting, we devise a wavelet temporal difference (TD) update operator to enhance tracking and prediction of MDP evolution. We theoretically prove the convergence of this operator and demonstrate policy improvement with wavelet task representations. Experiments on diverse benchmarks show that WISDOM significantly outperforms existing baselines in both sample efficiency and asymptotic performance, demonstrating its remarkable adaptability in complex environments characterized by non-stationary and stochastically evolving tasks.
Details
- Title
- Wavelet Predictive Representations for Non-Stationary Reinforcement Learning
- Authors
- Min Wang - Beijing Institute of TechnologyXin Li - Beijing Institute of TechnologyYe He - Beijing Institute of TechnologyYao-Hui Li - Beijing Institute of TechnologyHasnaa Bennis - Beijing Institute of TechnologyRiashat Islam - Microsoft Research New York CityMingzhong Wang - University of the Sunshine Coast, Queensland, School of Science, Technology and Engineering
- Publication details
- arXiv, Vol.6 October 2025
- Publisher
- Cornell University
- Date published
- 2025
- DOI
- 10.48550/arxiv.2510.04507
- ISSN
- 2331-8422
- Organisation Unit
- Healthy Ageing Research Cluster; School of Science, Technology and Engineering
- Language
- English
- Record Identifier
- 991168207202621
- Output Type
- Preprint
Metrics
4 Record Views