Files and links (1)
Preprint VersionCC BY V4.0, Open Access
Preprint
Offline Meta-Reinforcement Learning with Flow-Based Task Inference and Adaptive Correction of Feature Overgeneralization
arXiv, Vol.12 January 2026
Cornell University
2026
Abstract
Offline meta-reinforcement learning (OMRL) combines the strengths of learning from diverse datasets in offline RL with the adaptability to new tasks of meta-RL, promising safe and efficient knowledge acquisition by RL agents. However, OMRL still suffers extrapolation errors due to out-of-distribution (OOD) actions, compromised by broad task distributions and Markov Decision Process (MDP) ambiguity in meta-RL setups. Existing research indicates that the generalization of the$Q$network affects the extrapolation error in offline RL. This paper investigates this relationship by decomposing the$Q$value into feature and weight components, observing that while decomposition enhances adaptability and convergence in the case of high-quality data, it often leads to policy degeneration or collapse in complex tasks. We observe that decomposed$Q$values introduce a large estimation bias when the feature encounters OOD samples, a phenomenon we term ''feature overgeneralization''. To address this issue, we propose FLORA, which identifies OOD samples by modeling feature distributions and estimating their uncertainties. FLORA integrates a return feedback mechanism to adaptively adjust feature components. Furthermore, to learn precise task representations, FLORA explicitly models the complex task distribution using a chain of invertible transformations. We theoretically and empirically demonstrate that FLORA achieves rapid adaptation and meta-policy improvement compared to baselines across various environments.
Details
- Title
- Offline Meta-Reinforcement Learning with Flow-Based Task Inference and Adaptive Correction of Feature Overgeneralization
- Authors
- Min Wang (Corresponding Author) - Beijing Institute of TechnologyXin Li - Beijing Institute of TechnologyMingzhong Wang - University of the Sunshine Coast, Queensland, School of Science, Technology and EngineeringHasnaa Bennis - Beijing Institute of Technology
- Publication details
- arXiv, Vol.12 January 2026
- Publisher
- Cornell University
- Date published
- 2026
- DOI
- 10.48550/arxiv.2601.07164
- ISSN
- 2331-8422
- Grant note
- This work was partially supported by the NSFC under Grants 92270125 and 62276024; by the Fundamental Research Funds for the Central Universities, JLU, under Grant 93K172025K01; and by the Fundamental Research Funds for the Central Universities under Grant 2025CX01010.
- Organisation Unit
- Healthy Ageing Research Cluster; School of Science, Technology and Engineering
- Language
- English
- Record Identifier
- 991199944302621
- Output Type
- Preprint
Metrics
1 Record Views