Preprint
Towards Control-Centric Representations in Reinforcement Learning from Images
arXiv, Vol.25 October 2023
Cornell University
2023
Abstract
Image-based Reinforcement Learning is a practical yet challenging task. A major hurdle lies in extracting control-centric representations while disregarding irrelevant information. While approaches that follow the bisimulation principle exhibit the potential in learning state representations to address this issue, they still grapple with the limited expressive capacity of latent dynamics and the inadaptability to sparse reward environments. To address these limitations, we introduce ReBis, which aims to capture control-centric information by integrating reward-free control information alongside reward-specific knowledge. ReBis utilizes a transformer architecture to implicitly model the dynamics and incorporates block-wise masking to eliminate spatiotemporal redundancy. Moreover, ReBis combines bisimulation-based loss with asymmetric reconstruction loss to prevent feature collapse in environments with sparse rewards. Empirical studies on two large benchmarks, including Atari games and DeepMind Control Suit, demonstrate that ReBis has superior performance compared to existing methods, proving its effectiveness.
Details
- Title
- Towards Control-Centric Representations in Reinforcement Learning from Images
- Authors
- Chen Liu (Author) - Beijing Institute of TechnologyHongyu Zang (Author) - Beijing Institute of TechnologyXin Li (Author) - Beijing Institute of TechnologyYong Heng (Author)Yifei Wang (Author) - Peking UniversityZhen Fang (Author) - Beijing Institute of TechnologyYisen Wang (Author) - Peking UniversityMingzhong Wang (Author) - University of the Sunshine Coast, Queensland, School of Science, Technology and Engineering
- Additional notes
- Published as a conference paper at ICLR 2024.
- Publication details
- arXiv, Vol.25 October 2023
- Publisher
- Cornell University
- DOI
- 10.48550/arxiv.2310.16655
- ISSN
- 2331-8422
- Organisation Unit
- School of Science, Technology and Engineering
- Language
- English
- Record Identifier
- 99971752202621
- Output Type
- Preprint
Metrics
12 Record Views