Preprint
Good Better Best: Self-Motivated Imitation Learning for noisy Demonstrations
arXiv, Vol.24 October 2023
Cornell University
2023
Abstract
Imitation Learning (IL) aims to discover a policy by minimizing the discrepancy between the agent's behavior and expert demonstrations. However, IL is susceptible to limitations imposed by noisy demonstrations from non-expert behaviors, presenting a significant challenge due to the lack of supplementary information to assess their expertise. In this paper, we introduce Self-Motivated Imitation LEarning (SMILE), a method capable of progressively filtering out demonstrations collected by policies deemed inferior to the current policy, eliminating the need for additional information. We utilize the forward and reverse processes of Diffusion Models to emulate the shift in demonstration expertise from low to high and vice versa, thereby extracting the noise information that diffuses expertise. Then, the noise information is leveraged to predict the diffusion steps between the current policy and demonstrators, which we theoretically demonstrate its equivalence to their expertise gap. We further explain in detail how the predicted diffusion steps are applied to filter out noisy demonstrations in a self-motivated manner and provide its theoretical grounds. Through empirical evaluations on MuJoCo tasks, we demonstrate that our method is proficient in learning the expert policy amidst noisy demonstrations, and effectively filters out demonstrations with expertise inferior to the current policy.
Details
- Title
- Good Better Best: Self-Motivated Imitation Learning for noisy Demonstrations
- Authors
- Ye Yuan (Author) - Beijing Institute of TechnologyXin Li (Author) - Beijing Institute of TechnologyYong Heng (Author)Leiji Zhang (Author) - Beijing Institute of TechnologyMingZhong Wang (Author) - University of the Sunshine Coast, Queensland, School of Science, Technology and Engineering
- Publication details
- arXiv, Vol.24 October 2023
- Publisher
- Cornell University
- DOI
- 10.48550/arxiv.2310.15815
- ISSN
- 2331-8422
- Organisation Unit
- School of Science, Technology and Engineering
- Language
- English
- Record Identifier
- 99971752302621
- Output Type
- Preprint
Metrics
16 Record Views