Dual Track Multimodal Automatic Learning through Human-Robot Interaction.

Abstract

Human beings are constantly improving their cog- nitive ability via automatic learning from the inter- action with the environment. Two important as- pects of automatic learning are the visual percep- tion and knowledge acquisition. The fusion of these two aspects is vital for improving the intelligence and interaction performance of robots. Many auto- matic knowledge extraction and recognition meth- ods have been widely studied. However, little work focuses on integrating automatic knowledge extrac- tion and recognition into a unified framework to enable jointly visual perception and knowledge ac- quisition. To solve this problem, we propose a Dual Track Multimodal Automatic Learning (DT- MAL) system, which consists of two components: Hybrid Incremental Learning (HIL) from the vi- sion track and Multimodal Knowledge Extraction (MKE) from the knowledge track. HIL can in- crementally improve recognition ability of the sys- tem by learning new object samples and new ob- ject concepts. MKE is capable of constructing and updating the multimodal knowledge items based on the recognized new objects from HIL and oth- er knowledge by exploring the multimodal signals. The fusion of the two tracks is a mutual promotion process and jointly devote to the dual track learn- ing. We have conducted the experiments through human-machine interaction and the experimental results validated the effectiveness of our proposed system.

• Shuqiang Jiang, Weiqing Min, Xue Li, Huayang Wang, Jian Sun, Jiaqi Zhou, Dual Track Multimodal Automatic Learning through Human-Robot Interaction. IJCAI 2017: 4485-4491, Melbourne, Australia, August 19-25, 2017