中科院计算所视觉信息处理与学习组
中科院计算所视觉信息处理与学习组


您所在的位置 / 科学研究

科学研究
手语组

组  长:柴秀娟 副研究员

Email:chaixiujuan@ict.ac.cn

With the rapid development of the computing devices, it is more and more important for Human Computer Interaction (HCI) in our daily life. Next generation HCI should be natural, efficient, and user-centered. Vision-based gesture/sign language recognition is one of the core technologies for interaction. Therefore, this team focuses on the gesture analysis and Chinese sign language recognition.
研究内容

Natural Human Computer Interaction means that people can communicate with computing devices in a natural way through gesture, action, emotional expressions, etc., exploring the environment through the vision and manipulation of physical objects.

Our team focuses on the sign language recognition and gesture analysis. The detailed research topics are given below:

 

Research topic #1: Sign Language Representation and Modeling

Sign language is the main communication tool for hearing impaired person. However, sign language is still hard to understand for the well-hearing people. Our group aims to build the robust sign language representation and modeling. We have investigated the representations from the sparse to dense level.  


Fig.1 Framework of SLR with sparse observation alignment.

[1] Hanjie Wang, Xiujuan Chai, Xilin Chen, "Sparse Observation(SO) Alignment for Sign Language Recognition", Neurocomputing, Jan. 2016

[2] Hanjie Wang, Xiujuan Chai, Xiaopeng Hong, Guoying Zhao, Xilin Chen, "Isolated Sign Language Recognition with Grassmann Covariance Matrices", ACM Transactions on Accessible Computing, Jan. 2016.

Research topic #2: Signer-Independent Sign Language Recognition

Sign language recognition(SLR) is an interesting but difficult problem. One of the biggest challenges comes from the complex inter-signer variations. However, most of the researches focused on signer dependent situation, in which the signer of the probe has been seen in the training set. In real applications, the performance will decrease dramatically when the user is new to the system. Since collecting enough training data from each new signer to retrain the SLR model is not realistic, the signer independent SLR is an urgent problem for the practice of SLR technique. Our group explored this problem from two aspects: signer adaptation and building generic recognition model.

Fig. 2 Framework of signer-independent SLR with iRDML.

[3] Fang Yin, Xiujuan Chai, Yu Zhou and Xilin Chen. Semantics Constrained Dictionary Learning for Signer-independent Sign Language Recognition. The 22nd IEEE International Conference on Image Processing (ICIP2015), 2015, Quebec City, Canada.

[4] Fang Yin, Xiujuan Chai, Yu Zhou and Xilin Chen. Weakly Supervised Metric Learning towards Signer Adaptation for Sign Language Recognition. British Machine Vision Conference (BMVC2015), 2015, Swanse, UK.

[5] Fang Yin, Xiujuan Chai, Xilin Chen, "Iterative Reference Driven Metric Learning for Signer-Independent Isolated Sign Language Recognition", European Conference on Computer Vision(ECCV2016), Amsterdam, Netherlands.

 

Research topic #3: Continuous Sign Language Recognition

Although Sign Language Recognition(SLR) has achieved big progress in recent years, the continuous SLR remains challenging for the difficulties in sign spotting, ME modeling and real-time recognition. We focus on spotting-recognition framework to tackle the large scale continuous sign language / gesture recognition problem with the RGB-D data input. Our team won the first places in both 2016 and 2017 ChaLearn LAP large-scale Continuous Gesture Recognition Challenges. 

Fig. 3. Pipeline of continuous gesture recognition with hand-oriented ST feature.

[6] Xiujuan Chai, Zhipeng Liu, Fang Yin, Zhuang Liu, Xilin Chen, "Two Streams Recurrent Neural Networks for Large-Scale Continuous Gesture Recognition", International Conference on Pattern Recognition Workshop(ICPR2016), 2016.

[7] Zhipeng Liu, Xiujuan Chai, Zhuang Liu, Xilin Chen, "Continuous Gesture Recognition with Hand-Oriented Spatiotemporal Feature", IEEE International Conference on Computer Vision(ICCV2017)workshop, pp.4321-4329, 2017. 


Research topic #4: Related Researches on Gesture Analysis and Interactions

Gesture interaction means to convey meanings and operate the devices through the interpretation of body/hand language. Nowadays, gesture analysis is a widely used interactive technique in the VR and AR areas. Towards gesture interactions, we also investigate some related key technologies, including hand detection, gesture recognition and 3D hand pose estimation etc.

Fig. 4. 3D hand pose estimation with kinematic constrained cascaded Autoencoder.

[8] Yushun Lin, Xiujuan Chai, Xilin Chen. “Kinematic Constrained Cascaded Autoencoder for Real-time Hand Pose Estimation”,FG, 2018.    


Research topic #5: Activity Quality Assessment and Sign Language Learning Instructor

Action quality assessment is a new area which is getting more and more attention. The main target of action quality assessment is to evaluate the quality of actions. Nowadays, action quality assessment has many potential applications, such as scoring athletes’ performance in sports events, medical rehabilitation tests, dancing or yoga teaching, sign language self-learning and so on. We start the investigation of this problem by using deep learning technique and hope it can be integrated into our developed sign language learning instructor.

Fig. 5 End-to-End learning for score predicting on activity quality assessment.

[9] Xiujuan Chai, Zhuang Liu, Yongjun Li, Fang Yin, Xilin Chen, "SignInstructor: An Effective Tool for Sign Language Vocabulary Learning", ACPR, 2017.




部分论文

刊物论文

1.    Hanjie Wang, Xiujuan Chai, Xiaopeng Hong, Guoying Zhao, Xilin Chen, "Isolated Sign Language Recognition with Grassmann Covariance Matrices," ACM Transactions on Accessible Computing, 2016.

2.    Hanjie Wang, Xiujuan Chai, Xilin Chen, "Sparse Observation(SO)Alignment for Sign Language Recognition," Neurocomputing, 2016.

3.    柴秀娟,汪孔桥。基于分块局部纹理描述的手势识别。高技术通讯,20(5),pp487-492,2010年5月 (EI)

会议论文

1.    Xiujuan Chai, Zhuang Liu, Yongjun Li, Fang Yin, Xilin Chen. SignInstructor:An Effective Tool for Sign Language Vocabulary Learning. Asian Conference on Pattern Recognition, 2017. 【pdf】

2.    Zhipeng Liu, Xiujuan Chai, Zhuang Liu, Xilin Chen, "Continuous Gesture Recognition with Hand-Oriented Spatiotemporal Feature," IEEE International Conference on Computer Vision(ICCV2017)workshop, pp. 4321-4329, 2017. 【pdf】

3.    Fang Yin, Xiujuan Chai, Xilin Chen, "Iterative Reference Driven Metric Learning for Signer Independent Isolated Sign Language Recognition," European Conference on Computer Vision(ECCV2016), 2016.

4.    Fang Yin, Xiujuan Chai, Xilin Chen, "Iterative Reference Driven Metric Learning for Signer-Independent Isolated Sign Language Recognition," European Conference on Computer Vision(ECCV2016), Amsterdam, Netherlands, 2016.

5.    Xiujuan Chai, Zhipeng Liu, Fang Yin, Zhuang Liu, Xilin Chen, "Two Streams Recurrent Neural Networks for Large-Scale Continuous Gesture Recognition," International Conference on Pattern Recognition Workshop(ICPR2016), 2016. 【pdf】

6.    Xiujuan Chai, Zhipeng Liu, Fang Yin, Zhuang Liu, Xilin Chen, "Two Streams Recurrent Neural Networks for Large-Scale Continuous Gesture Recognition," International Conference on Pattern Recognition(ICPR2016), 2016.

7.    Xiujuan Chai, Hanjie Wang, Fang Yin, Xilin Chen, "Communication Tool for the Hard of Hearings," International Conference on Affective Computing and Intelligent Interaction(ACII2015), Xi'an, China, 2015. 【pdf】

8.    Fang Yin, Xiujuan Chai, Yu Zhou, Xilin Chen, "Weakly Supervised Metric Learning Towards Signer Adaptation for Sign Language Recognition," British Machine Vision Conference(BMVC2015), Swanse, UK, 2015. 【pdf】

9.    Hanjie Wang, Xiujuan Chai, Yu Zhou, Xilin Chen, "Fast Sign Language Recognition Benefited From Low Rank Approximation," IEEE International Conference on Automatic Face and Gesture Recognition(FG2015), Ljubljana, Slovenia, 2015.

10.    Fang Yin, Xiujuan Chai, Yu Zhou, Xilin Chen, "Semantics Constrained Dictionary Learning for Signer-independent Sign Language Recognition," IEEE International Conference on Image Processing(ICIP2015), Quebec City, Canada, 2015. 【pdf】


视觉信息处理和学习组
  • 单位地址:北京海淀区中关村科学院南路6号
  • 邮编:100190
  • 联系电话:010-62600514
  • Email:yi.cheng@vipl.ict.ac.cn
  • Valse

  • 深度学习大讲堂

版权所有 @ 中科院计算所视觉信息处理与学习组 京ICP备05002829号 京公网安备1101080060