Visual Information Processing and Learning
Visual Information Processing and Learning


Research
Sign Language Recognition

Leader:Xiujuan Chai (Associate Professor)

Email:chaixiujuan [at] ict.ac.cn

With the rapid development of the computing devices, it is more and more important for Human Computer Interaction (HCI) in our daily life. Next generation HCI should be natural, efficient, and user-centered. Vision-based gesture/sign language recognition is one of the core technologies for interaction. Therefore, this team focuses on the gesture analysis and Chinese sign language recognition.
Research

Natural Human Computer Interaction means that people can communicate with computing devices in a natural way through gesture, action, emotional expressions, etc., exploring the environment through the vision and manipulation of physical objects.

Our team focuses on the sign language recognition and gesture analysis. The detailed research topics are given below:

 

Research topic #1: Sign Language Representation and Modeling

Sign language is the main communication tool for hearing impaired person. However, sign language is still hard to understand for the well-hearing people. Our group aims to build the robust sign language representation and modeling. We have investigated the representations from the sparse to dense level.  



Fig.1 Framework of SLR with sparse observation alignment.

[1] Hanjie Wang, Xiujuan Chai, Xilin Chen, "Sparse Observation(SO) Alignment for Sign Language Recognition", Neurocomputing, Jan. 2016

[2] Hanjie Wang, Xiujuan Chai, Xiaopeng Hong, Guoying Zhao, Xilin Chen, "Isolated Sign Language Recognition with Grassmann Covariance Matrices", ACM Transactions on Accessible Computing, Jan. 2016.


Research topic #2: Signer-Independent Sign Language Recognition

Sign language recognition(SLR) is an interesting but difficult problem. One of the biggest challenges comes from the complex inter-signer variations. However, most of the researches focused on signer dependent situation, in which the signer of the probe has been seen in the training set. In real applications, the performance will decrease dramatically when the user is new to the system. Since collecting enough training data from each new signer to retrain the SLR model is not realistic, the signer independent SLR is an urgent problem for the practice of SLR technique. Our group explored this problem from two aspects: signer adaptation and building generic recognition model.

Fig. 2 Framework of signer-independent SLR with iRDML.

[3] Fang Yin, Xiujuan Chai, Yu Zhou and Xilin Chen. Semantics Constrained Dictionary Learning for Signer-independent Sign Language Recognition. The 22nd IEEE International Conference on Image Processing (ICIP2015), 2015, Quebec City, Canada.

[4] Fang Yin, Xiujuan Chai, Yu Zhou and Xilin Chen. Weakly Supervised Metric Learning towards Signer Adaptation for Sign Language Recognition. British Machine Vision Conference (BMVC2015), 2015, Swanse, UK.

[5] Fang Yin, Xiujuan Chai, Xilin Chen, "Iterative Reference Driven Metric Learning for Signer-Independent Isolated Sign Language Recognition", European Conference on Computer Vision(ECCV2016), Amsterdam, Netherlands.

 

Research topic #3: Continuous Sign Language Recognition

Although Sign Language Recognition(SLR) has achieved big progress in recent years, the continuous SLR remains challenging for the difficulties in sign spotting, ME modeling and real-time recognition. We focus on spotting-recognition framework to tackle the large scale continuous sign language / gesture recognition problem with the RGB-D data input. Our team won the first places in both 2016 and 2017 ChaLearn LAP large-scale Continuous Gesture Recognition Challenges. 

Fig. 3. Pipeline of continuous gesture recognition with hand-oriented ST feature.

[6] Xiujuan Chai, Zhipeng Liu, Fang Yin, Zhuang Liu, Xilin Chen, "Two Streams Recurrent Neural Networks for Large-Scale Continuous Gesture Recognition", International Conference on Pattern Recognition Workshop(ICPR2016), 2016.

[7] Zhipeng Liu, Xiujuan Chai, Zhuang Liu, Xilin Chen, "Continuous Gesture Recognition with Hand-Oriented Spatiotemporal Feature", IEEE International Conference on Computer Vision(ICCV2017)workshop, pp.4321-4329, 2017. 


Research topic #4: Related Researches on Gesture Analysis and Interactions

Gesture interaction means to convey meanings and operate the devices through the interpretation of body/hand language. Nowadays, gesture analysis is a widely used interactive technique in the VR and AR areas. Towards gesture interactions, we also investigate some related key technologies, including hand detection, gesture recognition and 3D hand pose estimation etc.

Fig. 4. 3D hand pose estimation with kinematic constrained cascaded Autoencoder.

[8] Yushun Lin, Xiujuan Chai, Xilin Chen. “Kinematic Constrained Cascaded Autoencoder for Real-time Hand Pose Estimation”,FG, 2018.    


Research topic #5: Activity Quality Assessment and Sign Language Learning Instructor

Action quality assessment is a new area which is getting more and more attention. The main target of action quality assessment is to evaluate the quality of actions. Nowadays, action quality assessment has many potential applications, such as scoring athletes’ performance in sports events, medical rehabilitation tests, dancing or yoga teaching, sign language self-learning and so on. We start the investigation of this problem by using deep learning technique and hope it can be integrated into our developed sign language learning instructor.

Fig. 5 End-to-End learning for score predicting on activity quality assessment.

[9] Xiujuan Chai, Zhuang Liu, Yongjun Li, Fang Yin, Xilin Chen, "SignInstructor: An Effective Tool for Sign Language Vocabulary Learning", ACPR, 2017.




Papers

Journal Papers

1.    Zhipeng Liu, Xiujuan Chai, Xilin Chen, "Deep Memory and Prediction Neural Network for Video Prediction", Neurocomputing. (Accepted)2018

2.    Hanjie Wang, Xiujuan Chai, Xiaopeng Hong, Guoying Zhao, Xilin Chen, "Isolated Sign Language Recognition with Grassmann Covariance Matrices," ACM Transactions on Accessible Computing, 2016. 【pdf】

3.    Hanjie Wang, Xiujuan Chai, Xilin Chen, "Sparse Observation(SO)Alignment for Sign Language Recognition," Neurocomputing, 2016. 【pdf】

4.    马志国,苗军,卿来云, 陈熙霖, 基于局部随机游走的超像素分割方法, 高技术通讯, no.10,pp.991-998, 2014.12. 【pdf】

5.    Hanjie Wang, Jingjing Fu, Yan Lu, Xilin Chen, Shipeng Li, “Depth Sensor Assisted Real-time Gesture Recognition for Interactive Presentation,” Journal of Visual Communication and Image Representation, Nov.2013. 【pdf】

6.    Yu Zhou, Xilin Chen, Debin Zhao, Hongyun Yao, Wen Gao, “Adaptive Sign Language Recognition With Exemplar Extraction and MAP/IVFS,” Signal Processing Letters, IEEE, vol.17, no.3, pp.297–300,Mar.2010.

7.    Qi Wang, Xilin Chen, Liang-Guo Zhang, Chunli Wang, Wen Gao, “Viewpoint invariant sign language recognition,” Computer Vision and Image Understanding, vol.108, no.1-2, pp.87-97,Oct.2007.

8.    Wen Gao, Gaolin Fang, Debin Zhao, Yiqiang Chen, “A Chinese Sign Language Recognition System Based on SOFM/HMM/SRN,” Pattern Recognition, vol.37, no.12, pp2389-2402,Dec.2004. 【pdf】

9.    Gaolin Fang, Wen Gao, Debin Zhao, “Large Vocabulary Sign Language Recognition Based on Fuzzy Decision Trees,” IEEE Transactions on System Man and Cybernetics, vol.34, no.3, pp305-314,May 2004. 【pdf】

10.    Gaolin Fang, Wen Gao, Zhaoqi Wang, “Incorporating Linguistic Structure into Maximum Entropy Language Models,” Journal of Computer Science and Technology, vol.18, no.1, pp131-136,Jan.2003.

11.    Wen Gao , Jiyong Ma, Jiangqin Wu and Chunli Wang, “Large Vocabulary Sign Language Recognition Based on HMM/ANN/DP,” IJPRAI, Vol.14, No.5, pp587-602, 2000.

12.    Xu Lin, Gao Wen, “Human-Computer Chinese Sign Language Interaction System,” International Journal of Virtual Reality, 2000.

13.    Xu Lin, Gao Wen, Yao Hongxun, Ma Jiyong, “Study on Translating Chinese into Chinese Sign Language,” Journal of Computer Science and Technology.vol.15 no.5 pp485-490, 2000.

Conference Papers

1.    Yongjun Li, Xiujuan Chai, Xilin Chen, “ScoringNet——Learning Key Fragment for Action Quality Assessment with Ranking Loss in Skilled Sports", Asian Conference on Computer Vision(ACCV2018), Perth, Australia, 2018.

2.    Xiujuan Chai, Zhuang Liu, Yongjun Li, Fang Yin, Xilin Chen, "SignInstructor:An Effective Tool for Sign Language Vocabulary Learning," Asian Conference on Pattern Recognition, 2017. 【pdf】

3.    Zhipeng Liu, Xiujuan Chai, Zhuang Liu, Xilin Chen, "Continuous Gesture Recognition with Hand-Oriented Spatiotemporal Feature," IEEE International Conference on Computer Vision(ICCV2017)workshop, pp. 4321-4329, 2017. 【pdf】

4.    Fang Yin, Xiujuan Chai, Xilin Chen, "Iterative Reference Driven Metric Learning for Signer-Independent Isolated Sign Language Recognition," European Conference on Computer Vision(ECCV2016), Amsterdam, Netherlands, 2016. 【pdf】

5.    Xiujuan Chai, Zhipeng Liu, Fang Yin, Zhuang Liu, Xilin Chen, "Two Streams Recurrent Neural Networks for Large-Scale Continuous Gesture Recognition," International Conference on Pattern Recognition Workshop(ICPR2016), 2016. 【pdf】

6.    Xiujuan Chai, Hanjie Wang, Fang Yin, Xilin Chen, "Communication Tool for the Hard of Hearings," International Conference on Affective Computing and Intelligent Interaction(ACII2015), Xi'an, China, 2015. 【pdf】

7.    Fang Yin, Xiujuan Chai, Yu Zhou, Xilin Chen, "Weakly Supervised Metric Learning Towards Signer Adaptation for Sign Language Recognition," British Machine Vision Conference(BMVC2015), Swanse, UK, 2015. 【pdf】

8.    Hanjie Wang, Xiujuan Chai, Yu Zhou, Xilin Chen, "Fast Sign Language Recognition Benefited From Low Rank Approximation," IEEE International Conference on Automatic Face and Gesture Recognition(FG2015), Ljubljana, Slovenia, 2015. 【pdf】

9.    Fang Yin, Xiujuan Chai, Yu Zhou, Xilin Chen, "Semantics Constrained Dictionary Learning for Signer-independent Sign Language Recognition," IEEE International Conference on Image Processing(ICIP2015), Quebec City, Canada, 2015. 【pdf】

10.    Xiaowei Zhao, Shiguang Shan, Xiujuan Chai, Xilin Chen, “Cascaded Shape Space Pruning for Robust Facial Landmark Detection,” International Conference on Computer Vision(ICCV2013),pp.1033-1040, Dec.2013. 【pdf】

11.    Xiujuan Chai, Guang Li, Xilin Chen, Ming Zhou, Guobin Wu, Hanjing Li, “VisualComm: A Tool to Support Communication between Deaf and Hearing Persons with the Kinect,” In Proceedings of The 15th ACM SIGACCESS International Conference on Computers and Accessibility(ACM SIGACCESS2013), Seattle, USA, Oct.2013. 【pdf】

12.    Xiujuan Chai, Guang Li, Yushun Lin, Zhihao Xu, Yili Tang, Xilin Chen, Ming Zhou, “Sign Language Recognition and Translation with Kinect,” In Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition(FG2013), Shanghai, CN, Apr.2013. 【pdf】

13.    Hanjie Wang, Qi Wang, Xilin Chen, “Hand Posture Recognition from Disparity Cost Map,” ACCV, Daejeon, Korea, Nov.2012. 【pdf】

14.    Xiujuan Chai, Zhihao Xu, Qian Li, Bingpeng Ma, Xilin Chen, “Robust Hand Tracking by Integrating Appearance, Location and Depth Cues,” ICIMCS, Wu Han, China, pp.60-65, Sept.2012. 【pdf】

15.    Qi Wang, Xilin Chen, Wen Gao, “Skin Color Weighted Disparity Competition for Hand Segmentation from Stereo Camera,” in Processing of British Machine Vision Conference, BMVC, Aberystwyth, UK, Aug.2010.

16.    Liangguo Zhang, Xilin Chen, Chunli Wang, Wen Gao, “Robust automatic tracking of skin-colored objects with level set based occlusion handling,” in Proceedings of The 8th Int'l Gesture Workshop, GW'09, Bielefeld, Germany, Feb.2009.

17.    Yu Zhou, Xilin Chen, Debin Zhao, Hongxun Yao, Wen Gao, “Mahalanobis Distance Based Polynomial Segment Model For Chinese Sign Language Recogniton,” in Proceedings of 2008 IEEE International Conference on Multimedia and Expo, ICME 2008, Hannover, German, Jun.23-26, 2008.

18.    Qi Wang, Xilin Chen, Chunli Wang, and Wen Gao, “A Verification Method for Viewpoint Invariant Sign Language Recognition,” International Conference on Pattern Recognition, ICPR 2006, Hong Kong, Aug.20-24, 2006.

19.    Qi Wang, Xilin Chen, Chunli Wang, Wen Gao, “Sign Language Recognition From Homography,” 2006 IEEE International Conference on Multimedia and Expo, ICME 2006, Toronto, Ontario, Canada, pp429-432, Jul.09-12, 2006.

20.    Chunli Wang, Xilin Chen, Wen Gao, “Expanding Training Set for Chinese Sign Language Recognition,” 7th International Conference on Automatic Face and Gesture Recognition (FG06), Southampton, UK, pp323-328, Apr.10-12, 2006.


Visual Information Processing and Learning
  • Address :No.6 Kexueyuan South Road
  • Zhongguancun,Haidian District
  • Beijing,China
  • Postcode :100190
  • Tel : (8610)62600514
  • Email:yi.cheng@vipl.ict.ac.cn
  • Valse

  • Big Lecture of DL

Copyright @ Visual Information Processing and Learning 京ICP备05002829号 京公网安备1101080060