With the rapid development of the computing devices, it is more and more important for Human Computer Interaction (HCI) in our daily life. Next generation HCI should be natural, efficient, and user-centered. Vision-based gesture/sign language recognition is one of the core technologies for interaction. Therefore, this team focuses on the gesture analysis and Chinese sign language recognition.
Natural Human Computer Interaction means that people can communicate with computing devices in a natural way through gesture, action, emotional expressions, etc., exploring the environment through the vision and manipulation of physical objects.
Our team focuses on the sign language recognition and gesture analysis. The detailed research topics are given below:
Research topic #1: Sign Language Representation and Modeling
Sign language is the main communication tool for hearing impaired person. However, sign language is still hard to understand for the well-hearing people. Our group aims to build the robust sign language representation and modeling. We have investigated the representations from the sparse to dense level.
Fig.1 Framework of SLR with sparse observation alignment.
[1] Hanjie Wang, Xiujuan Chai, Xilin Chen, "Sparse Observation(SO) Alignment for Sign Language Recognition", Neurocomputing, Jan. 2016
[2] Hanjie Wang, Xiujuan Chai, Xiaopeng Hong, Guoying Zhao, Xilin Chen, "Isolated Sign Language Recognition with Grassmann Covariance Matrices", ACM Transactions on Accessible Computing, Jan. 2016.
Research topic #2: Signer-Independent Sign Language Recognition
Sign language recognition(SLR) is an interesting but difficult problem. One of the biggest challenges comes from the complex inter-signer variations. However, most of the researches focused on signer dependent situation, in which the signer of the probe has been seen in the training set. In real applications, the performance will decrease dramatically when the user is new to the system. Since collecting enough training data from each new signer to retrain the SLR model is not realistic, the signer independent SLR is an urgent problem for the practice of SLR technique. Our group explored this problem from two aspects: signer adaptation and building generic recognition model.
Fig. 2 Framework of signer-independent SLR with iRDML.
[3] Fang Yin, Xiujuan Chai, Yu Zhou and Xilin Chen. Semantics Constrained Dictionary Learning for Signer-independent Sign Language Recognition. The 22nd IEEE International Conference on Image Processing (ICIP2015), 2015, Quebec City, Canada.
[4] Fang Yin, Xiujuan Chai, Yu Zhou and Xilin Chen. Weakly Supervised Metric Learning towards Signer Adaptation for Sign Language Recognition. British Machine Vision Conference (BMVC2015), 2015, Swanse, UK.
[5] Fang Yin, Xiujuan Chai, Xilin Chen, "Iterative Reference Driven Metric Learning for Signer-Independent Isolated Sign Language Recognition", European Conference on Computer Vision(ECCV2016), Amsterdam, Netherlands.
Research topic #3: Continuous Sign Language Recognition
Although Sign Language Recognition(SLR) has achieved big progress in recent years, the continuous SLR remains challenging for the difficulties in sign spotting, ME modeling and real-time recognition. We focus on spotting-recognition framework to tackle the large scale continuous sign language / gesture recognition problem with the RGB-D data input. Our team won the first places in both 2016 and 2017 ChaLearn LAP large-scale Continuous Gesture Recognition Challenges.
Fig. 3. Pipeline of continuous gesture recognition with hand-oriented ST feature.
[6] Xiujuan Chai, Zhipeng Liu, Fang Yin, Zhuang Liu, Xilin Chen, "Two Streams Recurrent Neural Networks for Large-Scale Continuous Gesture Recognition", International Conference on Pattern Recognition Workshop(ICPR2016), 2016.
[7] Zhipeng Liu, Xiujuan Chai, Zhuang Liu, Xilin Chen, "Continuous Gesture Recognition with Hand-Oriented Spatiotemporal Feature", IEEE International Conference on Computer Vision(ICCV2017)workshop, pp.4321-4329, 2017.
Research topic #4: Related Researches on Gesture Analysis and Interactions
Gesture interaction means to convey meanings and operate the devices through the interpretation of body/hand language. Nowadays, gesture analysis is a widely used interactive technique in the VR and AR areas. Towards gesture interactions, we also investigate some related key technologies, including hand detection, gesture recognition and 3D hand pose estimation etc.
Fig. 4. 3D hand pose estimation with kinematic constrained cascaded Autoencoder.
[8] Yushun Lin, Xiujuan Chai, Xilin Chen. “Kinematic Constrained Cascaded Autoencoder for Real-time Hand Pose Estimation”,FG, 2018.
Research topic #5: Activity Quality Assessment and Sign Language Learning Instructor
Action quality assessment is a new area which is getting more and more attention. The main target of action quality assessment is to evaluate the quality of actions. Nowadays, action quality assessment has many potential applications, such as scoring athletes’ performance in sports events, medical rehabilitation tests, dancing or yoga teaching, sign language self-learning and so on. We start the investigation of this problem by using deep learning technique and hope it can be integrated into our developed sign language learning instructor.
Fig. 5 End-to-End learning for score predicting on activity quality assessment.
[9] Xiujuan Chai, Zhuang Liu, Yongjun Li, Fang Yin, Xilin Chen, "SignInstructor: An Effective Tool for Sign Language Vocabulary Learning", ACPR, 2017.
