中国科学院 丨 所主站丨 中文
Research
Affective Computing
Audio-Visual Speech Understanding Group
Face Recognition
Media Learning and Knowledge Reasoning
Multimedia Computing and Multimodal Intelligent
Multimodal Biometric
Sign Language Recognition
Visual Modeling
Visual Scene Understanding
Media Learning and Knowledge Reasoning
Leader： Shuhui Wang (Professor) / Qingming Huang (Professor)
Email： wangshuhui [at] ict dot ac dot cn; qmhuang [at] ucas dot ac dot cn
Introduction of research group

The human-machine collaboration and interaction are facilitated by comprehensive understanding on multimodal data (e.g., vision, text and speech) and explainable/trustable multimodal reasoning for acquiring/exploiting knowledge from the multimodal information and the interaction process.

Towards trustable human-machine collaboration and interaction, the Media Learning and Knowledge Reasoning (MLKR) group is working on the learning theory, techniques and prototype systems on multimodal data. Major research topics include multimodal understanding, retrieval, recommendation, QA/dialogue and cross-modal content generation. For substantial development of next generation multimodal AI, MLKR group also enforces the theoretic study on new multimodal learning architectures and models inspired by inter-disciplinary thinking between AI and statistics, physics, brain science as well as social science.

During the past 5 years, MLKR group has published 100+ papers in top-tier conferences and journals, including TPAMI, IJCV, CVPR, ICCV, NeurIPS, ICML and ACMMM. Some of the works have been implemented into commercial systems such as multimodal-dialogue based psychological consultation and web content monitoring.

Papers

Journal Papers

  • Shuhui Wang, Ling Hu, Liang Li, Weigang Zhang, Qingming Huang, "Two-stream deep sparse network for accurate and efficient image restoration," Computer Vision and Image Understanding, Vol. 200, June 2020.
  • Yiling Wu, Shuhui Wang, Qingming Huang, "Online Fast Adaptive Low-Rank Similarity Learning for Cross-Modal Retrieval," IEEE Transactions on Multimedia, vol. 22, no. 5, pp. 1310-1322, May 2020.
  • Xuejing Liu, Liang Li, Shuhui Wang, Zhengjun Zha and Qingming Huang. Local-binarized Very Deep Residual Network for Visual Categorization. Neurocomputing, 430:82-93, 2021.
  • Qianqian Xu, Jiechao Xiong, Xiaochun Cao, Qingming Huang and Yuan Yao. Evaluating Visual Properties via Robust HodgeRank. International Journal of Computer Vision (IJCV), 129(5):1732-1753, 2021.
  • Zhiyong Yang, Qianqian Xu, Xiaochun Cao and Qingming Huang. Task-Feature Collaborative Learning with Application to Personalized Attribute Prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 43(11): 4094-4110, 2021.
  • Qianqian Xu, Zhiyong Yang, Yangbangyan Jiang, Xiaochun Cao, Yuan Yao and Qingming Huang. Not All Samples are Trustworthy: Towards Deep Robust SVP Prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 44(6): 3154-3169, 2022.
  • Junbao Zhuo, Shuhui Wang, and Qingming Huang. Uncertainty Modeling for Robust Domain Adaptation Under Noisy Environments. IEEE Transactions on Multimedia (TMM), 2022. (Accepted)
  • Shilong Bao, Qianqian Xu, Zhiyong Yang, Xiaochun Cao and Qingming Huang. Rethinking Collaborative Metric Learning: Toward an Efficient Alternative without Negative Sampling. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2021. (Accepted)
  • Yangbangyan Jiang, Xiaodan Li, Yuefeng Chen, Yuan He, Qianqian Xu, Zhiyong Yang, Xiaochun Cao and Qingming Huang. MaxMatch: Semi-Supervised Learning with Worst-Case Consistency. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022. (Accepted)
  • Zitai Wang, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao and Qingming Huang. Optimizing Partial Area Under the Top-k Curve: Theory and Practice. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022. (Accepted)
  • Ke Ma, Qianqian Xu, Jinshan Zeng, Guorong Li, Xiaochun Cao and Qingming Huang. A Tale of HodgeRank and Spectral Method: Target Attack Against Rank Aggregation is the Fixed Point of Adversarial Game. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022. (Accepted)
  • Zhiyong Yang, Qianqian Xu, Shilong Bao, Yuan He, Xiaochun Cao and Qingming Huang. Optimizing Two-way Partial AUC with an End-to-end Framework. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022. (Accepted)

Conference Papers

  • Jun Wei, Shuhui Wang, Qingming Huang, \"F3Net: Fusion, Feedback and Focus for Salient Object Detection,\" AAAI Conference on Artificial Intelligence 2020 (AAAI 2020), pp. 12321-12328, 2020.
  • Qianqian Xu, Jiechao Xiong, Zhiyong Yang, Xiaochun Cao, Qingming Huang, Yuan Yao, \"Who Likes What? – SplitLBI in Exploring Preferential Diversity of Rating,\" AAAI Conference on Artificial Intelligence (AAAI 2020), pp. 262-269, 2020.
  • Dechao Meng, Liang Li, Xuejing Liu, Yadong Li, Shijie Yang, Zheng-Jun Zha, Xingyu Gao, Shuhui Wang, and Qingming Huang, \"Parsing-based View-aware Embedding Network for Vehicle Re-Identification,\" IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2020), pp. 7103-7112, 2020.
  • Jun Wei, Shuhui Wang, Zhe Wu, Chi Su, Qingming Huang, Qi Tian, \"Label Decoupling Framework for Salient Object Detection,\" IEEE Conference on Computer Vision and Pattern Recognition(CVPR 2020), pp. 13025-13034, 2020.
  • Shuhao Cui, Shuhui Wang, Junbao Zhuo, Chi Su, Qingming Huang, Qi Tian, \"Gradually Vanishing Bridge for Adversarial Domain Adaptation,\" IEEE Conference on Computer Vision and Pattern Recognition(CVPR 2020), pp. 12455-1264, 2020.
  • Shuhao Cui, Shuhui Wang, Junbao Zhuo, liang li, Qingming Huang, Qi Tian, \"Towards Discriminability and Diversity: Batch Nuclear-norm Maximization under Label Insufficient Situations,\" IEEE Conference on Computer Vision and Pattern Recognition(CVPR 2020), pp. 3941-3950, 2020.
  • Beichen Zhang, Liang Li, Shijie Yang, Shuhui Wang, Zheng-Jun Zha, Qingming Huang, "State-Relabeling Adversarial Active Learning," IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2020, pp. 8753-8762, 2020.
  • Xinzhe Han, Shuhui Wang, Chi Su, Weigang Zhang, Qingming Huang, Qi Tian, "Interpretable Visual Reasoning via Probabilistic Formulation Under Natural Supervision," Proceedings of the 16th European Conference on Computer Vision (ECCV), LNCS 12347, Vol.2, pp. 553-570, Glasgow, UK / Cyberspace, August 23-28, 2020.
  • Zhenhuan Liu, Liang Li, Shaofei Cai, Jincan Deng, Qianqian Xu, Shuhui Wang, Qingming Huang, "IR-GAN: Image Manipulation with Linguistic Instruction by Increment Reasoning," 28th ACM International Conference on Multimedia (ACM Multimedia 2020), Seattle, United States, October 12-16, 2020.
  • Xuejing Liu, Liang Li, Shuhui Wang, Zheng-Jun Zha, Dechao Meng, Qingming Huang, "Transferrable Referring Expression Grounding with Concept Transfer and Context Inheritance," 28th ACM International Conference on Multimedia (ACM Multimedia 2020), Seattle, United States, October 12-16, 2020.
  • Dechao Meng, Liang Li, Shuhui Wang, Xingyu Gao, Zheng-Jun Zha, Qingming Huang, "Fine-grained Feature Alignment with Part Perspective Transformation for Vehicle ReID," 28th ACM International Conference on Multimedia (ACM Multimedia 2020), Seattle, United States, October 12-16, 2020.
  • Liang Li, Shijie Yang, Li Su, Shuhui Wang, Chenggang Yan, Zhengjun Zha, Qingming Huang, "Diverter-Guider Recurrent Network for Diverse Poems Generation from Image," 28th ACM International Conference on Multimedia (ACM Multimedia 2020), pp.3875–3883, Seattle, United States, October 12-16, 2020.
  • Shuhao Cui, Xuan Jin, Shuhui Wang, Yuan He, Qingming Huang. "Heuristic Domain Adaptation." NeurIPS, 2020.
  • Shuhao Cui, Xuan Jin, Shuhui Wang, Yuan He, Qingming Huang, "Heuristic Domain Adaptation," The Thirty-fourth Annual Conference on Neural Information Processing Systems(NeurIPS), Dec. 6-12, 2020.
  • Zhiyong Yang, Qianqian Xu, Shilong Bao, Yuan He, Xiaochun Cao and Qingming Huang. When All We Need is Piece of the Pie: A Generic Framework for Optimizing Two-way Partial AUC. International Conference on Machine Learning (ICML), pp. 11820-11829, Virtual Event, Jul.18-24, 2021.
  • Qianxiu Hao, Qianqian Xu, Zhiyong Yang and Qingming Huang. Pareto Optimality for Fairness-constrained Collaborative Filtering. ACM International Conference on Multimedia (ACM Multimedia), pp. 5619–5627, Chengdu, China, Oct. 20-24, 2021.
  • Xu Yan, Zhengcong Fei, Zekang Li, Shuhui Wang, Qingming Huang and Qi Tian. Semi-Autoregressive Image Captioning. ACM International Conference on Multimedia (ACM Multimedia), pp. 2708-2716, Chengdu, China, Oct. 20-24, 2021.
  • Jingru Gan, Jinchang Luo, Haiwei Wang, Shuhui Wang, Wei He and Qingming Huang. Multimodal Entity Linking: A New Dataset and A Baseline. ACM International Conference on Multimedia (ACM Multimedia), pp. 993-1001, Chengdu, China, Oct. 20-24, 2021.
  • Qianxiu Hao, Qianqian Xu, Zhiyong Yang and Qingming Huang. Learning Unified Embeddings for Recommendation via Meta-path Semantics. ACM International Conference on Multimedia (ACM Multimedia), pp. 3909–3917, Chengdu, China, Oct. 20-24, 2021.
  • Peisong Wen, Qianqian Xu, Zhiyong Yang, Yuan He and Qingming Huang. When False Positive is Intolerant: End-to-End Optimization with Low FPR for Multipartite Ranking. Annual Conference on Neural Information Processing Systems (NeurIPS), 2021. (Accepted)