What If We Could Not See? Counterfactual Analysis for Egocentric Action Anticipation

第一视角行为预测的现有工作一部分仅利用视频中的视觉特征，忽视了行为标签的语义关联，致使行为预测效果受限；另一部分在视觉特征的基础上引入了行为标签所包含的语义信息，但是受行为标签在数据集中长尾分布的影响，预测结果更偏向于高频标签，这些因素都不利于行为预测准确率的提高。为此我们基于因果分析的理论，提出一种反事实分析的方案，我们认为对于预测的结果而言，观察到的视觉信息对应每个案例的具体信息是主因，而行为标签对应抽象的语义信息，只能挖掘出数据集总体的统计特性，反映不出每个案例具体的信息，是副因。我们需要缓解行为标签之间的语义关联对于预测结果产生的副作用，在保留多模态信息的基础上凸显出视觉信息的主作用，削弱语义信息的作用。基于反事实分析方案的第一视角行为预测分为三个阶段：掺杂偏差的事实阶段、捕获偏差的反事实阶段以及去除偏差的最终阶段。首先在掺杂偏差的事实阶段中基于过去的视觉特征和行为类别得到未来的行为类别，对应事实阶段的预测结果。然后在捕获偏差的反事实阶段，想象完全看不见、只根据过去的行为类别这种抽象的语义信息预测出未来行为的反事实场景，此时的预测结果充分捕获了语义信息带来的预测偏差。最终在去除偏差的反事实分析阶段，从事实阶段的预测结果中扣除反事实阶段的预测结果，得到最终的预测结果。

Abstract

Egocentric action anticipation aims at predicting the near future based on past observation in first-person vision. While future actions may be wrongly predicted due to the dataset bias, we present a counterfactual analysis framework for egocentric action anticipation (CA-EAA) to enhance the capacity. In the factual case, we can predict the upcoming action based on visual features and semantic labels from past observation. Imagining one counterfactual situation where no visual representation had been observed, we would obtain a counterfactual predicted action only using past semantic labels. In this way, we can reduce the side-effect caused by semantic labels via a comparison between factual and counterfactual outcomes, which moves a step towards unbiased prediction for egocentric action anticipation. We conduct experiments on two large-scale egocentric video datasets. Qualitative and quantitative results validate the effectiveness of our proposed CA-EAA.

• Tianyu Zhang, Weiqing Min, Jiahao Yang, Tao Liu, Shuqiang Jiang, Yong Rui, “What If We Could Not See? Counterfactual Analysis for Egocentric Action Anticipation”, International Joint Conference on Artificial Intelligence (IJCAI 2021): 1316-1322, Canada, August. 19-26, 2021.