Chenliang Xu

pic

Actor-Action Video Understanding


Overview of Goals and Challenges:

Action is the heart of video understanding. As such, it has received a significant amount of attention in the last decade. The emphasis has moved from small datasets of a handful of actions to large datasets with many dozens of actions; from constrained domains like sporting to videos in-the-wild. However, existing works emphasize a small subset of the broader action understanding problem. First, they all assume the agent of the action, which we call the actor, is a human adult, ignoring the diversity of actions performed by other actors. Second, the prior literature largely focuses on action recognition, which is posed as the classification of a temporally pre-trimmed clip into one of k action classes from a closed-world setting. Here, we overcome both narrow viewpoints and introduces a new level of generality to the action understanding problem by considering multiple different classes of actors undergoing multiple different classes of actions. We are interested in simultaneously solving the set of 3W questions, namely, what action is happening, who is performing the action, and where is the action happening in space and time. We have developed various methods in advancing probabilistic graphical models to model actor-action interactions and have found that inference jointly over actors and actions outperforms inference independently over them. Our work has also enriched the scope of action understanding to consider temporally untrimmed long videos, adaptive video scales and weakly supervised learning setting.

Related Publications:

  1. Learning a weakly-supervised video actor-action segmentation model with a wise selection. J. Chen, Z. Li, J. Luo, and C. Xu. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020
  2. A weakly supervised multi-task ranking framework for actor-action semantic segmentation. Y. Yan, C. Xu, D. Cai, and J. J. Corso. International Journal of Computer Vision (IJCV), 2019
  3. Weakly supervised actor-action segmentation via robust multi-task ranking. Y. Yan, C. Xu, D. Cai and J. J. Corso. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  4. Action understanding with multiple classes of actors. C. Xu, C. Xiong and J. J. Corso. arXiv:1704.08723, 2017.
  5. Actor-action semantic segmentation with grouping process models. C. Xu and J. J. Corso. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  6. Can humans fly? Action understanding with multiple classes of actors. C. Xu, S.-H. Hsieh, C. Xiong and J. J. Corso. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015.