|
Currently, I am a first-year Ph.D. student in the Department of Computer Science, University of Rochester (UR), advised by Prof. Jiebo Luo. I aim at building multimodal interactive AI systems that can not only ground and reason over the external world signals, to understand human language, but also assist humans in decision-making and efficiently solving social concerns, e.g., robot. As steps towards this goal, my research interests include but are not limited to Multimodal Large Language Models, Video Language Alignment. Prior to that, I got my master's degree from Peking University (PKU) in 2023, advised by Prof. Li Yuan and Prof. Jie Chen. And I obtained the honored bachelor's degree from University of Electronic Science and Technology of China (UESTC) in 2020. ( Note: I am actively seeking a summer internship in the U.S. for 2024. Feel free to connect with me:) ） |
|
|
|
|
|
|
|
|
|
|
|
|
My current research mainly focues on vision+language. *Equal Contribution. Representative works are highlighted. |
|
Video-Text as Game Players: Hierarchical Banzhaf Interaction for Cross-Modal Representation Learning Peng Jin, Jinfa Huang, Pengfei Xiong, Shangxuan Tian, Chang Liu, Xiangyang Ji, Li Yuan, Jie Chen IEEE International Conference on Computer Vision and Pattern Recognition, CVPR 2023 (Highlight Presentation, Top 2.5%) [Paperlink], [Code] Area: Video-and-Language Representation, Game Theory, Video-Text Retrieval In this paper, we creatively model video-text as game players with multivariate cooperative game theory to wisely handle the uncertainty during fine-grained semantic interaction with diverse granularity, flexible combination, and vague intensity. |
|
Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representations Peng Jin*, Jinfa Huang*, Fenglin Liu, Xian Wu, Shen Ge, Guoli Song, David A. Clifton, Jie Chen Conference on Neural Information Processing Systems, NeurIPS 2022 (Spotlight Presentation, Top 5%) [Paperlink], [Code] Area: Video-and-Language Representation, Machine Learning, Video-Text Retrieval, Video Captioning To solve the problem of modality gap in video-text feature space, we propose Expectation-Maximization Contrastive Learning (EMCL) to learn compact video-and-language representations. We use the Expectation-Maximization algorithm to find a compact set of bases for the latent space, where the features could be concisely represented as the linear combinations of these bases. |
|
Weakly-Supervised 3D Spatial Reasoning for Text-based Visual Question Answering Hao Li, Jinfa Huang, Peng Jin, Guoli Song, Qi Wu, Jie Chen IEEE Transactions on Image Processing, TIP 2023 [Paperlink] Area: 3D Spatial Reasoning, Text-based Visual Question Answering Existing approaches are constrained within 2D spatial information learned from the input images and rely on transformer-based architectures to reason implicitly during the fusion process. spatial reasoning between texts and objects is crucial in TextVQA, we introduce 3D geometric information into a human-like spatial reasoning process to capture the contextual knowledge of key objects step-by-step. |
|
Cross-Modality Time-Variant Relation Learning for Generating Dynamic Scene Graphs [Paperlink] , [Code] Jingyi Wang, Jinfa Huang, Can Zhang, Zhidong Deng Proceedings of the International Conference on Robotics and Automation, ICRA 2023 Area: Cross-Modal Representation, Scene Graph Generation In the process of temporal and spatial modeling during dynamic scene graph generation, it is particularly intractable to learn time-variant relations in dynamic scene graphs among frames. In this paper, we propose a Time-variant Relation-aware TRansformer, which aims to model the temporal change of relations in dynamic scene graphs. |
|
|
|
|
|
|
|
Last updated on Oct, 2023
This awesome template borrowed from this good man~
|
