[Tuesday, August 29, 2017 at 10:00 AM in Wegmans Hall 2506]
Social multimedia refers to the multimedia content generated by online social network users. The increasing popularity of online social networks accumulates large amounts of social network activity records, making the analysis of online social activities possible. In particular, we research the problem of whether online user' generated content can be employed to understand people's real-world lives. The large-scale data have attracted people from both industrial and academic communities to mine interesting patterns from the hidden signals in the online users' activities. Researchers have successfully employed the signals extracted from social network content to finish tasks in a wide range of applications, including real world activity prediction and content recommendation. From this perspective, we can view social multimedia as sensors, which provide online signals reflecting people's real-world activities.
This thesis focuses on sentiment analysis of online social multimedia content. We first propose two approaches to visual sentiment analysis, which is directed to predict human beings' responses towards visual stimuli. Next, we build currently the largest visual emotion dataset, where each image is labeled by five Amazon Mechanical Turkers. Because the lack of high-quality dataset is one of the main difficulties in building effective machine learning models, we expect the release of this dataset will benefit the research community by empowering the development of efficient algorithms for visual emotion analysis. We then conduct a multimodality sentiment analysis. Two different approaches are proposed. The first approach tries to enforce a consistency between the prediction results from textual and visual signal respectively. The second method attempts to match textual and visual concepts and produces robust representations for the sentiment classification model. Furthermore, we try to automatically generate sentiment-bearing image captions with different sentiment labels. The results suggest that the model can focus on different image regions to produce the matching sentiment related descriptions.