The Values and Options In the Cancer Experience (VOICE)

The aim of this project is to improve the doctor-patient communication. We have analyzed 386 audio recordings of late stage cancer patients' visits to their doctors office. We have identified the styles of communication that are highly correlated with low discordance between doctors and patients.

Learn more about this project

Aging and Engaging

This project focuses on the older adults, who are socially isolated. The number of older adults in the US is 700 million. Social isolation can lead to many problems e.g., depression, suicidal tendency. This project is specially designed for older adults, allowing them to practice conversation with a virtual agent and receive feedback.

Learn more about this project

CoCo: Collaboration Coach

CoCo: Collaboration Coach is a fully automated feedback system for video-conferencing based group discussions. The platform allows participants to have discussions over video chat and thereafter receive feedback on their communication skills. The work analyzes the impact of machine generated feedback on people’s group discussion behaviors.

Learn more about this project


PARK is a web-based interface to collect the data from Parkinson’s patients. Using the online tool, patients are able to engage in a voice test and five motor tasks, from the standardized unified Parkinson's disease rating scale (UPDRS) test for diagnosing Parkinson's, that they would otherwise complete in the doctor's office. These tasks are recorded via webcam and microphone of the user’s computer. Facial and motion features are extracted from the videos. The goal is to not only detect symptoms of Parkinson's, but to detect minute physical changes in the condition which provide important information about whether or not a patient requires a change in medication. Doctors would then be able to make timely and responsive decisions.

Try it now

HEART (Holographic Eletromyographic Automated Rehabilitation Technology)

Currently, Stroke is the leading cause of adult disability in the US. Most of the stroke patients suffer from great difficulties and they need a better way to boost their rehabilitation process. Mirror therapy is the method medical professional perform to reconnect damaged brain circuits. In general, stroke rehabilitation process is best when patients are performing a great number of repetition. However, because of lack of physical abilities, the task of repetition soon seem boring and thus patients are not motivated to do what they are suggested. We are trying to use Mixed Reality (MR) technology, with the help of medical professionals, to create a rehabilitation game for the patients. This project would be interesting for patients to perform more repetition for their health.


AutoManner is an intelligent interface for making public speakers aware of their mannerism. In this project we used an unsupervised algorithm to extract common repetitive patterns of body language, which was also developed in ROCHCI.

Learn more about this project

Automated Prediction of Job Interview

Ever wondered why you have been rejected from a job despite being a qualified candidate? In this project, we provide a computational framework to quantify human behavior in the context of job interviews. We build a model by analyzing 138 recorded interview videos (total duration of 10.5 hours) of 69 internship-seeking students from Massachusetts Institute of Technology (MIT) as they spoke with professional career counselors. Our framework is not only able to predict job interview performance, but also can generate automated recommendations.

Learn more about this project

LISSA - Live Interactive Social Skills Assistance

The LISSA system aims to help individuals improve their conversational skills by simulating face-to-face conversation and providing both real-time as well as post-session summary feedback on users’ nonverbal behavior. At the same time, it manages a fairly open-ended conversation with some degree of understanding. We have developed a HMM based model for generating real-time feedback.

Learn more about this project

ROC Comment

ROC Comment is an automated interface for generating natural language comments on behavioral videos. We focus on the domain of public speaking, which many people consider their greatest fear. We collect a dataset of 196 public speaking videos from 49 individuals and gather 12,173 comments, generated by more than 500 independent human judges. We then train a k-Nearest-Neighbor (k-NN) based model by extracting prosodic (e.g., volume) and facial (e.g., smiles) features. Given a new video, we extract features and select the closest comments using k-NN model. We further filter the comments by clustering them using DBScan, and eliminating the outliers. We are working on generating comments using deep learning. Our system is available online.

Visit the site


RocSpeak is a platform that allows you to record your speech on the browser and receive automated feedback on many aspects. In addition, you may share your video to receive personal feedback. Our algorithms will compile the personal feedback you have received with the automated features to give you helpful insight.

Visit the site

TED Talk Analysis

This research captures the narrative trajectories or the "styles of storytelling" in public speeches using computational analysis of the linguistic styles. This analysis covers transcripts of over 2000 TED talks to estimate the potential audience responses. The ground truth of the analysis comes from over 5 million spontaneous annotations from the viewers in the TED Talk Website.

Learn more about this project