Story Cloze Test and ROCStories Corpora

Take the Story Cloze Test on CodaLab!
Story Cloze Test on LSDSem'17 page!
Please fill in this form to get access to the data, which is free to everyone.
The spring 2016 release includes 49,255 ROCStories and 3,744 Story Cloze Test instances. The next release will be in Winter 2017.
'Story Cloze Test' is a new framework for evaluating story understanding, story generation, and script learning. This test requires a system to choose the correct ending to a four-sentence story. We propose the Story Cloze Test to replace the state-of-the-art for evaluating narrative structure learning, the 'Narrative Cloze Test' (Chambers & Jurafsky, 2008).
To enable the Story Cloze Test, we created a new corpus of five-sentence commonsense stories, 'ROCStories'. This corpus is unique in two ways: (1) it captures a rich set of causal and temporal commonsense relations between daily events, and (2) it is a high quality collection of everyday life stories that can also be used for story generation.

Examples

Story Cloze Test

Context Right Ending Wrong Ending
Karen was assigned a roommate her first year of college. Her roommate asked her to go to a nearby city for a concert. Karen agreed happily. The show was absolutely exhilarating. Karen became good friends with her roommate. Karen hated her roommate.
Jim got his first credit card in college. He didn’t have a job so he bought everything on his card. After he graduated he amounted a $10,000 debt. Jim realized that he was foolish to spend so much money. Jim decided to devise a plan for repayment. Jim decided to open another credit card.
Gina misplaced her phone at her grandparents. It wasn’t anywhere in the living room. She realized she was in the car before. She grabbed her dad’s keys and ran outside. She found her phone in the car. She didn’t want her phone anymore.

ROCStories

Title Five-sentence Story
The Test Jennifer has a big exam tomorrow. She got so stressed, she pulled an all-nighter. She went into class the next day, weary as can be. Her teacher stated that the test is postponed for next week. Jennifer felt bittersweet about it.
The Hurricane Morgan and her family lived in Florida. They heard a hurricane was coming. They decided to evacuate to a relative's house. They arrived and learned from the news that it was a terrible storm. They felt lucky they had evacuated when they did.
Spaghetti Sauce Tina made spaghetti for her boyfriend. It took a lot of work, but she was very proud. Her boyfriend ate the whole plate and said it was good. Tina tried it herself, and realized it was disgusting. She was touched that he pretended it was good to spare her feelings.

Main Paper

A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli and James Allen In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies(NAACL HLT), 2016 [Presentation] [Bibtex]

Relevant Papers and Resources

CaTeRS: Causal and Temporal Relation Scheme for Semantic Annotation of Event Structures Nasrin Mostafazadeh, Alyson Grealish, Nathanel Chambers, James Allen and Lucy Vanderwende In Proceedings of NAACL Event Workshop, 2016 [Presentation] [Bibtex] [Annotated ROCStories Dataset]
Story Cloze Evaluator: Vector Space Representation Evaluation by Predicting What Happens Next Nasrin Mostafazadeh, Lucy Vanderwende, Wen-Tau (Scott) Yih, Pushmeet Kohli, James Allen In Proceedings of ACL Workshop on Evaluating Vector Space Representations for NLP (RepEval), 2016 [Bibtex]

Any questions? Please contact: me [at] nasrin [dot] info