Books are a rich source of both fine-grained information, how a character, an object or a scene looks like, as well as high-level semantics, what someone is thinking, feeling and how these states evolve through a story. This work aims to align books to their movie releases in order to provide rich descriptive explanations for visual content that go semantically far beyond the captions available in current datasets. To align movies and books we propose a neural sentence embedding that is trained in an unsupervised way from a large corpus of books, as well as a video-text neural embedding for computing similarities between movie clips and sentences in the book. We propose a context-aware CNN to combine information from multiple sources. We demonstrate good quantitative performance for movie/book alignment and show several qualitative examples that showcase the diversity of tasks our model can be used for.
Abstract
Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
Yukun Zhu*, Ryan Kiros*, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, Sanja Fidler
Arxiv, June 2015
* denotes equal contribution
Paper
MovieBook dataset: We no longer host this dataset. You can find movies and corresponding books on Amazon.
BookCorpus: Please visit smashwords.com to collect your own version of BookCorpus.
Data
@inproceedings{moviebook,
title = {Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books},
author = {Yukun Zhu and Ryan Kiros and Richard Zemel and Ruslan Salakhutdinov and Raquel Urtasun and Antonio Torralba and Sanja Fidler},
booktitle = {arXiv preprint arXiv:1506.06724},
year = {2015}
}
Citation
Aligning movie One Flew Over the Cuckoo's Nest
Aligning movie Harry Potter with its book
Results
"Generate" stories for CoCo Images
The club was a little emptier than I would have expected for the late afternoon, and the bartender, in red waistcoat and bowtie, was busy wiping down his counter, replacing peanuts and putting out new coasters. A television with the latest la liga news was hung in an upper corner, and behind him, rows of bottles were reflected in a giant bar mirror. Above the stools , a pergola-type overhead structure held rows of wine glasses. It was a classy place, with ferns in the corner, and not the kind of bar to which I was accustomed. My places usually had a more ... relaxed feel.
A few miles before tioga road reached highway 395 and the town of lee vining, smith turned onto a narrow blacktop road. On either side were parched, grassy open slopes with barbed-wire fences marking property lines. Cattle and horses grazed under trees whose black silhouettes stood stark against the gold-velvet mountains. Marty burst into song: “ home , home on the range, where the deer and the antelope play! Where seldom is heard a discouraging word and the skies are not cloudy all day!”
Aligning movie Harry Potter with book Fight Club
Aligning movie Batman with romantic novels
Then a frightened yell. “Hang on !” Suddenly, Jake was flying through the air. Nefertiti became airborne, too. He screamed, not knowing what was happening-then he splashed into a pool of water.
Its been years since we last played together, but as I recall, he was rather weak at the net. Or was it his serving? All I know is he plays tennis much better than he plays cricket. Perhaps , Mr. Brearly, Frances eventually replied, we should wait until we actually start playing. Then we can ascertain our oppositions faults, and make a plan based on the new information.
Since it was the middle of summer, there were candles in the fireplace instead of a fire. But it still cast a romantic glow over the room. There were candles on the mantle and on a table set up in the corner with flowers. As she looked around, her eyes instinctively turned to find max who was behind a bar opening a bottle of champagne. The doors were closed quietly behind her and her mouth felt dry as she looked across the room at the man who had haunted her dreams for so long.
The open doorway of another house provided a view of an ancient game of tiles. It wasn't the game that held reddings attention. It was the four elderly people who sat around a table playing the game. They were well beyond their productive years and the canal township had probably been their whole lives. Redding and lin ming stepped away from the doorway right into the path of a wooden pushcart.