Microsoft’s ‘CaptionBot’ Adds Incorrect Captions to Your Favourite Pictures




/ 9 years ago

Captionbot 2

Microsoft, as part of its new research into storytelling by artificial intelligence, has released CaptionBot, an AI designed to recognise images and add an appropriate descriptive caption. However, like its previous attempt at AI – chatbot Tay – CaptionBot isn’t entirely successful. As with Tay, though, the results are hilarious (and without any fascistic or incestuous overtones).

Captionbot 1

The accompanying academic paper, titled Visual Storytelling [PDF], describes how the Microsoft Sequential Image Narrative Dataset (SIND) applies value judgements to picture content, setting, composition, and human expression in an attempt to describe the scene. The paper adds:

“There is a significant difference, yet unexplored, between remarking that a visual scene shows “sitting in a room” – typical of most image captioning work – and that the same visual scene shows “bonding”. The latter description is grounded in the visual signal, yet it brings to bear information about social relations and emotions that can be additionally inferred in context.”

Captionbot 3

To set CaptionBot’s base level, 10,117 CC-licensed Flickr albums were ploughed through by Amazon Mechanical Turks, who assigned tradition captions to a series of pictures. An ‘average’ description of each picture was derived by the multitude of entries, and that average was reduced to an algorithm that CaptionBot could apply to fresh images in order to evaluate them.

Captionbot 4

“Captioning is about taking concrete objects and putting them together in a literal description,” Margaret Mitchell, lead researcher on the project, said in a Microsoft blog post. “What I’ve been calling visual storytelling is about inferring conceptual and abstract ideas from those concrete objects.”


Topics: , , , , ,

Support eTeknix.com

By supporting eTeknix, you help us grow and continue to bring you the latest newsreviews, and competitions. Follow us on FacebookTwitter and Instagram to keep up with the latest technology news, reviews and more. Share your favourite articles, chat with the team and more. Also check out eTeknix YouTube, where you'll find our latest video reviews, event coverage and features in 4K!

Looking for more exciting features on the latest technology? Check out our What We Know So Far section or our Fun Reads for some interesting original features.

eTeknix Facebook eTeknix Twitter eTeknix Instagram eTeknix Instagram
  • Be Social With eTeknix

    Facebook Twitter YouTube Instagram Reddit RSS Discord Patreon TikTok Twitch
  • Features


Send this to a friend
})