Using chinese puppetry as a form of inspiration, Google has created ShadowPlay. An interactive installation using TensorFlow  and TPUs as well as AI to recognize a person’s hand gestures and transform the movements into digital animations that represents the 12 animals of the Chinese zodiac.

ShadowPlay made its debut at the World AI Conference and Google Developers Day in Shanghai in September. In order to increase user experience, Google had gone on to develop a custom machine learning model that was trained on a dataset made up of examples of people’s hand shadows.

This allowed the system to recognize the shadow and match it to a corresponding animal .

“In order to bring this project to life, we asked Googlers to help us train the model by making a lot of fun hand gestures. Once we saw the reaction of users seeing their hand shadows morph into characters, it was impossible not to smile!”, says Miguel de Andres-Clavera, Project Lead at Google.