Enriching a digital assistant conversation via your TV

Martijn van der Pol

A great challenge when building voicebot solutions using popular devices like the Amazon Echo or the Google Home is to visualize content. When having a conversation with your favourite voice assistant there are plenty of reasons why you want to present and visualize content, very obvious for online shopping and product selection for example. Although the Amazon Echo Show does has a screen built in, all other devices lack this feature. So why not make this possible using a Google Home and Chromecast device.

Somewhere in July 2013 Google released the first generation Google Chromecast. It was, and still is, the most popular gadget for home entertainment and directly competes with the (much) more expensive Apple TV. Google made sure the Chromecast was cheap and user-friendly, making it an entry-level streaming device for the masses. By doing this, Google opened up a completely new world to developers for streaming content from within (mobile) applications.

Is it possible to display custom content on a TV, based on the dialogue you are having with Google Assistant? Let’s find out!

Could it be possible?

You may already guess the answer to this question by looking at the title of this blog post. Unfortunately, at the time of writing this blog (December 2017), Google does not allow developers to stream custom content from the Google Home to Chromecast natively, yet.

During the 2017 Google I/O conference, Google announced the so-called Visual Responses. This feature enables you to ask the Google Home things like “Show me nearby restaurants on the TV” and see the result displayed on your television using a connected Chromecast. Google was not specific about the release date, but they did mention the feature will not arrive for a while and will be limited at launch, as mentioned in this article by The Verge.

So, what’s next? How do we achieve the same result by using a more do-it-yourself approach and getting around this Chromecast barrier? Simple, just build your own Chromecast alternative using a Raspberry Pi!

Proof of concept

By building our own Chromecast alternative using a Raspberry Pi, we have full control over the content that will be displayed on the TV. By using a combination of Dialogflow, Google Cloud Functions and Firebase we managed to create a network independent alternative, yet similar, Chromecast application that operates in the cloud.

The animation below demonstrates the reveal of the selected location on the TV screen. This occurs the moment in which a specific location is mentioned in the dialogue between the user and Google Home. This additional layer on top of conventional interfaces can help airlines and travel agencies create a better user experience.

GoogleHomeToChromeCast - example1

Well, how does it work? Dialogflow is an online platform for building conversational interfaces and supports many integrations such as Facebook Messenger, Telegram, Google Assistant and much more. By triggering a webhook function stored in Google Cloud from Dialogflow, the cloud function receives all needed parameters (location and content type) given through Dialogflow. The cloud function will ask the Firebase database for specific content based on the given location and content type (video or photo) and will display the content on a website hosted by Firebase. The only task of the Raspberry Pi is to display the website in kiosk mode and wake up when any value in the Firebase database has changed. The following diagram shows the current architecture in a simple manner.

architecture alt google home - chromecast

The above is just a temporary workaround until Google releases its Visual Responses feature. But if you are just like us, and you do not have the patience to wait for Google, you can use this workaround to play around with some ideas and build some proofs of concept yourself!


Cognitive Services Hacking Innovation Raspberry Pi