blog/markdown/Build-your-customizable-voice-assistant-with-Platypush.md

My dream of a piece of software that you could simply talk to and get things done started more than 10 years ago, when I was still a young M.Sc student who imagined getting common tasks done on my computer through the same kind of natural interaction you see between Dave and HAL 9000 in 2001: A Space Odyssey. Together with a friend I developed Voxifera way back in 2008. Although the software worked well enough for basic tasks, as long as it was always me to provide the voice commands and as long as the list of custom voice commands was below 10 items, Google and Amazon in the latest years have gone way beyond what an M.Sc student alone could do with fast-Fourier transforms and Markov models.

When years later I started building Platypush, I still dreamed of the same voice interface, leveraging the new technologies, while not being caged by the interactions natively provided by those commercial assistants. My goal was still to talk to my assistant and get it to do whatever I wanted to, regardless of the skills/integrations supported by the product, regardless of whichever answer its AI was intended to provide for that phrase. And, most of all, my goal was to have all the business logic of the actions to run on my own device(s), not on someone else’s cloud. I feel like by now that goal has been mostly accomplished (assistant technology with 100% flexibility when it comes to phrase patterns and custom actions), and today I’d like to show you how to set up your own Google Assistant on steroids as well with a Raspberry Pi, microphone and Platypush. I’ll also show how to run your custom hotword detection models through the Snowboy integration, for those who wish greater flexibility when it comes to how to summon your digital butler besides the boring “Ok Google” formula, or those who aren’t that happy with the idea of having Google to constantly listen to everything that is said in the room. For those who are unfamiliar with Platypush, I suggest reading my previous article on what it is, what it can do, why I built it and how to get started with it.

Context and expectations

First, a bit of context around the current state of the assistant integration (and the state of the available assistant APIs/SDKs in general).

My initial goal was to have a voice assistant that could:

1. Continuously listen through an audio device for a specific audio pattern or phrase and process the subsequent voice requests.

2. Support multiple models for the hotword, so that multiple phrases could be used to trigger a request process, and optionally one could even associate a different assistant language to each hotword.

3. Support conversation start/end actions even without hotword detection — something like “start listening when I press a button or when I get close to a distance sensor”.

4. Provide the possibility to configure a list of custom phrases or patterns (ideally through regular expressions) that, when matched, would run a custom pre-configured task or list of tasks on the executing device, or on any device connected through it.

5. If a phrase doesn’t match any of those pre-configured patterns, then the assistant would go on and process the request in the default way (e.g. rely on Google’s “how’s the weather?” or “what’s on my calendar?” standard response).

Basically, I needed an assistant SDK or API that could be easily wrapped into a library or tiny module, a module that could listen for hotwords, start/stop conversations programmatically, and return the detected phrase directly back to my business logic if any speech was recognized.

I eventually decided to develop the integration with the Google Assistant and ignore Alexa because:

• Alexa’s original sample app for developers was a relatively heavy piece of software that relied on a Java backend and a Node.js web service.

• In the meantime Amazon has pulled the plug off that original project.

• The sample app has been replaced by the Amazon AVS (Alexa Voice Service), which is a C++ service mostly aimed to commercial applications and doesn’t provide a decent quickstart for custom Python integrations.

• There are few Python examples for the Alexa SDK, but they focus on how to develop a skill. I’m not interested in building a skill that runs on Amazon’s servers — I’m interested in detecting hotwords and raw speech on any device, and the SDK should let me do whatever I want with that.

I eventually opted for the Google Assistant library, but that has recently been deprecated with short notice, and there’s an ongoing discussion of which will be the future alternatives. However, the voice integration with Platypush still works, and whichever new SDK/API Google will release in the near future I’ll make sure that it’ll still be supported. The two options currently provided are:

• If you’re running Platypush on an x86/x86_64 machine or on a Raspberry Pi earlier than the model 4 (except for the Raspberry Pi Zero, since it’s based on ARM6 and the Assistant library wasn’t compiled it for it), you can still use the assistant library — even though it’s not guaranteed to work against future builds of the libc, given the deprecated status of the library.

• Otherwise, you can use the Snowboy integration for hotword detection together with Platypush’ s wrapper around the Google push-to-talk sample for conversation support.

In this article we’ll see how to get started with both the configurations.

Installation and configuration

First things first: in order to get your assistant working you’ll need:

• An x86/x86_64/ARM device/OS compatible with Platypush and either the Google Assistant library or Snowboy (tested on most of the Raspberry Pi models, Banana Pis and Odroid, and on ASUS Tinkerboard).

• A microphone. Literally any Linux-compatible microphone would work.

I’ll also assume that you have already installed Platypush on your device — the instructions are provided on the Github page, on the wiki and in my previous article.

Follow these steps to get the assistant running:

• Install the required dependencies:

# To run the Google Assistant hotword service + speech detection
# (it won't work on RaspberryPi Zero and arm6 architecture)
[sudo] pip install 'platypush[google-assistant-legacy]'

# To run the just the Google Assistant speech detection and use
# Snowboy for hotword detection
[sudo] pip install 'platypush[google-assistant]'

• Follow these steps to create and configure a new project in the Google Console and download the required credentials files.

• Generate your user’s credentials file for the assistant to connect it to your account:

export CREDENTIALS_FILE=~/.config/google-oauthlib-tool/credentials.json

google-oauthlib-tool --scope https://www.googleapis.com/auth/assistant-sdk-prototype \
      --scope https://www.googleapis.com/auth/gcm \
      --save --headless --client-secrets $CREDENTIALS_FILE

• Open the prompted URL in your browser, log in with your Google account if needed and then enter the prompted authorization code in the terminal.

The above steps are common both for the Assistant library and the Snowboy+push-to-talk configurations. Let’s now tackle how to get things working with the Assistant library, provided that it still works on your device.

Google Assistant library

• Enable the Google Assistant backend (to listen to the hotword) and plugin (to programmatically start/stop conversations in your custom actions) in your Platypush configuration file (by default ~/.config/platypush/config.yaml):

backend.assistant.google:
    enabled: True
    
assistant.google:
    enabled: True

• Refer to the official documentation to check the additional initialization parameters and actions provided by the assistant backend and plugin.

• Restart Platypush and keep an eye on the output to check that everything is alright. Oh, and also double check that your microphone is not muted.

• Just say “OK Google” or “Hey Google”. The basic assistant should work out of the box.

Snowboy + Google Assistant library

Follow the steps in the next section if the Assistant library doesn’t work on your device (in most of the cases you’ll see a segmentation fault if you try to import it caused by a mismatching libc version), or if you want more options when it comes to supported hotwords, and/or you don’t like the idea of having Google to constantly listen all of your conversation to detect when you say the hotword.

# Install the Snowboy dependencies
[sudo] pip install 'platypush[hotword]'

• Go to the Snowboy home page, register/login and then select the hotword model(s) you like. You’ll notice that before downloading a model you’ll be asked to provide three voice sample of yours saying the hotword — a good idea to keep voice models free while getting everyone to improve them.

• Configure the Snowboy backend and the Google push-to-talk plugin in your Platypush configuration. Example:

backend.assistant.snowboy:
    audio_gain: 1.0
    models:
        computer:
            voice_model_file: ~/path/models/computer.umdl
            assistant_plugin: assistant.google.pushtotalk
            assistant_language: it-IT
            detect_sound: ~/path/sounds/sound1.wav
            sensitivity: 0.45
            
          ok_google:
            voice_model_file: ~/path/models/OK Google.pmdl
            assistant_plugin: assistant.google.pushtotalk
            assistant_language: en-US
            detect_sound: ~/path/sounds/sound2.wav
            sensitivity: 0.42
            
assistant.google.pushtotalk:
    language: en-US

A few words about the configuration tweaks:

• Tweak audio_gain to adjust the gain of your microphone (1.0 for a 100% gain).

• model will contain a key-value list of the voice models that you want to use.

• For each model you’ll have to specify its voice_model_file (downloaded from the Snowboy website), which assistant_plugin will be used (assistant.google.pushtotalk in this case), the assistant_language code, i.e. the selected language for the assistant conversation when that hotword is detected (default: en-US), an optional detect_sound, a WAV file that will be played when a conversation starts, and the sensitivity of that model, between 0 and 1 — with 0 meaning no sensitivity and 1 very high sensitivity (tweak it to your own needs, but be aware that a value higher than 0.5 might trigger more false positives).

• The assistant.google.pushtotalk plugin configuration only requires the default assistant language to be used.

Refer to the official documentation for extra initialization parameters and methods provided by the Snowboy backend and the push-to-talk plugin.

Restart Platypush and check the logs for any errors, then say your hotword. If everything went well, an assistant conversation will be started when the hotword is detected.

Create custom events on speech detected

So now that you’ve got the basic features of the assistant up and running, it’s time to customize the configuration and leverage the versatility of Platypush to get your assistant to run whatever you like through when you say whichever phrase you like. You can create event hooks for any of the events triggered by the assistant — among those, SpeechRecognizedEvent, ConversationStartEvent, HotwordDetectedEvent, TimerEndEvent etc., and those hooks can run anything that has a Platypush plugin. Let’s see an example to turn on your Philips Hue lights when you say “turn on the lights”:

event.hook.AssistantTurnLightsOn:
    if:
        type: platypush.message.event.assistant.SpeechRecognizedEvent
        phrase: "turn on (the)? lights?"
    then:
        - action: light.hue.on

You’ll also notice that the answer of the assistant is suppressed if the detected phrase matches an existing rule, but if you still want the assistant to speak a custom phrase you can use the tts or tts.google plugins:

event.hook.AssistantTurnOnLightsAnimation:
    if:
        type: platypush.message.event.assistant.SpeechRecognizedEvent
        phrase: "turn on (the)? animation"
    then:
        - action: light.hue.animate
          args:
            animation: color_transition
            transition_seconds: 0.25

        - action: tts.say
          args:
            text: Enjoy the light show

You can also programmatically start a conversation without using the hotword to trigger the assistant. For example, this is a rule that triggers the assistant whenever you press a Flic button:

event.hook.FlicButtonStartConversation:
    if:
        type: platypush.message.event.button.flic.FlicButtonEvent
        btn_addr: 00:11:22:33:44:55
        sequence:
            - ShortPressEvent
    then:
        - action: assistant.google.start_conversation
        # or:
        # - action: assistant.google.pushtotalk.start_conversation

Additional win: if you have configured the HTTP backend and you have access to the web panel or the dashboard then you’ll notice that the status of the conversation will also appear on the web page as a modal dialog, where you’ll see when a hotword has been detected, the recognized speech and the transcript of the assistant response.

That’s all you need to know to customize your assistant — now you can for instance write rules that would blink your lights when an assistant timer ends, or programmatically play your favourite playlist on mpd/mopidy when you say a particular phrase, or handle a home made multi-room music setup with Snapcast+platypush through voice commands. As long as there’s a platypush plugin to do what you want to do, you can do it already.

Live demo

A TL;DR video with a practical example:

In this video:

• Using Google Assistant basic features ("how's the weather?") with the "OK Google" hotword (in English)

• Triggering a conversation in Italian when I say the "computer" hotword instead

• Support for custom responses through the Text-to-Speech plugin

• Control the music through custom hooks that leverage mopidy as a backend (and synchronize music with devices in other rooms through the Snapcast plugin)

• Trigger a conversation without hotword - in this case I defined a hook that starts a conversation when something approaches a distance sensor on my Raspberry

• Take pictures from a camera on another Raspberry and preview them on the screen through platypush' camera plugins, and send them to mobile devices through the Pushbullet or AutoRemote plugins

• All the conversations and responses are visually shown on the platypush web dashboard