Rhasspy apps with Hermes MQTT in AppDaemon

Ever since I have been thinking about the concept of Rhasspy apps, I thought it would be a good fit for AppDaemon. This is a multi-threaded Python environment that supports MQTT. That sounds a lot like a “skills server”, isn’t it? :wink:

The beauty of AppDaemon is that you just configure AppDaemon with the right MQTT settings (broker, port, username, password, …) and then the AppDaemon apps don’t have to bother anymore about these. This was one of my main concerns about the Snips skills system, where every app had to configure its MQTT broker and almost all of them hardcoded localhost:1883, which was a nightmare to maintain. This was one of the reasons why I developed the SnipsKit library to abstract this away from the apps.

AppDaemon apps can also access parameters in their configuration file (which is a YAML file). This would be interesting for Rhasspy apps too, because that’s a standardized way to configure your apps with for instance API keys and so on.

For these reasons, I think AppDaemon is a good base for a “skills server”. Of course you can develop such a system from scratch (and at least one community member has already done some nice work with hss-server), but there’s already this whole AppDaemon code base we can reuse for Rhasspy apps. The only downside is that all AppDaemon apps run in the same virtual environment or Docker container (I’m doing this in Docker), while the approach of hss-server and snips-skill-server isolates dependencies by running each app in their own virtual environment. (But there’s no reason why you couldn’t run multiple AppDaemon instances to isolate apps with incompatible dependencies.)

Now that I’m writing a book about home automation (almost finished!), I revisited the idea of AppDaemon apps for Rhasspy, because I cover both AppDaemon and Rhasspy in the book. For instance, here’s a basic app that tells you the current time by replying to the default GetTime intent:

"""Tell the time when someone asks for it. Uses Rhasspy's Hermes API over MQTT.

Copyright (C) 2020 Koen Vervloesem

License: MIT
"""
import json

import mqttapi as mqtt


MQTT_MSG = "MQTT_MESSAGE"
INTENT_GETTIME = "hermes/intent/GetTime"
TTS_SAY = "hermes/tts/say"
TIME_FORMAT = "%H %M"


class WhatsTheTime(mqtt.Mqtt):
    """Rhasspy app that tells the time."""

    def initialize(self):
        """Initialize the app by subscribing to the right intent."""
        self.set_namespace("mqtt")
        self.listen_event(self.on_time_request, event=MQTT_MSG, topic=INTENT_GETTIME)
        self.log("What's The Time app initialized")


    def on_time_request(self, event_name, data, kwargs):
        """Tell the time."""
        nlu_payload = json.loads(data["payload"])
        site_id = nlu_payload["siteId"]
        now = self.datetime().strftime(TIME_FORMAT)
        sentence = f"It's {now}"
        self.mqtt_publish(TTS_SAY, json.dumps({"text": sentence, "siteId": site_id}))

With some work on a helper library such as SnipsKit that I wrote for Snips apps I can condense this code to something much more simple. To give you an idea what I was thinking about, with SnipsKit you could write an app such as:

from snipskit.hermes.apps import HermesSnipsApp
from snipskit.hermes.decorators import intent

class SimpleSnipsApp(HermesSnipsApp):

    @intent('User:ExampleIntent')
    def example_intent(self, hermes, intent_message):
        hermes.publish_end_session(intent_message.session_id,
                                   "I received ExampleIntent")

if __name__ == "__main__":
    SimpleSnipsApp()

Is there interest in this approach? I can reuse a lot of the API design and code of SnipsKit to develop a similar helper library for Rhasspy apps, but thanks to AppDaemon I don’t have to bother anymore with the MQTT and configuration part.

8 Likes

I think that sounds like a great idea :slight_smile:
For myself I do not have any experience with AppDaemon, but it looks to a good backend

2 posts were split to a new topic: Secure architecture for Rhasspy apps

That sounds great actually. I was thinking about something like that myself. I have a few “skills” defined as Home Assistant intent_scripts (in YAML :cold_face: ). I’d like to convert all those YAMLs into AppDaemon apps. Some framework like what you suggest would be a great help.

EDIT: this way we’d use MQTT and bypass HASS intent API entirely. Is there some side effect to not involving HASS into intent handling? Maybe something we are not taking into account such as the conversation component?

I’ve never used the conversation component in Home Assistant, so I really don’t know :slight_smile: This example is just for the Hermes protocol, nothing more.

i was thinking about for the same need, so if there is a project, i’m interesting :slight_smile:

In the meantime I’ve been playing around with (a patched, some PRs pending [^1]) Rhasspy 2.5 and some AppDaemon apps.

I wrote that in a few hours so it still has some rough edges, but the whole concept is sound (the timer skill actually works :partying_face:). Very similar to what @koan was trying. It absolutely needs a framework :heart:

I also introduce the concept of “speak templates”, for randomly picking sentences in different variations when saying something (that’s why I don’t use dialogueContinue/end directly but I go through a template-selecting service in my own app). But that’s probably for a different topic and could be probably better designed.

[^1] Rhasspy 2.5 is still not ready for this, but I guess it’s normal since the developer started somewhat from scratch. It should be named version 3.0 eheh :rocket:

Nice work, @daniele_athome!

Because @maxbachmann has convinced me that we also need a more secure way to run Rhasspy apps using Docker or podman, I think we need to split this into two parts to prevent development fragmentation:

  • a library such as SnipsKit for Rhasspy/Hermes messages
  • frameworks for app execution, such as AppDaemon, Docker, …

It would be nice if this would make it possible to develop a Rhasspy app that with some minor modifications would work both as an AppDaemon app and in a Docker container or standalone.

I’m still thinking about how to do this, but I’m definitely interested in reviving the vision behind SnipsKit for Rhasspy.

5 Likes

Hey there, sorry for being a little absent.
I see you’ve been working on the library @koan, great stuff! I try to read everything from the forum topics I watch, but sometimes I can’t really keep up with all the answers. If there is something that you think we should discuss, please mention my username.

I finally found some time to code something for AppDaemon. As I said, the first step would be using the “AppDaemon way” (services and events). This is mostly done, at least for the basic components of Hermes (or more the ones I need most eheh): TTS and dialogue management. The plugin is here and the base class for apps is here. I’m accepting of course comments, requests, suggestions, etc.

I’m currently running my couple of skills based on the plugin on production and it’s been working great so far.
The next step would be (apart from implementing the whole Hermes protocol, but that would come slowly) to use the annotation-based approach, which would greatly simplify. I know I could have just started with that, but I wanted to base it on something AppDaemon devs already knows (so anyone could choose between the two approaches).

I’ve also started approaching folks at HASS to ask them if there will ever be a chance of integrating this plugin into mainstream code.

@koan what do you I think I should do to better align with your library? I mean I see there might be some roads leading, one day, to a possible “Rhasspy skill store” or something… those skills would probably be developed using rhasspy-hermes-app and that’s alright. Can we do something to let people switch easily, if they ever wanted to (in both directions)? I don’t know… naming things the same, maybe sharing some code (as in put it in a common dependency since we are both using Python), or something else. Please let me know what you think.

Impressive work, @daniele_athome!

For the decorator-based approach I wanted to make sure that the app developer doesn’t have to know about session IDs. So you can just end the decorated intent handler with return app.EndSession("Session ended") or return app.ContinueSession("Session continued") instead of having to pass around the intent’s session ID. Something like this in your decorator-based approach would be nice.

For the rest, maximally reusing the Rhasspy Hermes library is a key factor to align both libraries, and you’re already doing that. Maybe naming the decorators the same and using the same name and order of decorator arguments would be nice. I don’t know what else could be done. I don’t see the possibility of common code yet, but maybe that will be clearer when I see your decorator code.