Helper library to develop Rhasspy apps in Python

After the discussion about Rhasspy apps with Hermes MQTT in AppDaemon I did a first proof of concept of an ‘app library’ in Python. This makes it possible to write the toy app from that discussion as a standalone Python script like this (with the right import statements of course):

class WhatsTheTime(RhasspyHermesMQTTApp):

    @Intent("GetTime")
    def get_time(self, intent):
        now = datetime.now().strftime("%H %M")
        self.say(f"It's {now}", intent.site_id)


if __name__ == "__main__":
    WhatsTheTime()

And as an AppDaemon app like this:

class WhatsTheTime(RhasspyHermesAppDaemonApp):

    @Intent("GetTime")
    def get_time(self, intent):
        now = self.datetime().strftime("%H %M")
        self.say(f"It's {now}", intent.site_id)

I hate boilerplate code, so this is a big improvement in usability for app developers.

To make this usable, I should implement decorators (like the @Intent) for every Hermes MQTT message an app can react to, and methods (like the say) for every Hermes MQTT message an app can send (for instance I would have to end the current session in this example instead of just using the tts/say MQTT topic that the say method emits). This is quite doable because the Rhasspy Hermes library does all the heavy lifting.

Any comments about the API design or specific requirements before I start working on a first version?

One issue I’m not sure yet how to deal with is how to make it possible to reuse the same code to run as a standalone Python app (optionally in a Docker container) or as an AppDaemon app. Right now I have defined base classes for the first and second situation, and in the second situation the base app (RhasspyHermesAppDaemonApp) inherits from AppDaemon’s mqtt.Mqtt class. As you see from the two examples above, the standalone and AppDaemon versions are almost identical. You only inherit from a different base class, and in the AppDaemon case you use another function for the datetime because that’s what AppDaemon recommends. You also have access to all AppDaemon’s state in the latter case (not used in this example). But in most cases the method body would be exactly the same in both situations if you’re not using any functionality specific to the execution environment (for instance subscribing to non-Hermes MQTT messages or using AppDaemon’s internal state or methods).

I don’t think it’s possible to just write an app and decide at runtime whether you want to run it in AppDaemon or standalone. But as far as I can see now, it should be possible to write an API that is almost exactly the same for app developers on AppDaemon or standalone Python, they just have to choose a base class.

I’m also not sure yet how to publish the library practically: one library where you just import the right base class for your execution environment? But then you only use half of the library. So maybe make it two libraries, one for AppDaemon and one for standalone Python, but with (almost) exactly the same API? That seems more difficult to maintain, but is probably nicer for the app developer.

11 Likes

Thanks @koan for putting effort into this, it’s greatly appreciated.

I’ve updated my public repo with the latest changes to the AppDaemon apps which include things I discuss below.

Class inheritance

Could it inherit adapi.ADAPI? That’s the base class for mqtt.Mqtt. This way I can mix parent classes and use also, e.g., hassapi.Hass. I can’t mix mqtt.Mqtt and hassapi.Hass though because it caused some unexpected issues – can’t find the forum thread right now.

Separated libraries

I would say that despite it’s maintenance effort, having two different libraries can be beneficial. Think about libraries having dual APIs for JEE or Spring: they are designed and packed as different libraries, so it wouldn’t be the first time.
What we can however keep in common betwen the two libraries is business logic stuff (e.g. session management, more on that later). We’d have to define “communication factories” (e.g. AD call_service for mqtt vs. Python mqtt library such as hbmqtt) but I think with a little effort it can be done. Of course if you think it’s worth it, that is.
The alternative would be two completely separated implementations, but that would be even more costly IMHO.

Session management…

This is something I’ve bumped into lately while developing AppDaemon apps for Rhasspy. I’ll explain, please tell me if I’m making any sense or if it can be approached somehow differently.

One thing about apps is that when they are inside an estabilished session (a dialogue has started), the next steps might recognize intents that are commom among apps (e.g. “start a timer” -> “how long?” -> “40 minutes”). That “40 minutes” will be recognized as a generic intent (Rhasspy doesn’t support slot filling, but that could apply also to normal intents, not just slots, that are similar, e.g. a confirmation command) which could be captured by more than one app.
Also think about satellites. Sessions are per satellite. Multiple satellites, 1 app instance (I’m thinking in an AppDaemon way here).

In my apps I’ve introduced the concept of owning a session.
I’ve written a support app for interacting with the Rhasspy Dialogue Manager (using of course Hermes). This support app keeps the state of all sessions starting and ending.
A skill app, when the dialogue should continue after the first step, requests ownership to the dialogue support app. It then asks the dialogue app to continue the session (just a method call that triggers the MQTT publish, really).
On the next dialogue step, the app will ask the dialogue support app if it’s the owner of the session and continue handling the dialogue, either continuing or ending it.

This might be an overengineered solution, but I’d like to share my idea to see if (1) I’ve got session management right and (2) it can be improved in this new library we’re going to create.

If all that above makes sense: do we want session management in this framework? Session management doesn’t only involve session ownership, but also satellites – sessions are per satellite right?

…or not?

Let’s say we don’t want (or don’t need) session management by the framework:

  • The slot filling issue would be somehow solved by native slot filling support by Rhasspy (how? Anyway this will make Rhasspy handle slot management, completely excluding the skill app)
  • The “common intents” issue (e.g. saying “proceed” for confirmation) would be solved by having different intents for each skill app
  • The satellite issue would not be solved; every app should keep state of current sessions, one for each satellite they are interacting with

Intent configuration

My apps have configurable intent names. Using annotations would not make this possible. Although this wouldn’t be a big deal, it would make Rhasspy sentences.ini grow considerably for common utterances (e.g. confirmation command or similar common commands). I don’t know if that could be a big deal either, but I’ll just put it on the plate for discussion :slight_smile: That being said, I love the annotation approach from a developer point of view.

Sessions by satellite is an assumption as a satellite may be a group of purely capture points say several in a room as distributed mic / speakers.
For many operations its logical for those satellites to have a singular session for that collection.

But yeah intent is the only requirement for Rhasspy and intent sessions should be able to be passed to an intent processor but as far as I am concerned thats a completely seperate project to the core voice AI that should be rhasspy.

Intent is the helper library.

@koan I really like the minimalist approach using decorators. :+1::sunglasses:

I think sessions are not per satellite, they are per siteId.

Session management is the way to go to avoid issues that arise when using low level topics directly.

Each app/skill must provide the necessary intents to work so if a pseudo generic intent is needed like confirmation « yes », the skill should provide its own confirmation intent and do a continueSession with an intentFilter for this specific intent.

I don’t think that Rhasspy should provide any predefined intents or gazetteer slots. That’s the job of the skill and introducing dependencies between skills should be avoided. Maybe these « shared » intents/slots can be provided by the app/skill manager instead… I wonder…

Though I do think that Rhasspy should provide builtin slots/entities for grammar based stuffs like dates, durations, numbers, etc. (as listing all the possible values is impossible) as parsing these is the job of the NLU component.

My 2c :wink:

Sorry, I confused them. I meant sites, not satellites.

Session managament is needed anyway, by it could be assisted by the framework. The main use I’m thinking of is keeping session state. The framework could keep session state per-skill and pass it along with the intent:

# made up code here :)
@intent('HelloIntent')
def handle_hello(self, intent, session_state):
  # do something

The skill can then have full read-write access to session_state at will (and data will be preserved by the framework). The less the skill knows about sessionId and everything, the better.

Ok going this way we solve the issues of slot filling and common intents (by using skill-specific intents). Mine was a dangeours path anyway :slight_smile: maybe Rhasspy could use intent aliases… :sunglasses:

I wish :slight_smile: I’m currently using Rasa NLU with Duckling: it gets the job done for the most part. But I had to patch Rhasspy.

We had a discussion about it earlier: Parsing builtin slot values

I agree about this. But what is this ‘session state’? Shouldn’t the skill just know how to end or continue the current session? That could be delivered by methods of the base class for the skills.

With some tricks it is possible. I did this in SnipsKit to create translatable intent names, see an example.

Isn’t this session_state the same thing as customData In the dialogue manager session ? Does it provides a way to abstract the serialization/deserialization of this value (as it is supposed to be a string) and it’s propagation in dialogue management topics?

Yes that’s the purpose of customData. And an app can find this in the intent object (see the NluIntent class): https://github.com/rhasspy/rhasspy-hermes/blob/master/rhasspyhermes/nlu.py

1 Like

I would actually drop that based on @fastjack answer, if it’s ok for you. One less problem to worry about. Using specific intents per-skill and intent “namespaces” would be enough (e.g. daniele-athome:myskillapp:myintent).

The problem is that only the dialogue manager knows about customData. So it’s not being passed with the Hermes message that actually counts in this case, which is hermes/intent/<intent_name>. That’s because the NluIntent message is sent by the NLU module and not the Dialogue Manager. More details in the official Hermes documentation.
Anyway, as I understand it, fixing that in Rhasspy would allow customData to be passed to NluIntents too, making my “session state” idea obsolete (btw I’ve already started working on that, but it involves multiple modules I’m afraid).

1 Like

Snips worked like that, so for snips users who have already process with customData (like me :smiley:) , it would be easier to migrate.
But it is not a reason to not ask ourselves if there is no better methods to do that :slight_smile:

Ced

Now I think it’s the best method. I just didn’t connected all the dots before – I was missing some information.

I have never used the customData yet because all my apps had a simple question - answer dialogue, but I agree: the changes you propose could make this work in Rhasspy the way that Snips worked. Thanks for taking the time to look into this!

So, I haven’t forgotten about this, just busy with some other stuff. Yesterday I started to convert my proof of concept code into a repository I can publish, but this also made me think more about the design of the API. I remember when I was creating SnipsKit one of the Snips developers suggested a Flask-like API to me.

This would mean that the standalone version of the example app that I showed in the beginning of this post would look something like this:

from rhasspyhermes_app import StandaloneApp

app = StandaloneApp("TimeApp")

@app.intent("GetTime")
def get_time(intent):
    now = datetime.now().strftime("%H %M")
    app.say(f"It's {now}", intent.site_id)

if __name__ == '__main__':
    app.run()

Such a Flask-type API is more flat with no inheritance. You don’t need to define a class. A lot of Python web developers are familiar with it. I see the beauty in this approach, and I actually rewrote part of SnipsKit in this style before I stopped using Snips, but I never actually wrote voice apps this way, so I don’t know whether it’s a good choice.

An important reason to not choose this approach is that AppDaemon apps are class-based: they should subclass the mqtt.Mqtt class for MQTT apps and AppDaemon creates an object of this class and initializes it. So choosing this approach for standalone Rhasspy apps breaks API similarity with Rhasspy apps for AppDaemon. So I’m inclined to just implement this library with the class-based approach.

I’m curious what others think, though.

2 Likes

I think too that it’s not a good idea to break API similarity. But i like the Flask-like style :slight_smile:

Ced

I really like the Flask style implementation and would prefere it over the class based one just for simplicity. Another Pro-Flask point is that this kind of minimalistic API is easier to understand and implement. it just takes less code…

i understand however the more linear approach using classes. Would using this make it easier to adapt to changes at AppDaemon in future?

Well, I do like the Flask-style API too, it’s easier to reason about, especially for simple apps. And it saves an indentation level :slight_smile: it’s just that using the class-based approach for both types of apps would make the standalone and AppDaemon versions of the API as good as identical. If I’m using the Flask-style approach for standalone apps and the class-based approach for AppDaemon apps, the APIs would still be very similar (just compare the Flask-style example to the first AppDaemon example in this post), but if you have been writing standalone apps in the Flask-style API and then for some reason switch to developing for AppDaemon (or the other way around), you suddenly have to change your programming style a bit.

This may seem like hair-splitting, but I want to start from a clean base now so I don’t have to rewrite the fundamentals of the API later. So I want to look at all the options now and hear other opinions.

@daniele_athome you’re probably the one with most experience in AppDaemon from the participants in this discussion, what do you think about it?

1 Like

Can you use the alternative AppDaemon implementation approach outlined here? Using this approach based on ADBase might allow you to build something Flask-like as you are proposing. You would still need to sub-class ADBase for your app, but the MQTT or HASS plugins can then be used via “accessors” without resorting to sub-classing.

Nice find! I forgot about this alternative. But those plugin objects are still defined inside the class, no? And AppDaemon only initializes an app when it finds a subclass of ad.ADBase defined in a file. As far as I know, other code in the file outside the class definition is not executed by AppDaemon. How should the code for the AppDaemon time app look like with this approach, according to you?

Errr, is there anything that needs to change? I guess the final stanza calling app.run() is unnecessary as there is no “main” for AppDaemon apps. The meat of the question is what does StandaloneApp do behind the scenes to “run” your Rhasspy app in an AppDaemon app and hide the details?

For example, how do named (Rhasspy) apps defined with StandaloneApp be made known to AppDaemon? You still have to “configure” the AppDaemon app correct? Would the named app (e.g. “TimeApp”) map to an AppDaemon app by that name that would have to be configured via yaml in the standard AppDaemon manner? Does that mean that StandaloneApp has to generate the AppDaemon class corresponding to the named app on the fly? I presume this is possible in Python, but this starts to sound complicated. It also seems like with this flat structure, one could run into (function) namespace collisions that would be less likely in a class based approach given that unlike simple Python scripts, multiple apps live in the same Python environment with AppDaemon. Might this result in obscure issues with user contributed apps intended to run in AppDaemon?

What is it that one expects to gain in a Flask-like API versus the traditional AppDaemon approach? The two simple examples so far look pretty similar sans the class declaration. I’ve not used Flask so maybe others can be more detailed in why they like that style? Are we really gaining anything by hiding AppDaemon or just making things obscure? Your original class-based approach does seem reasonable. It can easily take advantage of the existing AppDaemon infrastructure without resorting to lots of hair to hide things. It does feel like a Flask-like structure is sort of an impedance mismatch with AppDaemon.

Sorry, seems like I have more questions than answers!

I would say that using a class-based approach or a function-based approach (both using annotations) wouldn’t really be a huge difference since an app (either AppDaemon app or standalone app) could just instantiate the “app” object and use methods provided by it to speak/listen to Hermes (IOC can be achieved for both ways with a little more effort). But there is one problem: what will the framework use for communicating with the external world (MQTT/Hermes)?

A standalone app would need some connection infrastructure (by leveraging an event loop for example), a configuration file for connection parameters, and so on.
An AppDaemon app can leverage a running application server, so no need for anything.

This difference alone forces us to create two different entry points.

Personally I’d try to leverage existing framework(s) – I would not reinvent the wheel by writing another framework from scratch (i.e. if we want to use a “Flask approach”, we should use Flask for real) – that is, if that was your idea @koan and if I understood it correctly.

I also think that creating a framework that can be used in multiple environments (that is, that can be used interchangeably with another framework or application server without touching the app code) brings more complexity and effort than benefits. Think about the compromises we’ll have to make, the complexity we’ll have to handle to make the framework “behave well” in every environment. Also think about the conventions will have to impose on the app’s creator that wants to use e.g. Flask stuff or AppDaemon stuff correctly.

In conclusion IMHO this library should focus on integrating with 1 framework/application server/whatever environment it runs in, since the real “interface” for this library is the Hermes protocol.
As much as I am personally interested in using AppDaemon – as a Home Assistant user I find it extremely convenient – any framework/environment will do. If someone wants to use another framework/environment it will require a new library.

However…

Beware: brainstorming stuff that came out of my mind almost as it was. I was tempted to not include this paragraph, but I’ll leave it here for whatever discussion it might generate :slight_smile:

If we really want to make a “general purpose” library, we could implement something like a core module that handles Hermes business logic, and then implement adapters that will make the library plug into other frameworks/application servers/environment. Those adapters would be in charge of publishing/subscribing to MQTT using the framework/environment they are supposed to interact with. An example workflow would be (I might be talking in AppDaemon terms, but I believe the same concepts can be applied to any framework/environment):

  1. The app would register to events using ways and methods provided by the library
  2. The library will use the adapter to implement that “listen to events” (the adapter will provide an interface for doing that) (*1)
  3. The adapter would listen to events by using the means provided by the enviroment
  4. The adapter will receive calls (service requests from the app or events from MQTT) that will be handled using business logic in the core module
  5. The core module will return something to the adapter (or signal it some how) that it handled the call
  6. The adapter will return the result to the app

A similar approach could be used for service requests (not event-based, think about service calls).

(*1) this part is where the library would scan for annotations and subscribes to topics accordingly, without actually knowing how, because the adapter would take care of it.

This seems an overengineered and overly complex design to me. Also I might have missed something because this is the result of a 30 minutes brainstorming. Or I might just be a bad designer :slight_smile: but to me the simple fact that it seems to be very complex makes me think that maybe it’s not worth the effort (in relation to the benefits).

In my next iteration of the proof of concept of the standalone app I’m now subclassing the HermesClient class and using the cli module, both from the Rhasspy Hermes library, which gives me all this functionality for free.

No that was not my intention. We don’t need HTTP endpoints and all this web stuff in the apps, we are using MQTT. I was just referring to the style of the API: creating a Flask object, attaching decorators of this object to functions and then running the object. This in contrast to subclassing a class of the API, attaching decorators to methods of this class and then creating an object of this subclass. As you already said, it’s not a huge difference in practice, but I feel that for casual developers this style is easier to reason about.

After thinking about this for a while, I’m also sure now that trying to create a framework that can be used both standalone and in AppDaemon will be too complex with too many compromises. It would be like driving a square peg in a round hole.

The “brainstorming stuff” that you wrote is exactly what I came up with, but it gave me nightmares about the time when I was designing bloated Java programs with factories and visitors and strategies and so on :slight_smile:

So for now I’m focusing on a library for standalone apps, because this is also the way to go if we want to deliver Rhasspy apps as Docker containers for better security. As you say, the real interface is the Hermes protocol. I see this library really as a “helper” library, a wrapper around the Rhasspy Hermes library to eliminate as much boilerplate code as possible and let developers create simple apps in a few lines of code.

4 Likes

Damn for a moment there I hoped you would go for AppDaemon eheh :smiley:
I feel like an AppDaemon plugin for Hermes is really missing, I’d like to implement it anyway. After our POCs will be out (actually if you can share what you have even if incomplete would be great), we can discuss using similar approaches so developers may easily switch from one to the other (such as naming convention stuff, e.g. name of the annotations, anything that can be shared really). I believe this could benefit everyone especially because we’ll have two different point of views to compare and work. What do you think?

Thanks for your work on this.

I’m now in my third iteration of the proof of concept, each time I had to start from scratch. First class-based, then Flask-like, now Flask-like using HermesClient and cli. I think this time it will result in a good foundation for the library. I hope this is ready and tested in a few days (sadly not much free time these days to program) and then I’ll publish it. Don’t expect too much of it yet, it’s just one or two Hermes messages wrapped to prove the approach is viable.

After that, It’s a good idea indeed to discuss some conventions so the APIs can share as much as possible in their naming, philosophy, …

By the way, do you mean “AppDaemon plugin for Hermes” or “Hermes plugin for AppDaemon”? I didn’t want to suggest the latter to you in my previous message because I didn’t know whether you wanted to implement this, but I actually think this could be a good approach to develop Rhasspy/Hermes functionality for AppDaemon: a dedicated Hermes plugin (like the HASS and MQTT plugins) to hide the MQTT details from Rhasspy app developers.

I meant Hermes plugin for AppDaemon, possibly by extending the already existing MQTT plugin (or by depending on it, I haven’t decided nor investigated yet).

I’m not sure whether to use an AppDaemon approach (i.e. listen_event, so callback-based) or an annotation approach. The latter would be easier to use (I would have preferred that AppDaemon itself was designed that way, but that’s another story) and more aligned with your design, but it would diverge from how AppDaemon recommends its apps should be implemented (EDIT: actually I could just do both, it doesn’t require that much more of an effort).

Ok, then we’re talking about the same, I just call it the other way around :slight_smile:

Yes, I realize it now. I’m sorry, English is not my mother tongue, sometimes I get lost in these kind of particulars.

So I’m finally happy with the API and I just published the first prototype: rhasspy-hermes-app.

For now this only supports the intent decorator to let a function react to an intent. In this function, you should return an EndSession object with a text string to end the current session. I did this so you shouldn’t have to refer to the intent’s session ID to end the session, because almost always you just want to end the current session and you don’t want to know anything about session IDs.

There’s much more to do of course to make this a usable library (but you can use it, the example app in the repository is already a little useful app). I have created an initial TODO list at the end of the README. Have a look, and I’d love to hear anyone’s opinion. Criticism of the current approach, wanted features, … just let me know here or open an issue in the repository.

5 Likes

Nice work! Thanks :slight_smile:

Ok, that makes sense. Also, since we said that state would be kept in custom data (any state? All of it?), all the app would need to do is just read custom data from the intent object.

I noticed this in the TODO list:

Let the app load its intents/slots/… from a file and re-train Rhasspy on installation/startup of the app.

How should this happen? Via Rhasspy API right? Like a service for requesting a new sentences file dedicated to the app. Maybe some utilities inside the library would be better for this, i.e. avoid the app to let it use Rhasspy client functions directly, mainly to avoid conflicts between apps, I was thinking of something like a “namespace” concept for apps (that would ultimately end up in the filename, e.g. namespace_appname_sentences.ini - ugly I know, but I think you get the idea).


I’ve begun experimenting with the AppDaemon plugin. I’m going to publish it to my public repository soon so we’ll compare notes. I’m using a classic AppDaemon approach for now (events/services), I’ll extend it to annotations like yours later. Maybe I should open a new topic for that :slight_smile:

Indeed. I will also implement a ContinueSession object and test a session that forwards custom data in a flow of a startSession, continueSession and endSession message.

The REST API has a /api/sentences URL which you can POST sentences to, but I don’t believe this is possible yet with the Hermes protocol. I opened an issue. @synesthesiam what do you think of this?

A namespace for sentences.ini files is a good idea. Probably also for intent names.

I’m looking forward to it!

Do we really want to put configuration APIs (not directly related to a pure messaging function) to MQTT? I mean wouldn’t it better to just let the app contact the HTTP API (via utility functions provided by the library maybe)?
I’m aware this will complicate things (the app would need HTTP credentials, for a start), but we’ll be polluting MQTT with configuration services. I don’t know, it doesn’t sound right… what do you think?

I see your point, but requiring apps or a library to use both HTTP and MQTT seems more convoluted, and is definitely more error-prone. And if the HTTP API offers this functionality, I don’t see why the MQTT API cannot offer this too.

I consider every interaction with Rhasspy as a messaging function, also configuring intents, I don’t see it as polluting. But that’s maybe because I’m much more comfortable with MQTT than with HTTP :slight_smile: This is not to say that every single aspect of Rhasspy should be configurable using MQTT messages, but adding intents/sentences seems like a common enough use case.

By the way, I have edited the issue I raised in the rhasspy-hermes repository and added some thoughts about how to handle namespace, because that’s important if this will be implemented in the MQTT API. Maybe you have some thoughts about it too.

1 Like

How about doing that during an earlier stage? Something like the setup stage. I mean configuring and training should happen only when something is installed or changed right? Normal app execution shouldn’t need this. We’ll let the build/setup infrastructure do this instead of doing it from inside the app code (I still have to think how, but I think you get the idea; it would have to be done outside the possible Docker container of the app, of course).

My main concern is exposing the MQTT system to potentially privileged operations. Privileged as in modifying configuration stuff. I understand some MQTT brokers implement ACLs and other authorization mechanisms, but this should concern Rhasspy itself. Rhasspy will have (one day I hope) an authorization layer for its HTTP API. It won’t be as easy or possible to do the same thing with MQTT.

Anyway, it’s not a big deal in the end (if I don’t want it, I would just ban the topic in mosquitto and do the training manually :smiley: or maybe I’m just a paranoid lol), but I just thought it would need proper attention before moving on.

Yes, that’s how Snips did it with the snips-skill-server. But then we need some rhasspy-skill-server that gets the sentences.ini file from an app you install, and this server has to communicate the content of this file to the NLU and ASR services, which potentially run on another machine. So then you’re back to MQTT or HTTP :slight_smile:

Privileged operations is also what I was talking about in my additions to the issue I linked to above. With an ACL and authentication this is quite easy to contain. Actually I have been running my example app this way for the past few weeks: it can only subscribe to one specific MQTT topic and publish on one other MQTT topic with this ACL file:

user rhasspy-app-time
topic read hermes/intent/GetTime
topic write hermes/dialogueManager/endSession

Ok course. It would still be a remote call anyway. Besides the protocol used, I was talking about not letting the app code do this, instead do it from a privileged account upfront. But as I said maybe I’m just a little too paranoid :slight_smile:

1 Like

Hi @koan

As I would like to implement an intent to manage Google Assistant searches (in the spirit of what is described here), I’m very interested by your framework proposal.

Unfortunately, I must be doing something wrong because I can not even run the time_app demo.

python3 time_app.py
  File "time_app.py", line 14
    return app.EndSession(f"It's {now}")
                                      ^
SyntaxError: invalid syntax

Do you have any clue?
fx

What Python version are you running (python3 --version)? You need Python 3.6 to use f-strings, and the dependency rhasspy-hermes needs it too. I have added this requirement to the installation instructions in the README. Note that the next version of rhasspy-hermes will need Python 3.7.

If it’s the f-string your Python is complaining about, you can always try if it works when replacing that line by:

return app.EndSession("It's " + now)

Indeed, I had Python 3.5.3 (on Debian 9). I reinstalled with buster instead of stretch and now it’s working well :slight_smile:

By the way, what’s the way to capture the audio stream only (Google Assistant will do the ASR)?
Will you have a way for this in your framework?

Not yet, the current code is just a proof of concept (that’s also why I have no tests, documentation or a PyPI package yet), so for now you can only listen to intents, which is what probably 90% of the apps would use :slight_smile:

But the HermesApp class subclasses the HermesClient class from the Rhasspy Hermes library, so you can definitely use it to capture the audio stream. It’s the hermes/audioServer/<SITE_ID>/audioFrame topic you have to subscribe to.

If you want this feature, maybe open an issue with a short explanation of why you need it and how exactly you’d want to use it. We can discuss the specifics there.

For Rhasspy 2.5, I’ve added a rhasspy/asr/<SITE_ID>/<SESSION_ID>/audioCaptured message that lets you get a hold of the recorded WAV data from a voice command for a session :slight_smile:

2 Likes

Hi @synesthesiam

Thanks for the tip. Unfortunately, I have the feeling that rhasspy-dialogue-hermes doesn’t set the appropriate flag to True when handling the ContinueSession message

See the code below (using default value which defaults to False if I understand well the code)

        # Start ASR listening
        _LOGGER.debug("Listening for session %s", self.session.session_id)
        yield AsrStartListening(
            site_id=self.session.site_id, session_id=self.session.session_id
        )

Maybe we need an additionnal flag in CotinueSession to notify if we want to receive the audioCaptured message?

In the meantime, I guess that I have to go with audioFrame…

Finally I got something working with the following approach.

Wakeword -> say “Ask Google” (ASR/NLU) -> in the on_intent function, publish continueSession (text=“What do you want to ask?”) -> once ASR is started, detect and store audio frame until ASR stops -> store a wav file from the audio frames -> trigger Google Assistant with the input wav and get the response in wav format -> open the wav file and publish it to the site_id to get audio feedback.

It’s likely not the most optimal (and surely not the most beautiful) piece of code! but take it as a proof of concept :slight_smile:

The great thing is that @koan framework made the starting part very easy. Thanks a lot for the good job!
Do you plan to add additional decorators to handle other messages than /hernes/intent?

What I found not so easy is to figure out how to publish some messages like AudioPlayBytes. Maybe rhasspy-hermes should provide more app-level API (instead of having to call publish myself which is too “low-level” in my opinion)?

2 Likes

Nice!

Yes, I do. I don’t know if it makes sense to handle all of the message types this way, but definitely for the most common ones.

Yes that was also the reason why I added the EndSession class and decided to hide publishing the message in the decorator so you could just return such and object and it will be published. You can always open an issue in the repository with a proposal of how you would hide these low-level details for other types of messages. There’s still much to implement in rhasspy-hermes-app.

1 Like

@koan I played around with your API and created a simple Akinator (the guess a person by yes/no questions game) app using some Node api and a horrible way to use it from Python. (See https://github.com/DanielWe2/rhasspy-hermes-app/commit/d039fcc59c2ab44958f79c852f6da330afc7a232)

It was really simple using your API. I have some questions though:

  • Most of my intents do basically the same. Is there a way to have one handler for multiple intents and figure out the intent name in the handler?
  • How do I handle the case when the user answers something that doesn’t match the intents from intent_filter? Currently it breaks the game. I would like play some help message in that case.
  • The game uses very generic intents like “yes” . We would need a way to only enable them in Rhasspy once the game has started.

Personally I prefer many short handlers instead of one big handler with an if/elif/else block, but I can see your point that for the one-line handlers in your example the latter approach can be useful. So you want a handler that runs on all intents or only on a specific list of intents?

I can add a decorator for the intentNotRecognized message. You can then let a handler react to this with a help message.

I think I saw a discussion about this in the last few weeks, but I can’t remember where (GitHub or the forum here). Rhasspy should indeed have a way to configure a specific intent as disabled by default.

I personally use something like GetWeather* right now to catch all weather related intents. All intents seems like it wouldn’t be all that useful but a way to either use a wildcard or add multiple decorators (one for each intent) or a list of intents to catch would be good.

My idea was to put a dict mapping intent names to answer “ids” (for the external api) at the top of my script and one handler for all answer intents and have less duplication of intent names that way.

But I thought a little further and figured out that I could possible combine all possible answers into one intent with multiple values slot values.

But I second Daenaras suggestion for a prefix (or regex match) for selection of intents. (A second decorator like on_intent_by_regex or a second parameter for on_intent would be a possibility). Ideally also for the intent_filter in ContinueSession.

That would be great.

If the goal with this module is to provide a way to build self contained apps/skills for Rhasspy, possibly combined with a community repository ( integrated into Rhasspy?) to share those, intent/slot management becomes a important topic.

Points are:

  • Every app should be able to add intents/slots (also with multiple translations)
  • There needs to be a way to protect against collision of intent names (Just prefix the app name by default as a simple form of name spaces? By default an app can only handle it’s own intents?)
  • We have the intent filter to filter which intents to handle in a dialog session. But I think we also need the opposite: Have intents that only trigger when used in an intent filter. That would allow apps to use pretty common phrases like “yes”, “no” without any collision issues.
    • Something like “global” or “always on” intents and intents that only trigger when in a active session with an app.
  • The same is valid for slots
  • To give the user transparency and control it would be good if the Rhasspy UI would show intents installed by apps in a special menu grouped by apps. Maybe even allow to modify/disable them.

A way to integrate a configuration page for an app into the UI (also to disable it) would make it more user friendly and ties in with the whole configure by UI concept of Rhasspy.

I thought about writing simple Home Assistant app using your service and the home assistant api. But without adding intents/slots that’s not yet possible. How much work is need on the Rhasspy side to make that possible?

I thought about this point and some of your other points too, I opened an issue about this a week ago. Maybe you can chime in there with your remarks/ideas, because this feature definitely requires some changes in Rhasspy.

Great job :+1:
I developed a more or less similar solution, but want to give your solution a try. So I ported some of my work and it works so far.
In my case I need additional arguments to run my app, but the arguments parser isn’t accessible. The better solution could be adding the parser as an optional parameter like this:

def __init__(self, name: str, parser: argparse.ArgumentParser = None):
    """Initialize the Rhasspy Hermes app."""
    if parser is None:
        parser = argparse.ArgumentParser(prog=name)

With this I can add arguments before starting the app.
What do you think about?

1 Like

Yes, this was already in the back of my mind, this looks like a good solution. I added your change, thanks!

There seems to be enough interest in this library, I’ll see if I can publish a first package on PyPI one of these days, then it’s easier to use it in your projects.

2 Likes

An other change I would made, especially for “not native python speaker” like me, is to type the incoming intent in your example.

@app.on_intent("GetTime")
def get_time(intent: NluIntent):

So it’s easier for beginners to understand what kind of data comes in and what properties i have. Otherwise you must understand what your code is doing and where the data comes from.

Isn’t a must but very helpful.

1 Like

Good idea. At the moment I’m documenting the Rhasspy Hermes library, which defines all these classes. When the documentation is published, it will be clearer too what data Rhasspy Hermes App expects. Afterwards, I will work on documenting Rhasspy Hermes App.

1 Like

I forked your project and made same changes to subscribe raw topics. I need this for handle other events on my mqtt broker. I created a pull request, so feel free to comment, or change, or deny.

1 Like

Thanks! I’ll have a look tomorrow.

I think topic decorators should not include the topic itself as a string but rather be a specific decorator:

@on_topic(‘hermes/dialogueManager/sessionEnded’)

should be:

@onSessionEnded

This will allow to hide the underlying topic names so they can be changed without impacting the dependent code.

Just an intuition. What do you think?

Yes this was the whole point of the library, to hide this. The decorated function will get a Rhasspy Hermes object as an argument. But I can see that some people need a way to subscribe to MQTT topics, for instance for non-Hermes topics, and they like to do this the same way with a decorator. The decorated function then gets the topic and payload as arguments. So I think an on_topic decorator can be complementary.

1 Like

That’s right if you only think in the rhasspy universe. To understand my intention for this decorator “hermes/dialogueManager/sessionEnded” is very bad example. I only use this for testing as an example. In my app I subscribe also topics which are not part of rhasspy. A own decorator for all these topics is too much work, not so easy to write and hard to bring all these decorators in the on_raw_message loop.
If you take a look at the second parameter of the decorator you will see a regex pattern for topic matching. This is the more important feature and the reason why the topic name is forwarded to the method.

I will give you an example how I use this in real life.
I have 2 3d printers. For long running jobs I have an intent to enable rhasspy to inform me if a printer has finished or goes in error state. Each printer has his own topic for reporting. The topic pattern is printer/<name of printer>/state.

Now I can subscribe a topic for each printer or I use a wildcard topic like printer/+/state. With this annotation @app.on_topic("printer/+/state", re.compile(r"^printer/([^/]+)/state")) I get both topics in one method. For differentiate between the printers I need the topic to extract the name from. And at least, if I get third printer I do not have to change my code.

This is only one example where I use wildcard topics. Maybe I’m the only one that needs such kind of generic topic subscription. If it’s not useful in general I would only refactor the app so that I can easier create a subclass to implement this feature only for me.

@koan @H3adcra5h Fair enough! I see both your points. I was just reacting to the dialogueManager example and did not take the “custom” topic decorator into consideration. :+1:

Not a fan of providing both the subscribed topic and a RegExp. If the goal of the library is to ease the boilerplate for users, wouldn’t it be easier provide some kind of templating for the topic name like:

printer/{name}/state will subscribe to printer/+/state and upon message will extract the middle part of the topic and provide it as name.

printer/# will match any topic starting with printer/.

The decorated function can for example accept 3 arguments: topic, payload and properties (extracted from the topic using the above).

What do you think?

Yes that was also my reaction on the pull request. The topic seems redundant if you use the regex pattern.

Your idea of templating seems interesting too. Will this be useful for your use cases, @H3adcra5h?

@fastjack I try to understand your point.

You prefer having placeholder in the topic instead of the wildcard. The implementation should parse and create the regex for matching the topic. On an incoming event it extract the the data, map it to the right placeholder and provide this as properties or dictionary.
Then the topic as argument is unnecessary. Like this:

@app.on_topic("printer/{name}/state")
def test_topic(properties: dict, payload: bytes):
    _LOGGER.debug(f"name: {properties.get('name'), payload: {payload.decode('utf-8')}")

Is it right?

Actually, do you need the regex pattern or a template at all? In all your examples you just use a wildcard topic and this can be just used like this:

@app.on_topic("hermes/hotword/+/detected")
def test_topic(topic: str, payload: bytes):
    content = json.loads(payload.decode('utf-8'))
    _LOGGER.debug(f"topic: {topic}, model: {content['modelId']}")

Your code should just work like this. Because in on_raw_message the concrete topic that matches the wildcard (like "hermes/hotword/foobar/detected") is passed, and you pass this to the decorated function with function(topic, payload). So in the function the topic name is in the topic argument.

Right if the app creates a regex to match the topic to the right decorated function.
I have to look how complex wildcards can be and if I’m able to create the right regex on the fly. I’m not a specialist for regex.

It seems there are only + and #. So it should be possible for me to create a regex on the fly.
But the more I think about it, with placeholders could also be a nice solution, or a mix of both.

With placeholders would look nicer (and more easy to read for ppl without programming background) but I want to note that the placeholder should not be fixed in one place only.

For the example of printer/{name}/state it works but someone might come along that wants to filter by state also, or has more abstruse hermes topics because one IoT device sets a strange topic that switches the order around.

Yes, I had it in mind. Like: printer/{room}/{name}/state

1 Like

The generic decorator can then easily be provided multiple topics if required:

@on_topic("{device}/{name}/state", "light/{name}/on", "light/{name}/off")

I don’t know if this is really required but should not be hard to implement and allow more freedom to the library users. Maybe this is a bit too far :wink:

Step 2 or 3, I think :wink:

I think in this case a bit to far might be better. I have only started playing around with mqtt when I turned up in this forum but I have found some strange mqtt topics around the internet. I even came across one example where someone somehow had the state in the middle.

Most ppl would never do this but since the library is a way to make it easier for less experienced users some strange cases might turn up and depending on how long ago they formed the habit they might be unwilling to change. And like I said, some IoT devices set pretty strange topics, too.

I was mistaken here. I overlooked that the code checks for equality between the key and the topic. But I think with some minor changes this could be made to work.

So there are two separate ideas in the current discussion:

  1. I think we should drop the regex pattern, so the on_topic decorator only needs one argument: a string. With some modifications the code could interpret this string as a topic wildcard and create a regular expression that matches all topics matching this wildcard. In the on_raw_message function the incoming topic would then be matched with the regular expression to see which decorated function should be called. This way you can decorate a function with @app.on_topic("printer/+/+/state") and the function would be called for topics printer/office/brother/state, printer/livingroom/hp/state and so on. The same could be done for the # wildcard.
  2. I like @fastjack’s idea of the templates and named properties. But because there are very few restrictions on MQTT topic names and printer/{room}/{name}/state is a valid MQTT topic, I think we shouldn’t use a string for this argument, but define (or re-use? this looks like very general functionality, so maybe we could re-use some existing minimal templating language module for his) a specific class for this. Or maybe just define another decorator for this, such as on_topic_template.

I think having 1. first and later extending the code to have 2. would be good. 1. would be a nice feature for the first published version of rhasspy-hermes-app on PyPI, so people can already react to Hermes messages that don’t have their own decorator yet in the library.

I made a new version and a pull request, so everyone can try it. This 4 kinds of subscriptions are working so far:

@app.on_topic("hermes/hotword/{hotword}/detected")
def test_topic1(data: TopicData, payload: bytes):
    _LOGGER.debug(f"topic1: {data.topic}, hotword: {data.custom_data.get('hotword')}, payload: {payload.decode('utf-8')}")


@app.on_topic("hermes/dialogueManager/sessionStarted")
def test_topic2(data: TopicData, payload: bytes):
    _LOGGER.debug(f"topic2: {data.topic}, payload: {payload.decode('utf-8')}")


@app.on_topic("hermes/tts/+")
def test_topic3(data: TopicData, payload: bytes):
    _LOGGER.debug(f"topic3: {data.topic}, payload: {payload.decode('utf-8')}")


@app.on_topic("hermes/+/{site_id}/playBytes/#")
def test_topic4(data: TopicData, payload: bytes):
    _LOGGER.debug(f"topic4: {data.topic}, site_id: {data.custom_data.get('site_id')}")

The data contains the matched topic and custom data with the values of the named placeholder.
As you can see, you can mix it.

All topics are only examples and do not have to make sense. :wink:

2 Likes

This is very nice. If you’re ambitious, you could try adding @fastjack’s suggestion to add multiple topics. I think this is doable by changing the decorator’s signature to:

def on_topic(self, *topic_names: str):

And then put almost all the code inside the wrapper function in a loop for topic_name in topic_names: and then probably some minor changes elsewhere.

I’ll have a closer look tomorrow and test it before merging the PR.

It wasn’t as easy as you think, but I refactored it. Maybe some parts could/should be better implemented, because I don’t know all python features.

So, thanks to @H3adcra5h you can do now something like this in the library:

@app.on_topic("hermes/+/{site_id}/playBytes/#")
def test_topic4(data: TopicData, payload: bytes):
    _LOGGER.debug(f"topic4: {data.topic}, site_id: {data.data.get('site_id')}")

And even:

@app.on_topic(
    "hermes/dialogueManager/sessionStarted",
    "hermes/hotword/{hotword}/detected",
    "hermes/tts/+",
    "hermes/+/{site_id}/playBytes/#",
)
def test_topic1(data: TopicData, payload: bytes):
    if "hotword" in data.topic:
        _LOGGER.debug(f"topic: {data.topic}, hotword: {data.data.get('hotword')}")
    elif "playBytes" in data.topic:
        _LOGGER.debug(f"topic: {data.topic}, site_id: {data.data.get('site_id')}")
    else:
        _LOGGER.debug(f"topic: {data.topic}, payload: {payload.decode('utf-8')}")

The examples are purely fictional, of course. The goal is to have specific decorators for Hermes messages that work on Rhasspy Hermes classes, but in the mean time, you can use this on_topic decorator for Hermes messages that don’t have their own decorator yet. This also makes it easy to react to both Hermes messages and non-Hermes MQTT messages.

Thanks again, @H3adcra5h!

2 Likes

I just created a first rough API documentation. You can generate it with:

make docs

Afterwards you can find the generated HTML pages in docs/build/html. The plan is to extend this documentation and publish it for easier discovery of the library’s functionality.

With the new “on_topic” decorator I am trying to handle the “intent not recognized” event like this:

@app.on_topic("hermes/dialogueManager/intentNotRecognized")
def intent_not_recognized(data: TopicData, payload: bytes):
    _LOGGER.debug("not recognized: " + str(data) + str(payload))

    payload_json = json.loads(payload.decode("utf-8"))
    session_id = payload_json.get("sessionId")
    site_id = payload_json.get("siteId")
    if session_id in akinator_sessions:
        text = 'Das habe ich nicht verstanden'
        app.publish(
            DialogueContinueSession(
                session_id=session_id,
                site_id=site_id,
                text=text,
                intent_filter=intent_filter,
                custom_data=None,
                send_intent_not_recognized=True,
                slot=None
                )
        )

The problem is: It only kind of works. Rhasspy seems to push some or most of the TTS message into the STT system. Causing an infinite loop of intent_not_recognized events. It should wait until the TTS output is finished before recording. It does that normally, why not when the ContinueSession is published from the intent not recognized event. Is that a Rhasspy bug?

Regarding the api:

It would be great if there were a parameter in the decorator to say the system “payload is json” and it doing the json-load and decode for you. Even better in my case a on_intent_not_recognized decorator´

And what if you subscribe to hermes/nlu/intentNotRecognized instead of hermes/dialogueManager/intentNotRecognized?

I have just added an on_intent_not_recognized decorator in https://github.com/rhasspy/rhasspy-hermes-app/commit/485cfceeda63f66b478fee1b7fb5db9651170231. Note that this is for hermes/nlu/intentNotRecognized. Let me know if you need the other one too.

I think hermes/dialogueManager/intentNotRecognized is the correct one to use. It’s behavior is depending on the send_intent_not_recognized parameter of the DialogueContinueSession message. (True will issue the intentNotRecognized event False will end the session. I looked at the Rhasspy source).

Sadly with both it will capture part of the TTS response as input.

That seems like a bug in Rhasspy then. Can you open an issue in the rhasspy repository?

Thank you. I would need the other one. But there are probably use case to have both. The nlu one will react to every not recognized intent the dialogueManager only in an active dialog session (at least that’s my understanding)

I will do it later. I want to have a second look at the Rhasspy source before that. Yesterday it confused me, as it seems to handle the dialogue continue session message always in the same way.

I have added your suggestions as issues in the repository. You can subscribe to an issue to follow progress.

It seems I got confused while testing. The issue with intentNotRecognized triggering a loop because it started to listen to it’s output is triggered by a 10 second TTS timeout in DialogueHermesMqtt.

My help text is pretty long (13 seconds). I thought I could also reproduce it with a shorter text but I can’t anymore. I have opened an issue: https://github.com/rhasspy/rhasspy/issues/62

I wrote a small countdown timer app using your library:

I think everything is there for that use case.

Some things that could help:

  • An easy way to access the slot values
  • An easy way to start a session (with type notitfication)
  • (in Rhasspy: a builtin slot type for duration)

That’s looking nice!

What would you suggest?

Yes, it makes sense to add a method to start a session.

Built-in slot types are indeed a feature that could help a lot of use cases. But that should be embedded in Rhasspy itself indeed. There have been some discussions about it already.

I’m now working on tests (current code coverage: 44%) and there’s autogenerated API documentation. I want to expand both before I create the first release on PyPI.

I just published the first release of Rhasspy Hermes App on PyPI: 0.1.0.

You can install it with:

pip3 install rhasspy-hermes-app

If the API of this library changes, your app possibly stops working when it updates to the newest release of Rhasspy Hermes App. Therefore, it’s best to define a specific version in your requirements.txt file:

rhasspy-hermes-app==0.1.0

This way your app keeps working when the Rhasspy Hermes App library adds incompatible changes in a new version.

I have expanded the documentation, including some information for people who want to contribute.

All contributions are welcome! New features, bug fixes, documentation, tests, …

@H3adcra5h I have added a lot of documentation, mypy annotations and tests to the code, but for now I have left your raw topic additions more or less alone because you know that part of the code better. If you could take a look at it, that would be awesome.

4 Likes

I am no real Python developer and don’t really know what is best practice for such things.

One idea would be: add a new parameter “supported_slots” to the decorator.

Could look like this:

@app.on_intent("StartTimer", supported_slots=["minutes", "seconds"])
def handle_start_timer(intent, minutes, seconds):
  or
def handle_start_timer(intent, slot_minutes, slot_seconds):

If the Intent has slot values for minutes or seconds handle_start_timer will get called with that otherwise it will provide “None”. One variant of this would be to also put the expected type into the decorator and let the decorator do the validation.

Another option: Just provide a helper function like mine in your module to get the slot values out of an intent. As you can see in my source (maybe I am doing it wrong) it is not that simple.

Ok, I have created an issue about easier access to slot values to welcome a discussion and follow progress on this.

I have released version 0.2.0 with two additions:

Added

  • Method HermesApp.notify to send a dialogue notification. See #10.
  • Decorator HermesApp.on_dialogue_intent_not_recognized to act when the dialogue manager failed to recognize an intent. See #9.

The latest version can be installed from PyPI:

pip3 install --upgrade rhasspy-hermes-app
1 Like

I have released version 1.0.0 now, with one big change:

Changed

  • All decorators of this library now only work with async functions. Pull request #16 by @JonahKr. Existing code should only add the async keyword before the function definition to keep the code valid. See Usage for some examples.

This is a breaking change, hence the major version bump. It has been contributed by @No_one and makes it easier to integrate Rhasspy apps with asynchronous APIs, improving performance. The latest version can be installed from PyPI.

4 Likes

Just a quick question. I don’t see any mqtt config in any of the examples on github. Am I correct in assuming that it just uses whatever config rhasspy itself uses?

No, it uses localhost:1883 as default, but you can change all MQTT connection settings as command-line options to your app. See the example output of python3 examples/time_app.py --help in the usage section of the documentation.

I’m still thinking about additional ways to pass these parameters, by the way.

Okay, if that is the case, I am definitely someone needing an additional way of setting things.

I am trying to get my complete chaos of weather app to be less of a chaos and have it use this as an optional output. Thing is, I am actually using a config ini to set parameters such as mqtt options (I am using paho mqtt directly right now and it will be an option even after adding support for this) so setting this with command line parameters would circumvent the config file that I need for quite a bit of other stuff.

It seems adding support for this needs to wait until there is a way to feed values from my config file into this app or I have given up on having mqtt settings in my config file.

There’s currently a pull request that adds support for something like this:

app = HermesApp("AdviceApp", host = "192.168.178.123", port = 12183)

This works with all parameters that are recognized on the command line. You could read your MQTT parameters from your configuration file and feed them like this into the app.

I still have to find some time to test it, especially how it interacts with parameters passed on the command line.

I’m cautious about adding direct support for a specific configuration format, because everyone has his own preferences for this, but the Rhasspy profile configuration format could be a sensible addition.

I would definitely prefer that format over the command line parameters, but I think I need to rethink a few things of my library because right now I don’t think my concept fits that well into yours. I need to play around with this a bit before I can figure that out.

It just occurred to me while writing this that reading mqtt settings out of a per library config is definitely not the way to go, so even the current way with command line parameters would work. My brain seems to be fried right now from coding to long and figuring out how to make my chaos less chaotic. Sorry for bothering you with this, I sometimes forget I write a library and not a standalone app.

Now I’m a bit confused: I thought you were talking about your weather app, but now you mention you’re writing a library.

I am talking about my weather app, my limited knowledge of python tells me that something that I can call and it does something is an app, if I need to call it from somewhere else and feed information into it is is a library. I might be wrong about that assumption, the python course at university definitely didn’t teach about things like that, only about what an if is and does.

Basically my weather something is supposed to be called by whatever whoever uses it wants to use, be that a script using rhasspy-hermes-app, a command script or something else. I just take whatever input, parse it with whatever the config tells me to use it as a parser and output to whatever the config tells me to output to. So if rhasspy-hermes-app is used the mqtt settings have to be done there, trying to fit my settings for “just output this to this mqtt server” in there is not the way to go.

Oh right, that way. Well, I’m curious, I’ll follow its development, because it looks like a very interesting app to have.

I have released version 1.1.0 , with one small but interesting change:

Changed

  • Command-line arguments can now also be passed as keyword arguments to the constructor of a HermesApp object. Note that arguments on the command line have precedence. Pull request #37 by @No_one with help from @maxbachmann.

Keep your issues, comments, pull requests coming. And feel free to let us know here how you use or want to use Rhasspy Hermes App or what can be improved.

1 Like

Might be me being stupid but how do I get rhasspy to actually answer via tts? I have set up the GetTime example, I connect to the mqtt server just fine and I do get the EndSession message and it looks right but nothing happens. Do I need to config rhasspy in a specific way for it to work?

EDIT: never mind… it works with a wakeword but not with the wake up button in the web gui. I have no idea why it does that but I am pretty sure that it is something rhasspy does differently with the wakeup button and normal working behaviour.

Yes I bumped into this too a while ago. I don’t remember the exact difference, but I think the wakeup button doesn’t initiate a full session. It’s best to test your app with “real” sessions beginning from a wake word you told yourself.

I normally don’t work with real sessions because I have been unable to actually get a wakeword working in a non annoying way. Best I got so far is raven and even that breaks after a while and activates on silence because my respeaker mic ends up creating random noise on one channel after a while without restart, those driver still don’t work well. It was a pretty big fight until I got the mic to behave good enough to actually record a wakeword.

It would be really nice if the button would actually work the same way rhasspy responds when I use a wakeword because it would testing just that much easier but I can see why that is pretty low on the priority list.

That’s a good point. I opened an issue on GitHub:

1 Like

It has been a while, but I released version 1.1.1, which is a minor release:

Changed

  • Updated dependencies. The most important one is the upgrade to rhasspy-hermes 0.5.0.

This fixes an encoding issue.

I have been busy with some other stuff and so I haven’t been active lately with Rhasspy and Rhasspy Hermes App, but if you have some ideas, requests, issues, just let me know here or on the project’s GitHub page. I’ll be happy to look into it.

1 Like

hi, i’m working on an vocal assistant for my thesis and your helper library helps me a lot so thanks already. My question is: is it possible to start a session without any wakeword. For example the launch of a survey if it has been more than 24 hours since the user has used the tool. I saw that there was the notify function which could also be useful to notify the user that it has been a while since he has used the service but I do not understand the side id argument, it is on localhost:1883 that the function should be set?

You should use the notify method indeed. The site_id argument is the site ID you entered in Rhasspy’s settings. So it’s not a hostname/port combination, but the name you assigned to that device, such as default.

I just realized you want to launch a survey. The notify method is used if you want to inform the user of something without expecting a response. But if you want to ask the user something and expect a response, so like a real session but initiated by the app instead of by the user telling the wake word, you have to start a session with a DialogueAction initialization description.

ok i understand. But I try to put in a function app.notify(“test”,“default”) with the default parameter as i can see in my rhasspy interface but it doesn’t work.

With the examples and the documentation I managed very easily to use the sessions and associate them to intents but it is very confusing to launch a notification or a dialog. I guess it’s the same as opening a client mqtt.Client() and connect the client on the localhost with port 1883 and then publish(DialogueStartSession(init=notification, site_id=site_id)). But this means not using this library anymore. Would it be possible to create an example of using notification and login as for the rest to make this library even more efficient as it is intended to facilitate the use of rhasspy via hermes?

I’m not sure why this doesn’t work, I’ll try to reproduce this later.

But you can still use Rhasspy Hermes directly in your Rhasspy Hermes App code. The HermesApp class inherits from HermesClient, so you can just publish any Rhasspy Hermes message from your app with app.publish(...).

I have added an example app to show how to use the notify method:

This works for me. Let me know if you have some problems in your setup.

1 Like

I have now also added documentation on how to continue a session in Rhasspy Hermes App:

This works now because rhasspy-nlu-hermes issue #3 has been fixed in Rhasspy 2.5.10.

2 Likes

I’ve published a new bugfix release for rhasspy-hermes-app, version 1.1.2. I also enabled GitHub Discussions in the repository.

2 Likes