Rhasspy Home Assistant add-on: is it possible to create skills?

Help
1

Hi all,
new to the amazing Rhasspy, but I’ve managed to have it working as an home assistant add-on.
I am using it to pass vocal commands to HA and control my house, amazing!
HA is installed as a container on a RPi4.

Now that I’ve correctly set-up different intents (ie control lights, get time, manage shopping list etc) I am trying to create a skill to tell me the next football match of my favourite team, but I can’t manage how to do that.

I’ve also tried to chat with Chat GPT lol
ChatGPT suggested me to create a folder “skills” (under …profile/it/skills) with a .py with the script I need to do an API call to fetch the matches info.

From the log anyway, it seems that the problem is that the intent is sent to HA to the handling.
So ChatGPT suggest to change the profile.json in different way, but till now no solution…

Another possibility would of course let HA do the API call etc., but I would try that way in order to be more free with the python code (the skill is basically a .py, while at least from what I’ve learn thare are some limits using python with HA)

Could you help me please? I think that a good guide to “how to set up a skill in a Rhasspy as HA addon” can be a big benefit for the whole community and open a lot of possibilities.
Sorry if the topic has already been discussed in other thread, I’ve not found anything that seem releted to this.
Thanks a lot!

2

check out on the rhasspy github for skills like the daily joke skill, and you can see how skills can be added. having home assistant do it, really makes sense if you have the info in home assistant already.

3

Shameless plug here:

You can also check out my github.

I’ve been slowly building a set of skills for Rhasspy. My time skill is pretty flushed out and has a decent read me to guide you through the setup. I recommend the docker way. The Grocy integration is working but still need more intents created. I am also almost done with my timer skill which will allow the creation of multiple timers (with optional names).

Looking for suggestions for more. A Jellyfin client control skill is on the roadmap.

1 Like
4

Looks interesting, i’ve done everything with node-red so far.

5

Thanks a lot for your feedbacks.

I read the documentaction and the examples you provided, and ended up with some guesses, but still have some doubt, would appreciate if you can provide more hints…

I guess that all the information needed are here:
https://rhasspy-hermes-app.readthedocs.io/en/latest/index.html

My main doubts is: given my set-up (Rhasspy as add-on (so container) of an Home Assistant instance running in docker on a RPi4) do I need to install rhasspy-hermes-app in the same container of rhasspy or directly on the RPi4 (pip3 install rhasspy-hermes-app)?
Second, after the hermess-app is installed, how do I tell the main rhasspy instance (so the HA add-on) to send the intent to rhasspy-hermess-app? Because for the moment I’ve set up “hass” to handle the intents (and fortunately it works).

Or I am completely going in the wrong way?
Thanks a lot!

6

The rhasspy-hermes-app library isn’t useful by itself. It’s a starting point for skill developers (like I am trying to be) to use to create a python app that can connect to the MQTT bus and respond to intents fired onto the MQTT bus. So an example of what you could do is, let’s say you like my time skill. You need a place to host it. It can be run in a docker container by itself or run it as a service on an existing linux server that you have. The key is the server it is running on has to have access to the MQTT broker that the rest of Rhasspy is using. Then you start up the python script (either via docker or service) and it connects to MQTT and starts listening for the intents defined in the skill script. Then you create sentences in Rhasspy that will fire the intents defined in the script. When you say the sentence to Rhasspy, it will then resolve the sentence to the intent name, send the fire intent command to MQTT with the intent name, the skill script sees that on the MQTT bus and starts responding.

You can then have multiple scripts running to handle different things. Currently I am running the time skill and just started running my timer skill. Working on my Grocy skill still.

I also have it tied to Home Assistant so I have a few intent scripts in there to handle Home Assistant related things. I have found the combo of both to be very useful.

As I started with, the rhasspy-hermes-app is for a developer to use. So if you are up for it you can use it to write your own skill but the library itself just helps you to connect to MQTT broker and listen for Rhasspy intents (and more). But please feel free to use mine. If something doesn’t work for you let me know and I’ll see what I can do. I am trying really hard to make it something useful for everyone. Adding in multi-language support for all of my skills is on of my next things on my roadmap.

2 Likes
7

Thanks @JoeSherman.

Could you please give me more details?

For example, to start from scratch:

  • how do I retrive the username and the password used by the default internal rhasspt MQTT? (cause also from you git, the option user and pwd are needed);
  • do you suggest to create an empty cointainer in which install the hermes-app library and all the skills (.py script)? If this is the case, could you please provide me an example of docker command with all the settings needed?

Sorry if I’m going completely in the wrong direction, I’m trying my best but it’s not easy…

8

Hello,

  1. I am using a MQTT broker in my HA instance so i got it from there. In HA any user also has access to MQTT so I made a rhasspy user in HA and I use that. The docker container I created will accept the URL, port, username, and password as environment variables.If you are using the internal Rhasspy MQTT there are some instruction in the wiki on how to connect to it remotely. I recommend an external mQTT though.

  2. I have a dockerfile in my repo. You can use that to build the container. There should be some basic instructions in the read me. But basically use the dockerfile to build a local image (docker build command). Then you use the local image to create a container (docker up command). Supply the environment variable values to the container and when it starts up it should start the skill py script auto automatically and connect to your MQTT.

9

I know that I’m asking too much, but you guys are amazing and the only resource I have…

@JoeSherman when you have time, could you provide a detailed guide with also the code needed for each step?
I’m struggling trying to follow your “high level” indications.

Thanks!

10

No problem. Today was a family day so didn’t have time but I’ll be sure to give you some details tomorrow after work.

1 Like
11

@giof94is,

Here’s my best shot at a step by step for installing my time skill from here: GitHub - MCHellspawn/hermes-app-time

  1. Login to a Linux server with Docker installed, make sure this server has access to the same network as Rhasspy.
  2. Create a folder to work from for the install and navigate into it.
mkdir time-skill
cd time-skill
  1. Clone the git repo
git clone https://github.com/MCHellspawn/hermes-app-time.git
  1. Build the docker image, this is just the image the container will use. This will run through the “dockerfile” in the repo. It will start with the python 3.8 alpine Linux image and then add in the python requirements, copy the files needs from the cloned repo and save the image in your local docker registry for use later.
sudo docker build hermes-app-time -t <image_name>

replace <image_name> with whatever you want to call the image.
5. Create and start the docker container

sudo docker run -it -d \
        --restart always \
        --name <container_name> \
        -e "TZ=<TZ Name>" \
        -e "MQTT_HOST=<MQTT Host/IP>" \
        -e "MQTT_PORT=<MQTT Port (Typically:1883)" \
        -e "MQTT_USER=<MQTT User>" \
        -e "MQTT_PASSWORD=<MQTT Password>" \
        <image_name>

Replace the following:
= Whatever you want to call the running Docker container
= A valid Linux Time Zone name (example list here: Linux Time Zones All Available List - Ed Chen Logic)
<MQTT Host/IP> = The hostname or IP address of your MQTT broker
= The port of your MQTT broker, usually 1883 but Rhasspy’s internal one uses something different
= The username used to connect to your MQTT broker
= The password used to connect to your MQTT broker
<image_name> = The image name defined in step 4
6. Log out of that server and login to your Rhasspy device that is doing the intent handling
7. Navigate to the profile folder
For example:

cd /profiles/en

Your profiles folder may be different. It is listed on the settings page on your Rhasspy device’s web UI.
8. Navigate to the slot_programs folder and create a new folder for this skill and navigate into it

cd slot_programs
mkdir time
cd time
  1. Copy the slot program down from my repo and mark it executable
wget https://raw.githubusercontent.com/MCHellspawn/hermes-app-time/master/slot_programs/timezones
chmod +x ./timezones
  1. Open the Sentences page on your Rhasspy device (the same one you put the slot program on)
  2. [Optional] Create a new sentence file (I create 1 sentence file for each skill for organization)
  3. Create sentences to fire the intents
[TimeGetTime]
timezones = $time/timezones
what is the [current] time
what time is it
tell me the time
what time is it in [the] (<timezones>){timezone} [timezone]
what is the [current] time in [the] (<timezones>){timezone} [timezone]
time

[TimeTzDiff]
what is the time difference between (<TimeGetTime.timezones>){timezone1} and (<TimeGetTime.timezones>){timezone2}
  1. Save the sentences and retrain Rhasspy.

Give it a shot. If you hasve any trouble let me know.

1 Like
12

Thanks a lor @JoeSherman!

I will give it a try as soon as I can.
I have only one doubt: when you say “Log out of that server and login to your Rhasspy device that is doing the intent handling” → actualy I’ve set up HomeAssistant as intent handle, cause I would other vocal commands to be executed with it.

Having sad so, is the process I must follow the same? My lessons learned is that Rhasspy will try to pass the intent name (and it’s data, like slot etc., eventually) to HA, that without an intent_script configured will give “unrecognized intent” error, isn’t it?

13

Hi,

Yeah, should be right. I have the same setup. The I guess a better way of saying it would be to log in to the device that handles intent recognition.You can determine this by looking at your Rhasspy settings on each device.

On my satellites this is set to Hermes MQTT:
image

But on the base this is set to an actual intent recognition engine (in my case Fsticuffs). This is the device where the slot programs get added.
image

In my setup, all recognized intents get sent both to Home Assistant and the MQTT bus. So for intent I want home assistant to handle it does, and intents I have custom Skill Apps running on my network, like this time skill, get handled by those apps. As long as no 2 systems handle the same intent things work just fine.

14

@JoeSherman thanks man!
I was trying to follow your detailed guide, but I ended up with the following problem when trying to create the docker image:
~/time-skill $ sudo docker build hermes-app-time -t test_time_skill
[+] Building 11.7s (6/16)
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build definition from dockerfile 0.1s
=> => transferring dockerfile: 761B 0.0s
=> [internal] load metadata for Docker 0.5s
=> [internal] load build context 0.0s
=> => transferring context: 4.61kB 0.0s
=> CACHED [ 1/12] FROM Docker 0.0s
=> ERROR [ 2/12] RUN apk add --no-cache tzdata 11.0s

[ 2/12] RUN apk add --no-cache tzdata:
#0 0.788 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/aarch64/APKINDEX.tar.gz
#0 5.794 fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/aarch64/APKINDEX.tar.gz
#0 5.794 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.14/main: temporary error (try again later)
#0 10.80 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.14/community: temporary error (try again later)
#0 10.80 ERROR: unable to select packages:
#0 10.81 tzdata (no such package):
#0 10.81 required by: world[tzdata]

dockerfile:2

1 | FROM python:3.8.10-alpine
2 | >>> RUN apk add --no-cache tzdata
3 | RUN apk add --no-cache gcc libc-dev
4 | ENV TZ=America/New_York

ERROR: failed to solve: process “/bin/sh -c apk add --no-cache tzdata” did not complete successfully: exit code: 1

This also executing everything as root (with sudo su before).

I’ve tried to edit the cloned git (hermes… folder) and put a ‘#’ (comment) before " RUN apk add --no-cache tzdata". So the error disappeared, but the same problem happened with “RUN apk add --no-cache gcc libc-dev”. So another comment…
Then another error with “RUN pip install -r /requirements.txt”
=> ERROR [ 6/10] RUN pip install -r /requirements.txt 45.1s

[ 6/10] RUN pip install -r /requirements.txt:
#0 7.240 WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘NewConnectionError(’<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fb7294820>: Failed to establish a new connection: [Errno -3] Try again’)‘: /simple/dataclasses-json/
#0 12.75 WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘NewConnectionError(’<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fb7291610>: Failed to establish a new connection: [Errno -3] Try again’)‘: /simple/dataclasses-json/
#0 18.76 WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘NewConnectionError(’<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fb7291c40>: Failed to establish a new connection: [Errno -3] Try again’)‘: /simple/dataclasses-json/
#0 25.77 WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘NewConnectionError(’<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fb7291cd0>: Failed to establish a new connection: [Errno -3] Try again’)‘: /simple/dataclasses-json/
#0 34.79 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by ‘NewConnectionError(’<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fb7291790>: Failed to establish a new connection: [Errno -3] Try again’)': /simple/dataclasses-json/
#0 39.80 ERROR: Could not find a version that satisfies the requirement dataclasses-json==0.4.2 (from versions: none)
#0 39.80 ERROR: No matching distribution found for dataclasses-json==0.4.2

dockerfile:15

13 | RUN pip install --upgrade pip
14 | COPY requirements.txt /
15 | >>> RUN pip install -r /requirements.txt
16 |
17 | RUN addgroup rhasspy-app \

ERROR: failed to solve: process “/bin/sh -c pip install -r /requirements.txt” did not complete successfully: exit code: 1

Very strange…any suggestions?
Thanks a lot!

15

This does seem very strange indeed. It can’t find the tzdata package for the O/S, and it seems to be having problems finding all the python dependency packages. Almost like it doesn’t have network/internet access? But not sure how it would have downloaded the alpine image if that were true, unless that was downloaded previously? I’m going to try and run through my own instructions on my server and see if I have any issues.

16

Hi @JoeSherman any news? This problem is driving me crazy, can’t understand why…I’m sure it has a working internet connection, and I’ve also tried to disable both ufw and pi-hole…any suggestion?

17

@giof94is

I did dig into this but wasn’t able to find any issues on my servers. I’m not an expert at linux admin by any means, mostly a self-hoster trying to figure this all out myself, so there must be something about your setup that I don’t understand, or even know enough to ask. Maybe someone else here might know?

I can tell you I blew away my skill container and followed the instructions on that site exactly and it rebuilt it perfectly on my setup. I’m running docker on an Ubuntu server running on a Xeon workstation I have. Are you running this on an x86 machine or an ARM device like a PI? Maybe that’s the difference?

18

Thanks @JoeSherman
I’m running on a RPi4, 64bit.
I’ve though the same, but going on the package repo I found also the version for my setup…try to give it a look if you can, but my error apart should be like that.l amd the error anyway is connection/network error.

Anyway, what are these pcks and are them must to have? I’ve tried to remove them from the dockerfile, the container build but then it goes in a restart loop…
The idea was to see if for other script/skill I could follow your guide / install the pck later, but I’ve crashed also on that…

Thanks a lot for your help and patients, would be nice if also other from community will give an hand on that!

19

Yeah, I am really trying to get a set of intents that will make Rhasspy useful for more people. Part of that includes more people being able to successfully install them and get them running. So I will definitely look into it more. I have a few RPi’s here so I will get one setup for testing. I will also get you the list of dependencies, can’t right now but should be able to tomorrow.

Once I get it working on my Pi I will create a 2nd set of install instructions. For now I have to say it looks like this is x86 compatible only. But not for long I swear.

20

@JoeSherman I’ve found the solution!
It was a “docker network error”, adding “–network=host” to the docker build command solve the issue (probably changed the DNS docker use to get the pcks from the repo).

Now, following your guide, I’ve been able to reach the intent recognized and replied by the program skill.
The only problem is that it seems not recognizing the time zone.
I’m in Italy, and for example if I ask “what time is it in New York” it replies with the current time in Italy, not in NY. Looking at the container logs (sudo docker logs …) it seems that the app doesn’t receive a time slot from Rhasspy: (cfr timezone slot empty)
INFO:2023-04-25 13:59:29,375] TimeApp: Intent: 9376acbf-ec3c-4ef6-bb2f-28e7c53c0be9 | Started: TimeGetTime
[INFO:2023-04-25 13:59:29,377] TimeApp: Intent: 9376acbf-ec3c-4ef6-bb2f-28e7c53c0be9 | Timezone slot empty
[INFO:2023-04-25 13:59:29,379] TimeApp: Intent: 9376acbf-ec3c-4ef6-bb2f-28e7c53c0be9 | Date/Time: 2023-04-25 13:59:29.379540
[INFO:2023-04-25 13:59:29,387] TimeApp: Intent: 9376acbf-ec3c-4ef6-bb2f-28e7c53c0be9 | Responded to TimeGetTime
[INFO:2023-04-25 13:59:29,390] TimeApp: Intent: 9376acbf-ec3c-4ef6-bb2f-28e7c53c0be9 | Sentence: it’s 01 59 PM
[INFO:2023-04-25 13:59:29,391] TimeApp: Intent: 9376acbf-ec3c-4ef6-bb2f-28e7c53c0be9 | Completed: TimeGetTime

Any suggestion? As you indicated, I’ve put and marked as executable the timezones program in the slot_programs of my profile folder, as done for other slot programs that do work. It seems that Rhasspy is not able to associate the text to the slot and send it in the apposite key of the json.
Moreover, the [TimeTzDiff] intent is not recognized at all, even if I copy exactly the sentence defined in the .ini file and put two valid timezones (tried with different combination)…

Thanks

21

Awesome news!!

If the app isn’t receiving the slot then there may be something wrong with the sentences. Can you share your sentences for grocy?

22

@JoeSherman issue solved, probably a restart of Rhasspy / Host is needed to execute the slot program and grab its results to the memory, I can’t explain it differently because it’s the only action that worked, at least in my case.

Now I would like to share with you new evidences, hoping these feedbacks could be useful for you to improve your work:

  1. With this sentences linked to the intent [TimeGetTime]:
    “che ore sono a (){timezone} [timezone]” (what time is it, in italian)
    → Rhasspy recognize the time zone slot only if I type on the front end (never with voice, probably for the special character ““) “che ore sono a America New_York” (New York without _ or before America it’s not recognized as slot) → maybe the slot_program can be improved, for example with a replace(””," ") and all the permutation (ie 1 row with America New York and a 2 row with New York America).

  2. The time difference is recognized by Rhasspy (obviously also in this case the time zone must be explicated like above) with the right intent that is sent to the skill but the following error appears in the container log:
    [INFO:2023-04-25 15:47:11,410] TimeApp: Intent: 8d52d9be-62bc-4422-84a6-8cc9d262fb45 | Completed: TimeTzxDiff
    [ERROR:2023-04-25 15:47:11,415] HermesApp: on_raw_message
    Traceback (most recent call last):
    File “/usr/local/lib/python3.8/site-packages/rhasspyhermes_app/init.py”, line 212, in on_raw_message
    await function_i(nlu_intent)
    File “/usr/local/lib/python3.8/site-packages/rhasspyhermes_app/init.py”, line 332, in wrapped
    message = await function(intent)
    File “hermes-app-time.py”, line 91, in get_tz_difference
    hours = diff_hours_tz(timezone1slot.value[‘value’],timezone2slot.value[‘value’])
    File “hermes-app-time.py”, line 40, in diff_hours_tz
    utc_dt = datetime.now(datetime.tzinfo(ZoneInfo(str(from_tz))))
    TypeError: ‘getset_descriptor’ object is not callable

Thanks!

23

Hey @JoeSherman,
I’ve changed in the hermes-app-time.py:
→ utc_dt = datetime.now(datetime.tzinfo(ZoneInfo(str(from_tz))))
with → utc_dt = datetime.now(ZoneInfo(str(from_tz)))

seems working…difference between NY and Rome:
[INFO:2023-04-25 16:14:43,362] TimeApp: Intent: 2562c452-ce77-440f-ba7f-64a377fd3e31 | Completed: TimeTzxDiff
[INFO:2023-04-25 16:14:43,481] TimeApp: Intent: 2562c452-ce77-440f-ba7f-64a377fd3e31 | Difference in hours: 6
[INFO:2023-04-25 16:14:43,485] TimeApp: Intent: 2562c452-ce77-440f-ba7f-64a377fd3e31 | Completed: TimeTzxDiff

24

Awesome! And my first bug with the code! This is great! Such a cool feeling.

Tha ks for finding the bug. I will update my code and publish a new version. I will also work on the special character thing. I will probably be ok just replacing those with spaces but I’ll see what makes the most sense.

Thank you again!!

25

So this gave me a chance to try out the github issues so thank you!

I created 2 issues. And they are both already closed. I wish it was this easy at work.

I found a pretty decent fix for the slot program. I also thought of a way I want to enhance it. I want to make it display the list again but strip off the first “country” word. So, you would be able to say “America New York” or “New York”. It wouldn’t work for every city but the core timezone names.

I suppose a nice next version of this would be to load a city to timezone database and build an intent to translate city/town name to time zone and do the same thing. That could be used for both diff and time in time zone (city) intents.

26

If you have Home assistant instance, why do you need another app running in another container? Why not use Home Assistant to handle intents or events?

I’m running Home Assistant and Rhasspy on same Raspberry Pi 4 and using events to control my smart devices. For skills I’m using automatons written in yaml. So, what the benefits of using another app?

27

I am under the belief that Home Assistant is good for somethings and not for others. My work on the Grocy skill is a precursor for my major adventure which is a JellyFin/Media Player skill.

As I sat and started working with Rhasspy connected to my HA instance I found the idea of trying to teach Rhasspy how Home Assistant controls Jellyfin made no sense to me. “Why am I not teaching Rhaspy to talk to Jellyfin?” Home Assistant’s integration to Jellyfin is great and all but lacking buy a ton, I still can’t find a way to get a media listing from their API, so how would I provide Rhasspy a slot program for song names or whatever.

This is when I decided I was going to have HA control HA’s things, and build skills for Rhasspy when it made more sense. I admit this line is gray, and some will differ in opinion.

To that point I have also started building a bunch of media player related event scripts in HA that I plan to share as well, and while these technically could be used to control Jellyfin, the capabilities I plan for my skill are far beyond what the HA integration can provide (I think). So I plan to use those skills to control other media players like my TVs and such. And I use HA to control the IoT “Home Automation” devices.

After writing the Grocy skill I have to admit that I think it is probably possibly to recreate the functionality in HA. But I don’t know if the HA integration to Grocy covers all the objects I did/plan to. But I’m pretty sure I could make a set of intent scripts that would make Grocy very usable through HA.

But as I said before, I mostly did the Gorcy skill as a precursor to the Jellyfin one, just so I could get my feet wet with a well documents python API (pyGrocy), focus on building out the skill, and then take that to the Jellyfin one. While building it I also kept in mind the concepts the HA group is with making it a multi-lingual skill, and while mine is not yet, the additional of the random response sentence started the framework towards it.

The whole thing resulted in a python project template/class that I have shared on my Github which is a wrapper for the Hermes App class the Rhasspy people put out. This is setup to be able to read from a local config file and connect to a Rhasspy config to “install” it’s sentences automatically (does not overwrite existing), and can be used to store connection information for API the skill needs or various other thing. I’ve been trying to make the skills more “deployable”, excuse the term, but more commercialized, polished. My hope is if I can create a nice set of easy to deploy skills that are actually useful. No offense to all the time skills out there (I built one too) but we need our voice assistant to do more than tell the time. For that matter it needs to do more than be able to turn my lights on and off.

And yes I know HA is working on that. The stuff that is coming in the 2023.5 release is potentially game changing. But I am worried it will still be bound by the HA API, and it’s integration’s capabilities. If that’s the case, then to me, it will still be in need of external skill programs to “complete” the eco-system. I may move my work over to the new Wyoming protocol at some point, we’ll see.

I know that was long winded, and I admit this is all just my thoughts and opinions. But I hope you can understand the reasoning for some of it. And I understand this is not a one-size-fits-all type of thing.

1 Like