How I'm Integrating between Rhasspy and Home Assistant

Hello!

I started using Rhasspy a couple years back and have been trying to get others to use it as an offline voice assistant for Home Assistant (HA). Unfortunately a lot of folks don’t really want to bother setting it up (either by writing python, doing intent handling in HA yaml, nor using the few baked in intents in HA already).

So, I kicked around this idea of creating a fully managed Rhasspy instance and am calling it Home Intent. You just have to install this one thing and you are good to go! All the Rhasspy settings are pre-determined, it pulls entities from HA and sets up sentences in Rhasspy, and then acts as an intent handler for it all.

Really, it’s a Home Assistant intent handler with extra steps. It’s designed to be fully pluggable, so it could integrate with other non-HA systems as well (I’ve written up some docs on how to do that here).

The code is out on GitHub: GitHub - JarvyJ/HomeIntent: an offline voice assistant with tight integration with Home Assistant<
and I’ve written a bunch of documentation: https://homeintent.jarvy.io/

It’s still pretty early and only supports a few HA components (Home Assistant - Home Intent), but more are on the way!

Let me know what you guys think!

7 Likes

Nice work, but in all honesty I think the people you mention are not going to set this up either.
I hope you can prove me wrong though, because it might help a lot of people struggling with the HA integration :slight_smile:

1 Like

I’m hoping once I get the base of it working enough and I have satellite support in (which is more about creating an easy plug 'n play solution), I can make it installable as a hass.io addon. So folks can just install it there, and then follow instructions on getting a satellite setup on a pi zero w or an esp32 and they’ll be good to go.

I was also thinking about making it distributable as an OS that you can just flash to a pi. But that has a lot more complications, like how do you create an OS image and keep it up to date. Maybe that’ll come even further down the line!

1 Like

Having it as an addon is good idea! Keep us posted

1 Like

Well, it’s Christmas :slight_smile: !

I think there is different profiles using Rhasspy :

  • the really beginner : he has various tutorials to play with Rhasspy OR with HA
  • the experimented makers : the doc of HA and Rhasspy are quite exhaustive to make interactions
  • the intermediate… like me : it is hard and your job would fill this lack ! Tuto aren’t enough explicits to achieve a fully functional installation.

I’d would like to set up my own project (something you don’t find in “all in one” solution), so your help should help me to achieve my goal.
After a first glimpse at your home page, I have some feedback :

  • the only wake word available is “Jarvis” ? Should i train my own (raven) ?
  • A big thanks for the integration of respeaker HAT !
  • Another challenge… Easy bluetooth connection for speakers ?
  • Perhaps, you should add a reminder of docker configuration needed for HA (supervised I suppose), and Rhasspy ?

Great job !

Glad you like it - Thanks for the feedback!

To address some of the concerns:

  • Currently only Jarvis is available. It uses Porcupine since it works the best OOTB. In theory you can override the rhasspy config and use something else instead, but it would require a bit more setup. This actually made me think about having the rhasspy_profile.json merge with the default settings instead of straight override. That way people can customize a section, but still get the other defaults. I’ll add it to the backlog!
  • I think a lot of folks use the ReSpeaker hat, so I added some directions as to how it could work! I should probably get one and try it out at some point.
  • For bluetooth speakers, you’ll probably need to connect it to the main pi/computer running Home Intent and then set the sounds device settings correctly. Eventually I want users to be able to select it interactively via a UI, but that’ll come a bit later.
  • And yes, I should probably update the installation page with how exactly installation is supported right now and that it require some docker knowledge to get it going (for now at least!)

Thanks Again!

After reading that you are dealing with respeaker hats without having one on your own, I took a look at it. First off, the rhasspy tutorial on that is not the best place to send ppl because right now it has you get the drivers from the official github with is almost never up to date because the devs do next to nothing to update for newer kernels. There is a guy that tries to keep everything current here that the devs merge into their repro from time to time which is more current. For now he supports any kernel from 5.9 to 5.11 with the v5.9 branch, and it seems 5.12 broke something again so there is a separate branch for that. Any problems, even with that branch, can be directed to the official github since hintak is the one providing almost all active support there anyway. Right now the official repro seems to be up to date, but I am pretty sure that it will not be after a few kernel updates.

I would advise linking both the official and the hintak repro and advising ppl to use the official repro as long as they use RaspiOS and the pi ppl haven’t gotten to kernel 5.12. Running debian on the pi or any other distro, I would advise HinTak. I haven’t tested 64bit compatability with either repro, but I know HinTak actually uses 64bit debian so hopefully it works on RaspiOS, too.

The next thing is, using those drivers you should have support for the 2-mic hat, the 4-mic square hat, the 4-mic linear hat and the 6-mic hat and it seems they added a new 4-mic square hat from the pictures on the readme. This also means that setting the mic to plughw:CARD=seeed2micvoicec,DEV=0 is a bad idea, since that only works for the 2-mic hat. You should instead advice to set it to the device that has seeed in the name because all of them should have that in common.

Also put a warning somewhere that every kernel update can break respeaker drivers, either because the kernel got to new or because you need to switch the repro fork to a newer version for any mic that isn’t the 2-mic hat. For best user experience I would suggest you ask users to exclude kernel updates in their package manager once they have the mic working if they don’t know how to or don’t want to manually check if the newer kernel version is supported yet. Actually, suggest they do so anyway and only undo this if they want to upgrade the kernel and have either checked if their branch is still compatible or they uninstalled it before the upgrade to install the new branch. Otherwise you will get quite a few linux beginners that did a system update and suddenly the mic broke.

The next thing I would recommend is adding a small tutorial on how to use alsamixer to adjust the mic gain and level and how to save that so the driver won’t override after reboot. The default settings for this are pretty bad in most cases and make using the mic for rhasspy hard. Gerneral advice is turn the volume up to something between 70 and 87 and the gain to 50-80, depending on how far away you plan to use the mic from. They unhelpfully removed this and the command for saving the config from their readme a while back, but I don’t see any commit that fixes this or saves automatically. The command to save is sudo alsactl --file=/etc/voicecard/ac108_asound.state store for the 4-mic square and line as well as the 6-mic, the 2-mic differs and uses sudo alsactl --file=/etc/voicecard/wm8960_asound.state store. Which file to save in depends on the chip used on the mic array, so if they ever add something with a different chip, this is where one needs to look.

Respeaker mics normally come with inbuilt leds, so it would be a good thing to either integrate hermes-led-control into your project or at least linking to it so ppl with respeaker hats can use their leds also.

And with this writeup I think you understand why the general advice in this forum is don’t get a respeaker that isn’t a 2-mic, because you get less work and at least the same quality with literally everything else. I personally still use respeaker 4-mic-square (or more correct, a clone of it) but I had quite a few headaches with it over the 2 or so years I used this. They do their job and the inbuild leds are nice to have but the official support is less than optimal and if you can get anything that is supported by the kernel without extra drivers you are far better off.

3 Likes

Cool! I think fans of ha might persist with this if satellites were easy to add, as it then can become part of a whole house automation.

The android client means being able to use old mobiles and tablets in multiple locations without going into setting up satellite hardware etc. An interface that adds clients would simplify this and make it more attractive to ha users as an alternative or supplement to big voice assistants.

Just means having a way of setting additional port numbers in the docker config and site IDs for each client and the relevant port number.

Oh wow, that is a lot more work than I figured! Thanks for the writeup. For now I might just put a warning on the docs about some of the issues you’ve outlined and link to your post. I may end up revisiting adding full support for the ReSpeaker later in the future, but it seems like quite a bit of work to get it going.

Again, thanks for calling this out!

Thanks Daenara, it mostly echoes my findings and adds some missing pieces for me.

I understand that HA and Rhasspy are trying to be all things to all people, supporting any combination of hardware and choice of modules at each stage of the voice processing. Great job … but means there is no one tutorial, not even one clear path for beginners to follow. Add that FOSS documentation tends to be technical, and it is left to out-of-date YouTube videos and blog posts to provide the user level. Too many things a beginner has to get their head around, in order to get something running.

I have got a satellite RasPi3 with reSpeaker 4-mic HAT and Rhasspy working with a base RasPi4 with Home Assistant OS and Rhasspy add-on. It’s not hard - once you know what you’re doing. Adding HermesLedControl and MQTT broke it, so try again.

Currently wanting to add another satellite which will be RasPi Zero W with reSpeaker 2-mic HAT, Rhasspy and HermesLedControl. This seems like such an obvious combination that I am surprised there isn’t already a beginners level tutorial; or it been packaged as a commercial product (like the RasPi & KODI media centres). … so I am documenting my build step by step.

I am already using HinTak’s drivers, but will have to add your warning about newer kernel versions (and research how to prevent them). I’ve used alsamixer in testing, but need to save and restore settings. Adding HermesLedControl seems to want shared MQTT - another thing to learn :wink:

I am working on that, time is however limited.

Currently i try to get an rhasspy satellite to work with the hass.io rhasspy and mqtt addon.
I found out, installing hermesledcontrol using the install script also installs the respeaker driver, mosquitto and some more, so you dont have to do it yourself.

But i do not get the leds to light at all (wrong mqtt settings?)
I had an working satellite which stopped audio output after plugging the audio jack out and in again while running. I dont get it to say anything after that.
I tried to update the OS and rhasspy docker to 2.5.11
After that i had sound again but with a very bad high noise.
Then my SD card got in an write only state without informing me (looked like i could write/flash/… but do not really change anything).
I made a new card with raspios lite, copied ~/.config from the old one, installed hlc with driver/mosquitto. and run

$ docker run -d -p 12101:12101 \
      --name rhasspy \
      --restart unless-stopped \
      -v "$HOME/.config/rhasspy/profiles:/profiles" \
      -v "/etc/localtime:/etc/localtime:ro" \
      --device /dev/snd:/dev/snd \
      rhasspy/rhasspy \
      --user-profiles /profiles \
      --profile de

Same terrible noise.
After changing it to

$ docker run -d -p 12101:12101 -p 12183:12183 \
      --name rhasspy \
      --restart unless-stopped \
      -v "$HOME/.config/rhasspy/profiles:/profiles" \
      -v "/etc/localtime:/etc/localtime:ro" \
      --device /dev/snd:/dev/snd \
      rhasspy/rhasspy \
      --user-profiles /profiles \
      --profile de

surprisingly the noise disappeared.

But it seems something is missing for mqtt. I do not want tu use the internal rhasspy mqtt but the one from homeassistant. So i should not need port 12180 but the one from profile.json should be used.
No idea why its not working.

found the problem: https://github.com/project-alice-assistant/HermesLedControl/issues/112
import logging had to be added to the top of /home/pi/hermesLedControl_v2.0.15/models/LedPattern.py