Assist + Rhasspy working together

So we’ve all seen the “Year of the voice” release from HA which is super exciting, I’m really happy with everything they are doing for the software. It’s amazing.

Why is no one talking about the marriage of this HA release with Rhasspy!? It’s freaking awesome! The biggest thing HA lacks is wake word functionality (I know it’s scheduled for later this year) … Why wait when you can do it now! Rhasspy already has it covered. The mixture of the 2 is really powerful. Rhasspy grabs your command, converts it to intent, sends it over to HA, the HA does the actions and sends TTS to piper - no need to send back to Rhasspy. I’ve even got confirmations setup to come through my Sonos speakers. No need for a separate speaker just for the voice activation box, use the ones you have!

Don’t get me wrong, I look forward to HAs next release because the UX on Assist is damn near flawless. Making intents and creating action is stupid easy. The Alias additions are my personal fav.

Has anyone else tried it?

OMG…
I’m new to Rhasspy and newly installed it on my machine. Then I see you need Home Assistant to really put it to use. For my reference is HA the same as Home Assistant, just shorter?
If it is I feel like Homer Simpson right now. And doubly so if HA comes with Rhasspy. That could have saved me alot of time I’m betting. Being that they are co-dependent.

Hi, welcome. Yes, HA is for Home Assistant. They are not co-dependan and HA does not come with Rhasspy installed. HA is generally the brains of your smart home and you connect everything to it.

HA has hundreds of add-ons and Rhasspy is just one you can use for voice control. HA does all the actual heavy lifting of automation and connecting iot devices. HA just realsed a built-in voice control called “Assist” which is actually quite good but has no wake word detection. It requires you to tap and hold a button of some kind to listen to you.

Don’t stress, you can install HA and Rhasspy on the same device. Their both really easy to install. Setup can take a bit of learning but their both really great tools.

Actually you do not.
But what you do need is some intent handling, otherwise the things you ask or notting getting a response.
You can add your own software as intentHandler, maybe some other smart home software

https://rhasspy.readthedocs.io/en/latest/intent-handling/

Thanks for clearing that up… This coding/scripting is alot like the military. Everyone has an acronym for everything… So is ‘Hass’ Home Assistant as well?

Thx

romkabouter,
Thanks for the reply! JeffC saw that you reached out to me and said I was in good hands as far as advice goes… He did suggest i install docker then HA OS as you suggested. But I thought Docker was like if you are on a Windows machine and need to partition it to run a Linux based program…
I looked at the HA site and instructions for Linux/Ubuntu they have 3 different images. VirtualBox, KVM, and Vmware Workstation… and I am not 100% which I should go with.
Thx.

I have an Intel Nuc with proxmox.
Proxmox is virtualisation software running on a linux base. Easy to install on a device like a NUC or the Dell hardware you have.
You can read something about it here

100 questions incoming!

Are you still using rhasspy for wake word? Also, what sort of processing speeds are you getting with HA assist? Are you using vosk? Gpu?

Thanks man! I’m trying to get a sense for the best way to set this up. Right now I’m running rhasspy and it’s great. I’ve just tried HA assist and it was 13 second waits. Probably because I have a non AVX CPU. But even so, I’ve heard it is slow. If I could figure out where to put my money into something that can run HA assist fast, I would…but I also don’t want to be forwarding gpus through proxmox and what not.

A lot has happened in the past 11 months with Home Assistant and Voice Assist.
Rhasspy 3 … not so much, but just last week Mike re-stated his intention to update Rhasspy 3 with things he has learnt from Voice Assist.

If you are using Home Assistant I suggest you do searches on the HA forum.

I started with HA and Rhasspy all on a RasPi 4 (which runs surprisingly well), but am now running HAOS in a VM under proxmox on an old i5 business PC (with old basic video card because it runs headless); and for a satellite using the same Raspberry Pi 3B with USB mic and speaker I used with Rhasspy 2.11 (now running wyoming-satellite and wyoming-openwakeword). I also have to wait almost 15 seconds after I stop speaking before Voice Assist recognises the silence and starts to process my command - but after that it’s pretty fast. I assume there is a Voice Activity Detection setting I have missed.

I just started with Rhasspy and I love it. I have tried to handle the “No intent recognized” but can’t. Why I want to do that? I want to call chatgpt from HA when the intent has not being recognized by Rhasspy. Anyone knows how can I do it? It is possible? I do not see any config to send the intent to HomeAssistant if it is not in Rhasspy. Thanks!

Are you using Rhasspy 2.5, Rhasspy 3, or Home Assistant’s Voice Assist version ?

I recall forwarding unrecognised intents to ChatGPT being discussed previously, but it wasn’t something I personally followed.

I am using Rhasspy 2.5 in a Raspberry Pi. Didn’t find the unrecognised intents topic in here. Did you find it? Thanks

That’s probably because it was over in Home Assistant’s community :wink:

If you are using Rhasspy with Home Assistant, I strongly recommend you change your focus to HA’s Voice Assist.
Rhasspy 2.5 is great (and i still use it for my production Home Assistant setup) … but Mike has applied his Rhasspy v3 improvements (and more) to Voice Assist … and that’s where the bulk of development effort is currently.

Ok thank you for taking the time to explain to me all of this. It really helps. I was about to setup everything with Rhasspy because it looks stable and very fast. I have tried to use HA Voice Assistant and never worked fine. Defining fine: I use it in Spanish instead of english. No idea if that is related to the issue but most of the times, Voice Assistant does not respond. I have HA installed in my Raspberry PI 4. Everything works like a charm. I use HA for around 8 years or more. Sadly Voice Assistant never worked fine to me. I will give a new try then. But again, Voice Assistant and all the setup it needs for recognize a voice and etc has been a pain in the time…
And Rhasspy 3 is only in english. Which does not make any sense.

Rhasspy 2.5 is stable, fast and works well … but is now a few years old.

Mike started developing Rhasspy v3 incorporating new ideas he had, and it was only released as a preview … I would recommend that you skip it.

Most Rhasspy users were using it with Home Assistant, so it was a “no brainer” when Mike was employed by Nabu Casa to integrate Rhasspy into Home Assistant. I for one have been surprised how much time was spent building the framework in HA for Voice Assist - but the tight integration has been worth the wait (even though it has meant more learning).

A pity that you have had problems with Voice Assist … but I am also not particularly surprised. Home Assistant supports so many devices and methods - and Voice Assist is no different. It doesn’t help that HA’s documentation standard is more like developers internal notes, than intended for users :frowning:

Personally i consider Chapter 5 early this year was when Voice Assist became a viable alternative to Rhasspy 2.5. Still more work to do - and plenty of extra features have been suggested (like the fallback to chatGPT, and multiple concurrent languages).

How long since you tried Voice Assist ? I certainly found it confusing trying to set up … hence my post to add bits of explanation I felt were missing. Are you wanting to run Home Assistant and Voice Assist on the same Raspberry Pi 4 ? I understand they can run together … and there are Voice Assist options which don’t require so much processor or memory.

Languages other than English do add complexities - though I expect that Spanish would have a reasonable size user base to help identify and fix any issues - but I think Voice Assist not responding may be more likely due to the microphone and recording setup.

Thank you once again for taking the time to write and respond. I have tried using a Voice Assistant, but it doesn’t seem to recognize Spanish very well. For example, when I say “turn the room light off” (in Spanish), the command fails. The debug output shows a misinterpretation: “turn therum light off” (in Spanish). I have experimented with different phrasings, but none have been successful. Text input works fine, but unfortunately, voice recognition in Spanish leaves much to be desired.

Despite being a developer, I’m looking for a more out-of-the-box solution. Home Assistant is fantastic, but I think there needs to be a stronger focus on improving the UI/UX—the user experience is not as good as it could be. Nonetheless, I adore Home Assistant. It has significantly enhanced my life, especially with its stability over time, which I greatly appreciate.

What I really want is an intelligent assistant that doesn’t require setting up each individual intent. Intents are outdated—Alexa uses them, and they feel deprecated. We should have an AI assistant that understands commands effortlessly, whether it’s turning on the bedroom lights (and discerning which bedroom if there are two by using proximity sensors, phone location, or even voice detection) or fetching a recipe, directions, or even trivia like who the best soccer player in the world is (obviously, Messi).

So far, as a developer, I’ve created a middleware that bridges Rhasspy with Home Assistant (HA). Instead of using HA intents, I send commands directly to my middleware, which is connected to an AI (ollama) and has full knowledge of all my devices through the HA status endpoint. For example, if I say “turn on my garden lights,” the AI understands this command as referring to both ‘garden1’ and ‘garden2’ entities—I don’t need to create numerous specific intents.

I chose Rhasspy because, despite my attempts, I struggled with Python; I am more proficient with Node.js. Unfortunately, the Python code I wrote had some issues, so I stopped pursuing it. Rhasspy handles what I need it to do, though it cannot process commands without predefined intents yet. I will continue working on this integration, aiming to make it simpler and smarter.

Please let me know if you have any questions or concerns.

Your approach is what I think a home assistant should do. Congrats for having succeeded in this way, a home AI. Do you plan to describe what you have done and how you proceed exactly ?

Hello @JanWolf - Here’s a detailed overview of my latest project. I’ve successfully integrated Rhasspy with a custom-built middleware called RhasHomeBridge, enhancing our voice-controlled environment. This middleware acts as an intermediary, managing both Intent Handling and Intent Recognition processes through remote HTTP requests directed at RhasHomeBridge.

The system enables complex commands, ranging from basic requests like “turn on my living lights” to more intricate interactions such as narrating a horror story. Depending on the command’s nature, RhasHomeBridge determines whether to forward it to Home Assistant (HA), Ollama, or ChatGPT.

One of the innovative features is the system’s feedback mechanism. If a response from Ollama is satisfactory, I can command the system to retain this information. Conversely, if an action fails, the system is instructed to log the error and adapt to correct itself in future interactions.

Significantly, this setup eliminates the need for manually writing out intents or managing multilingual translations—Ollama handles these complexities seamlessly. The configuration is streamlined to avoid extensive setup, making the system more user-friendly.

My aim is to reduce the need for constant configuration and to move towards a more plug-and-play approach. I envision a future where AI handles more of our routine tasks, minimizing the need for detailed configuration files. For optimal performance, while Ollama can run on a Raspberry Pi, a more robust hardware setup is advisable to enhance processing speed and efficiency.

Ultimately, this project represents a step towards creating a personal AI ‘pet’ that seamlessly integrates with various devices, providing a smarter, more connected home environment.

Nice but could you publish details on your middleware you called RhasHomeBridge. Maybe you plan to make it available on github ?

Yes, I will once I finish the integration. Now I am trying to make the ReSpeaker to work with Rhasspy which I couldn’t yet. Any guidence on that you have? Thanks!