Rhasspy, MQTT, node-red, proxmox, raspberry pi3, nuc - Part 1

Show Us
1

First I’d like to say how great Rhasspy is.
I have only been playing with it for a couple of weeks but it is the easiest install and configuration I could imagine. And the documentation is impressive.

So down to my system.

To start with I have a fairly mature HA system that comprises the following hardware:

  • main system is on an Asus Tinker Board with 2GB RAM. This runs node-red, openhab for Z-Wave devices, Influxdb and Grafana for charting and node-sonos-http-api and socos for sonos
  • A raspberry pi running deconz for Zigbee devices
  • A sonoff rfbridge433 with Tasmota for rf remote controls (using only as a receiver at the moment)
  • Broadlink rmpro mini for IR
  • Daikin Airbridge for HVAC control
  • Unifi Controller and nmap for presence detection - mobile phones only at present
  • Yeelight light strips
  • Xiaomi gateways - mainly replaced by deconz but still used for status lighting
  • Sonoff ifan02 for ceiling fans with Tasmota
  • A nuc with proxmox with a container for the Rhasspy server.
  • A raspberry pi as a Rhasspy satellite

Node-red is the heart of the system which made integration with Rhasspy a breeze through MQTT.

In the couple of weeks I’ve been playing with it I have so far managed to integrate:

  • 73 Lights, 72 power plugs/wall sockets, 11 fans, 21 cover devices (shutters, curtains and skylights)
  • Report on the status of 60 door and window sensors
  • Arm/disarm devices
  • Control groups of items, like saying “Turn on all lights in John’s Room” or set all lights to red.
  • Report who is home
  • Ask for the status of almost any device, like Is John’s light on (and get a yes no answer) or What is the status of John’s light, door, window etc
  • Ask what is the weather and get the Dark Sky summary for my area
  • Ask things like the time, date, temperature (from Dark Sky), the temperature/humidity in any room in the house.
  • Control my Sonos system, ie Play/Pause, volume control, etc, select tracks by Artist name, Playlist, Favourites and also ask questions such as the current artist name or track details
  • Add things to a shopping list and have them read back

to be continued…

5 Likes
2

Part 2

For Rhasspy server I am using a container configured on a 3rd Generation i3 Nuc.
I have allocated 10GB for memory which it sometimes needed during training, but with a bit of tweaking it doesn’t seem to need it any more.
It is setup to use MQTT, webrtcvad, kadi and fsticuffs. Everything else is diabled.
The container is running Ubuntu 19.04 and Rhasspy is installed using the .venv method.
It also has an instance of node-red running.

For the client I have only tested so far on a Raspberry Pi 3b+ with a Docker installation.
The microphone is a PS3eye.
This also has MQTT. It uses pocketsphinx for wake word and remote STT and Intent recognition back to the server.
The SiteId is set with the same name as the Room name as defined in node-red. This is also the prefix used in my system for all devices in that room.

The HA node-red instance is set up to subscribe to the MQTT intents and then handles the processing.
It sends the TTS through the Sonos speaker via node-sonos-http-api to the speaker in the room with the client. node-sonos-http-api has a say function that will pause playing music and resume after the response is sent.

3 Likes
3

Part 3

In node-red I have written a function that goes through all the devices it knows about and generates a number of slot lists based on the type of information and/or device.
It then publishes the list to the MQTT broker on the Rhasspy server (I use mosquitto) where the local node-red instance writes them out to the slots folder and subdirectories, so any change in the HA system will be passed on to Rhasspy. If any changes are detected, node-red then uses the API to retrain the system and then restarts the clients.

Training currently takes about 6 1/2 minutes.

Things to do:

  • HVAC commands
  • Replace pocketsphinx for Wake Word
  • Alarms and Timers
  • more, tbd :grinning:
4 Likes
4

That is a huge system! Good work

5

Impressive, thanks for sharing this and inspiring others!

6

Very cool :sunglasses:

Is this on 2.4?

7

Yes, I have been playing with 2.5 in a test config but have been having trouble with my Pi Zero and respeaker combo.

I have moved the wakeword to Snowboy to reduce false positives.
I also implemented MaryTTS locally on the Rhasspy server.

1 Like
8

its nice and great…
i am new to Rhasspy and want to ask you about can we use rhasspy for home automation using only one raspberry pi ?

thanks in advance…

9

Honestly it would depend on the size of you system and how complicated you want to make it.

I might not be the right person to ask as my Rhasspy system is very large and complex, and would never run on a Pi. The rest of my home automation is split on an Asus Tinkerboard for node-red, influxdb, grafana and Openhab (which is only used for its z-wave binding and nothing else) and a Pi for deConz for Zigbee, although I think the Tinkerboard would probably have enough power to run this as well.

If you are talking just Rhasspy on a single Raspberry Pi 4 as a master and then satellites in rooms it may be able to handle it.
It might also work combined with Home Assistant or the like on a single Pi if your system is fairly small.

In my case I have about 80-90 sensors, 131 lights/light strips/powerpoints/plugs etc, 6 shutters, 6 curtains, 9 ceiling fans to control. Plus I have sonos speakers and my entire list of albums, artists, playlists and favourites imported as slot lists, plus my previous grocery orders as a list of things I can add to a shopping list. So mine is certainly way too large for a Pi of any sort.

I’m pretty sure there are a fair number of people on this forum running this on Raspberry Pi’s though.

10

I am currently running rhasspy, FHEM and node-red on my Pi4 with 4gb RAM which runs great. It also runs MaryTTS and a mosquitto mqtt broker.
I would recommend using atleast a pi4 with 2GB of RAM. If you want to run other things besides rhasspy aswell.

11

thanks …
i have completed rhasspy setup with docker in pi 3B+ successfully,
can we get recognized text intent output of Rhasspy in pi terminal for controlling gpio pins ?

thanks in advance …

12

If you know some Python, that would be fairly straightforward. For GPIO access, use one of:

And then you can let your Python program react to your intents with one of:

Let me know if you need some guidance.

13

You can also use nodered if your not the biggest python fan :see_no_evil: There you can listen for the intents with mqtt and switch gpio too but with nearly no programming needed. only prerequisite is that you shouldn’t fully hate javascript :wink:

14

thanks @koan and @JGKK …
First, I’ll try if i got stuck I’ll get to you @koan…
and thanks again…

15

hey @koan, i tried its been long time but i would like to ask for your guidance in command intent handler on same topic since its long time,
i remind you topic is to control pi gpio with rhasspy intents ?

thanks in advance … :grinning:

16

Well, you first have to define the intent you want to have recognized. How does the intent look like?

17

i want to set sentences as intent ( like turn on lights , lights please ) and other stuff and by intent i mean STT output of rhasspy …
and based upon that ( speech to text output ) wanna control gpio…

18

can we see any code you produced on this hughe project? I mean I am totally interested who you solved the shopping list skill.

19

Hi guys, I as well am wondering how to control the GPIO pins with Rhasspy. Very novice user and would appreciate the guidance! I have an 8 relay board connected to RPi 4 and would like to be able to control each relay individually with simple voice commands, offline and hardwired with no additional server or mobile app required. Is this possible to have everything required locally on the RPi (32gb card) and hardwired via GPIOs to the relay board for use in a garage? Really hoping for a fully local, disconnected, hardwired solution! Cheers :slight_smile:

And btw, wow amazing project @jrb5665!:ok_hand: definitely the most involved and ambitious I’ve come across. Very excited about the potential here with Rhasspy!

20

As you would gather there is a lot of code :slight_smile:
For the shopping the real key is populating the slot list file for words it will accept then sentences includes the following

[Shopping]
add ($shopping/shopping) [and] [($shopping/shopping)] to [(my|the)] shopping [list]
(remove|delete|cancel):delete ($shopping/shopping) [and] [($shopping/shopping)] from [(my|the)] shopping [list]
clear [(my|the)] shopping [list]
(list|whats in|what is in):list [(my|the)] shopping [list]

to generate the list of words I do the following:

  1. Setup a rule in my email to send the confirmation emails from Coles to a folder called Coles.
  2. In node-red use the email node to watch that folder
  3. Split the emails into individual lines
  4. Use a switch node to throw away unwanted lines
  5. I feed that through a function node with the following code to create a list in the flow context. It looks for the start of the items based on the content “ORDER SUMMARY” and the end when it finds “Estimated Total”
    if (msg.payload.includes(")"))
    {
    return;
    }
    if (msg.payload.includes(“ORDER SUMMARY”))
    {
    flow.set(“done”, false);
    return;
    }
    if (msg.payload.includes(“Estimated total”))
    {
    flow.set(“done”, true);
    return;
    }
    if (flow.get(“done”) === true)
    {
    return;
    }
    var prevordered = flow.get(“previousOrders”) || [];
    if (prevordered.includes(msg.payload))
    {
    return;
    }
    prevordered.push(msg.payload);
    flow.set(“previousOrders”, prevordered);
    return msg;

the output is then sent through a filter node that waits for 5 seconds of no more messages and then triggers the list of words to be created and sent to a nodered instance on the Rhasspy server via mqtt.

The code to create the word list is:
var words = [];

flow.get(“previousOrders”).forEach(item =>
{
var newitem = item.replace(/[^a-zA-Z0-9 ]/g, “”).replace(/[A-Z0-9]/g, function(x){return " " + x}).replace(/ */g," “).trim().toLowerCase();
newitem.split(” ").forEach(word =>
{
[“kg”, “g”, “cm”, “m”, “s”, “mg”, “ml”, “l”, “pm”].forEach(unit =>
{
if (word.endsWith(unit) && !isNaN(word.replace(unit, “”)))
{
word = word.replace(unit, “”);
}
})
if (isNaN(word) && (word.length > 2) && !words.includes(word))
{
words.push(word);
}
});
});
flow.set(“shoppingWords”, words.sort());

return {topic: “slotlist/shopping/shopping”, payload: words.sort().join(";;")};

On the rhasspy server I have a node subscribing to mqqt messages slotlist/# which receives any lists and if the list has changed it saves it into the slots folder structure and uses the api to trigger a re-training. Every now and then I have to manually apply unidentified words but given I populated the list with a couple of years worth of orders initially, most of the things we buy are already there.

Then in main nodered instance if it sees the shoppinglist intent it processes accordingly.

Similar logic is also used for all my slot lists including the lists of all my home automation devices, music playlist, favourites, album and artist lists which are extracted from Sonos using socos and a bash script on my home automation machine.

I hope this helps. Please let me know if you need more specific details in any area

1 Like