UDP Wakeword detection with Rhasspy mobile app

Help
#1

Hi, I’m trying to set up the udp wakeword detection for hours now but I can’t get it to work. I followed the rhasspy docs but it isn’t clear for me what to do.

  • I’m running on docker and opened the port 12202 (looking at portainer)
  • in the app I entered the ip of the rhasspy base : port (192.168.178.40:12202)
  • in rhasspy base under audio recording I selected pyaudio and entered ip:192.168.178.40 port:12202
  • in rhasspy base under wakeword I selected porcupine and entered the port 12202
    but it still doesn’t work.

One thing I noticed is, if I enter 192.168.178.40:12101 in the app at the first line (not under wakeword)
I get a cannot connect with rhasspy message
Could anyone help? Thanks

Rhasspy mobile app wake word configuration
Rhasspy mobile app
#2

You don’t need to set up a recording system on the rhasspy base as the app will provide audio data. on wake word you might try to put the ip of the docker container. however it is strange that the app can not connect to rhasspy even if this does not affect the wake word. To check if the app is not working or there is some other problem you can try to set a rhasspy satellite to send udp packet to rhasspy base.

#3

Thanks for the reply,
I tested again and for some reason the app can connect now but the wakeword still does not work. I think I’ve tried every combination of docker host ip, docker container ip and phone ip on both sattelite and base but it still does not work. Detection over mqtt works. Do I have to open a port on my phone too as the docs say:

For this case, set both microphone.<MICROPHONE_SYSTEM>.udp_audio and wake.porcupine.udp_audio to the same free port number on your satellite

#4

I try to explain the wake word configuration more clearly.

First, we open a UDP port to the container. Add in the docker run command “-p 20000:20000/udp” (in my example I will use the port number 20000) you must open a UDP no TCP port. So we can now check portainer.
image
In the app we enable wake word over UDP in the text field we put ip of the machine where rhasspy is running and after a colon the port we have opened previously. in my case 192.168.1.10:20000.

image
image1080×839 78.4 KB

in the rhasspy base under Wake Word in UDP Audio (Input) first put the ip address of docker container:the port we have opened previously: siteId which was set in the app. in my case 172.17.0.2:20000:telefonovirtuale
image
image1833×403 9.78 KB

now if everything worked by pronouncing the wake word the microphone symbol of the app should turn red indicating that it is listening for a command.

1 Like
#5

Thank you very much for that detailed explanation. Turns out that I entered everything right but restarting rhasspy from the web UI didn’t apply the changes. I had to restart the whole container in order to apply them.
Thanks for the help.

1 Like
#6

Why there are two different ip Addresses?
And one class c and class b private IP? I dont understand how this can work. I think both IP Addresses should be equal?

#7

Hi, i solved my Problem. I am new to docker and its Container thing, esspecialy how networking is with docker Container. I installed portainer, great Software, and then i understand were the class b ip addresses came from.
Wake Word detection is more or less good. I think i have to Play around with own models and train them. Sometimes i have to wait 4 seconds before the wake Word put on the mic red. If you speak wake Word blind, you do not know how long waiting until you can speak the command. But this is not the problem of your mobile App… Great work!

#9

Maybe in future versions I could make it possible to play a sound when a wake word is detected.

1 Like
#10

Hi @razzo04, rather than create a new issue I thought I’d progress with this one already in process on topic of the wakeword detection with Rhasspy mobile app. (great app by the way!). I have set everything up as defined above, but something is going very wrong…
I cannot get Rhasspy to wake via the app with the UDP wakeword setting. So, I have created a small Node Red to analyse port 20000 UDP traffic to make sure that the APP UDP message is getting through. I find that as soon as I “Start Wake Word” in the app, there are UDP messages flooding through to port 20000 on the Rhasspy Host machine. Can you think of anything that I might have done that could cause this please? I attach screen shots of the Rhasspy Wakeword entry, the App entry and the Node Red detected port 20000 traffic on the Rhasspy server. The messages starting 14:15:03 are the ones from the Rhasspy mobile App… thanks in advance for your thoughts on how to progress this one.

image
image613×834 10.7 KB

image
image
image693×692 29.4 KB

#11

Most likely you are using docker to run rhasspy, so Docker by default applies isolation between the host network and the container network so you can’t then bind the ip of the host machine instead you have to use the docker container ip or “0.0.0.0” which indicates that we listen for any interface.

#12

@razzo04. OK thanks. I’ll look into it further. I run Rhasspy on a RPi within HASSIO. Node red is also within HASSIO on the same RPi. Node red is therefore seeing the traffic from the Rhasspy mobile app UDP messages ok with the IP address of HASSIO / Rhasspy.

I am curious as to why the mobile APP is sending 15 messages per second by UDP. Is it expected? The wake word can’t be spoken in 1/15th of a second, and so could not logically be contained in one of these UDP messages. Its possible that I misunderstand the purpose of the UDP wake word function on the App in connection with the UDP messages, but I thought it was to listen for the potential spoken wakeword and pass it to Rhasspy.

I’ll post an update if I get any further by changing the IP address.

#13

If rhasspy runs on hassio some configurations are different, here you can find some suggestions.

The purpose of the UDP wake word is to start listening if wake word is detected. 15 messages are sent in one second because rhasspy expects to receive frames of 2092 bytes otherwise they will simply be placed in a buffer waiting for it to fill up, even if the wake word is on multiple frames it will be detected anyway.

#14

Hi,

I have everything working with multiple mobile app devices, but the wake word only works with the first device. Not sure if there is a simple formatting issue in the base udp in that I am missing, eg is it 172.17.0.2:20000:android-one, 172.17.0.2:20000:android-two, 172.17.0.2:20000:android-three I have also tried 172.17.0.2:20000:android-one,android-two,android-three

All three work fine when pressing the mic with the voice response to the correct device(satelliteiD). If I enter 172.17.0.2:20000:android-one, 172.17.0.2:20000:android-two the android-one device will wake, but no others. If I make android-two the first item, eg 172.17.0.2:20000:android-two, 172.17.0.2:20000:android-one android-two will work on the wake work, but android-one will stop working with the wake word. All continue working fine with the mic press.

Puzzled…

#15

Should be fixed in 2.5.11 and with other wakeword than raven.

#16

Am on 2.5.11 and 1.7.0 mobile rhasspy and using porcupine with grasshopper as a wakeword, and everything works fine other than the wake by multiple devices. So not sure if a bug or just a formatting issue in the wake word UDP input field. I know if there is an errant trailing space for example it will break, so maybe I have the format wrong for multiple devices. The multiple siteids is fine though.

#17

If you need to use wake word with multiple devices you need to open multiple ports on the container and use for every phone a unique port because the UDP stream is just audio(wav chuck) and so it can’t figure out which device is sending and this creates some random behavior. In the UDP Audio (Input) field you can put for example this: 172.17.0.2:20000:android-one,172.17.0.2:20001:android-two,172.17.0.2:20002:android-three

#18

Ahh, of course, many thanks!

#19

The concept of siteIDs are not well explained. But it can be figured out with a lot of trial and error.

In the Rhasspy Mobile App, the only place to set the siteID, is in the MQTT settings. Which is counter intuitive since the Mobile App was partially working with Rhasspy set to “internal” MQTT inside docker.
Something about the auto setup does configure it correctly to use the internal MQTT configuration over the standard Web API port of 12101. But it did not seem to set a siteID that would for the UDP wake word.

So I was stuck with UDP wake word, not working, until I set up an external MQTT server, and set up the Mobile App’s MQTT settings with the siteID.
What is more confusing is, before I set up the MQTT, I got it to partially work by following the documentation:
set both microphone.<MICROPHONE_SYSTEM>.udp_audio and wake.porcupine.udp_audio to the same free port number on your satellite

But my set up had USB mic connected to the host running Rhasspy, so it would interfere or break the wake word for the USB mic. At one point while following that bit from the documentation, it seemed to work. I could wake from both the USB mic and/or the mobile phone. But the siteID was the same, so waking from one, woke the other mic. And then it would barely recognize the intents.
The solution for my setup, was NOT to follow their instructions. But to just set up external MQTT, set a different siteID in the Mobile App, then put that siteID into the Satellite SiteID field for every Rhasspy service.