Rhasspy mobile app

Development
1

I state that I am not an experienced programmer but I still wanted to help with the development of this beautiful project. I started developing a prototype mobile app for Rhasspy, for now, interface with the rest api but later I would like to expand it for MQTT support. It made it with flutter to have multi-platform support but for now, I can only compile for android given the lack of a physical macOS device. Feel free to comment and recommend the best solution to implement a mobile app. Excuse me for my English.

12 Likes
2

What I would really love in regards to Android apps are two thing:

  • Being able to turn an old smartphone into satellite
  • Being able to take out my phone whereever I am and submit a voice command, ideally trhough a fast accessible widget
3

I managed to run the app on an old samsung phone running android v5.1.1 all functions worked including mqtt.
Thanks for the idea of the widget I will try to implement it. in the future I would like to add the possibility to activate it through a wake word.

4

Cool. I never created an Android app. Quite interested but too busy to learn it.

5

I really love your app. It’s working really good in my setup and it’s way better than screaming snowboy or other wake words in my living room.
I’m excited to see how the wake word implementation will work.
I’m using rhasspy with home assistant and returning text that will be spoken through the audio jack from my raspberry is nice.
Is it somehow possible to have that text spoken by the app? As it says Speech to text in the feature list. I tried some rhasspy configurations but I can’t get it to work.
It it’s not possible this feature would be quite nice especially when using the whole thing in different rooms.
Keep up the great work :slight_smile:

6

If you mean that the ability to speak text when there is no active session (so when you didn’t click the microphone) there is the possibility to use hermes/dialogueManager/startSession by sending the payload


{
"init": {
"text": "some text", the text to be spoken
"type": "notification"
},
"siteId": "siteId", the siteId in which the text will be spoken
"customData": null,
"lang": null
}

so the text will be pronounced by the app if the device has active screen on otherwise Android disables for apps the ability to access the network to save battery but you can fix this by changing its battery-saving settings for the app.

About the Wake Word soon i want to release the new version that will include the ability to use the various wake word already available on rhasspy thanks to the ability from the app to share audio through UDP.

1 Like
7

That was a fast response :slight_smile:
I can set a text in the homeassistant intent which will then go through the text to speech and played by the jack output of the pi.
For example: What’s the temperatur in the the living room? -> Pi starts talking: There are 22°C in the living room. But that speaker is located in the living room. So having this audible feedback from the mobile would be great.
So Im clicking the microphone. Speak the request and will get an audible response from the mobile.

8

This feature is implemented I use the app with home assistant, send the intent through the events then thanks to Appdaemon send an Endsession or Continuesession after that the text is spoken by the app. If you can share additional information about your configuration so I can understand what is the problem.

9

Hey I really like your app, so far its working on my end :). But I wasn’t able to figure out how the wake word feature works
 I have started the wake word feature but when I use the wake word nothing happens. Do you have a guide or sth that can help me out here?

10

For now the wake work consistenze only in send udp packet contents audio data so the detection is not done locally, but in this way you can use all the wake word detectors available in rhasspy just insert in the app ip and the port to which you want to send the audio and on Rhasspy add in UDP Audio (Input) ip (of machine where is rhasspy):port:siteId. if you are on docker you have to open an additional port before run the container “-p port:port/udp”.

1 Like
11

Hey @razzo04 congratulations on the wonderful idea and application. Could you please share your Rhasspy setting (profile) or or some simple guide? Im trying to use your app, rhasspy and node-red. App connected to rhasspy api and to mqtt without issues. Speech to text working, but i have a problem with intend handling and text to speech.

EDIT: When i set siteid as default in the your application, i can hear a voice output from the Rhasspy, but still get error message “no one managed the intent
”

12

I added an explanation for how to configure the app. Be patient with my bad English.
However the app gives the error “no one managed the intent” if within 4 seconds after receiving the intent doesn’t receive an endSession or continueSession message from rhasspy.

1 Like
13

Thanks for explanation, i have to check wheres the problem with endSession or continueSession. Maybe its because i use api/text-to-speech for responses in the node-red.

14

Yes it is possible, I have never tried the app with node-red and api/text-to-speech since I used python and MQTT but in the future I will run further tests to improve compatibility.

15

Did you try to implement local wake word?

16

I haven’t tried yet, but in the future I’ll try.

1 Like
17

@MihataBG
I noticed that with the latest versions of rhasspy has been modified messages
endSession and continueSession removing the field siteId so sometimes the app showed
the message “no one managed the intent” even if this had arrived correctly. Let me know if the problem is solved with the latest version of the app.

18

@razzo04 With the latest version of app and version 2.5.6 of Rhasspy im still receiving the message, but i think that the problem is from using api/text-to-speech. Maybe i have to check how can use python and MQTT in my case. What topic you use for text-to-speech?

19

I don’t send text-to-speech manually but sending from AppDaemon endSession or continueSession rhasspy takes care of sending the request for text pronunciation. if you want to send the request manually you can do it through the topic “hermes/tts/say” and as payload

{
"text":"",
"siteId":"" // must be the same in the app settings
}
20

Hi, I really appreciate your work. But I’m having the same issue like MihataBG. I have to type in the siteid of the rhasspy base in order to get tts to work. My assumption is that if I use nodered (or the web UI), the message first goes through rhasspy and because of that has the siteid (in my case) “base” and not the one the app would react to. Does anyone knows a way to tell the tts service what siteid it should use or am I understanding something wrong?

21

browsing the code I found that the text-to-speech api accepts the siteId as a parameter so you can specify on which satellite to play the message through the url http://localhost:12101/api/text-to-speech?siteId=yourSiteId.

22

Thanks, that worked well. It’s all written in the docs you just have to know what to search for.
Here is a list of possible http endpoints:

23

Thank you sharing your wonderful aplication!
Since updating to rhasspy 2.5.7 I have recognized a problem with german “umlaute” in the mobile application.
Instead of “wie spĂ€t ist es” “wie spÀt ist es” comes out of the voice recognition, therefore the intent recognition fails. With a previous version of rhasspy I had no problems with “umlaute” in the mobile app.
Do you have any ideas on this problem?

24

I avoided that problem from the beginning by avoiding Umlaute at all. My slots and sentences in rhasspy all use ue oe ae instead of ĂŒ ö Ă€ :slight_smile:

I have to try this app very soon. Might be super useful.

25

I don’t know I’ve never tried the app with german language since it’s not my native language.
Does the problem disappear with previous versions of the app or does the situation remain unchanged?

26

Yes that workes. Thank you for this tip. Simple and effective :wink:

1 Like
27

Thank you for your respond! This issue seems to relate to the rhasspy server (base) 2.5.7. Switching the server (base) back to 2.5.5 let disappear the issue, without touching the mobile app.

With the tip from @HorizonKane i can avoid the problem and for me everithing work fine.

28

Hi, I’m trying to set up the udp wakeword detection for hours now but I can’t get it to work. I followed the rhasspy docs but it isn’t clear for me what to do. Could anyone help? Thanks

Edit:

Thanks for the answer,
I’ve reached my reply cap on this topic cause I’m new to this forum. Is there a way to pm you?

  • I’m running on docker and opened the port 12202 (looking at portainer)
  • in the app I entered the ip of the rhasspy base : port (192.168.178.40:12202)
  • in rhasspy base under audio recording I selected pyaudio and entered ip:192.168.178.40 port:12202
  • in rhasspy base under wakeword I selected porcupine and entered the port 12202
    but it still doesn’t work.

One thing I noticed is, if I enter 192.168.178.40:12101 in the app at the first line (not under wakeword)
I get a cannot connect with rhasspy message

29

This is my old post, let me know if help.

1 Like
30

This is odd; I would have expected the opposite. The JSON coming out of Rhasspy before 2.5.7 was being auto-converted to ASCII by Python’s json.dump and I had to set ensure_ascii=False everywhere.

Which STT and intent recognition system are you using?

31

Hi Syntox.

this is a thread about the rhasspy mobile app. You should open a new thread for your problem and will get help for sure.

32

Ok, I did. Then I just ask about the last part.

33

Sorry Syntox I misunderstood your post. I thought it was about general rhasspy problems. My mistake.

34

I am using Kaldi with Fsticuffs

Also note: The 2.5.7 raspi satelitte is not showing this issue (only the mobile app).

1 Like
35

I am trying to use the mobile app, but am running into a problem - I can use the speech to text in the app and it will transmit to my rhasspy (I see the intents in the app and my speech in the “recognize” field in the rhasspy web interface), but it doesn’t actually execute the intent (I get a “no one managed the intent: switches” on the app). I have Handle checked in both the app and the rhasspy web interface. If I press enter on the recognize field in the web interface it works. Any ideas on why the intent would not be executing from the app?

36

Probably with the modification happened in rhasspy 2.5.7 in which don’t force anymore use of ASCII character, the system that used previously to convert bytes to strings complained about the presence of not ASCII character now this problem should be solved with the latest version. Let me know if the problem persists.

1 Like
37

What system did you use to handle the intent?

38

Yes, with 1.6.3 the issue is fixed! Thank you very much!!!

39

I’m using Home Assistant as the intent handler

40

Thank you for helping me find bugs.

41

Strange, I also use Home Assistant as the intent handler. Did you have other rhasspy satellites worked? Did you manage the intent with events or intents?

42

I just tested a rhasspy satellite and it worked properly with the exact same siteId as the mobile app (the command shows in the base rhasspy webpage recognize box and the intent actually executes).

I am using Home Assistant event handler to handle the intents.

On the mobile app, I have the MQTT server configured, and the check is verified.

I can’t seem to put in the Rhasspy IP (IPADDRESS:12101) itself in the app settings, it says “cannot connect to rhasspy”.

The standard rhasspy satellite has no problem connecting to the same endpoint.

This may be the issue, any idea why it wouldn’t be able to connect?

I’m running rhasspy 2.5.7 and mobile app 1.6.3.

43

@razzo04
I’d like to start/stop recording with a phone shake.
Im trying to start a broadcast intent with automate but im failing miserably. do you know if its posible and how?
start/stop with a shake would be superb as it doesn’t need constant streaming for wake word detection.

44

I tried to use the automate app and it works instead of using the broadcast intent I used app start so that if the app is not running it will start and should start listening for command and if it is running it should just start listening. You must add in the parameter extras {“StartRecording”: “”} otherwise the app will only start and will not know that it must start listening. However, if you want there is also an android widget that does the same thing as app start.

image
image738×1600 105 KB

45

Hi Razzo04
thanks for you help. the point is start rhasspy without unlocking the phone but unfortunatelly it isn’t working.

now im having the same issue than RandomRhasspyUser

I can’t seem to put in the Rhasspy IP (IPADDRESS:12101) itself in the app settings, it says “cannot connect to rhasspy”.

and the “no one managed the intent: hassturnoff” issue too

mqtt msgs shown :
:

Captura
Captura473×504 11 KB

rhasspy configuration:

Captura2
Captura2440×596 18.9 KB

46

Android prevents an application that has no graphics element on the screen from performing certain tasks such as recording audio especially if the phone is locked to overcome this the app must have a Foreground notification at the moment there is no direct setting in the app that allows this but you can enable wake Word select on UDP and enter 127.0.0.1:20000 so the UDP packets will not be sent but will remain in the device. Concerning the problem that intent is not being handled, I suspect that the problem is that rhasspy is not sending the intent to home assistant. under the section Intent Handling, do you have inserted the siteId set on the app?

47

Hi, great app. I have the mobile App got working with nodered. At the moment i can click on the microphone symbol and talk and have to click again when finished. That is working already and i can switch on and off an wled stripe. Great for now.
But how exactly can i use wake word from your app? I dont understand your explanation in this thread about wake word using. I have a raspberry PI 3b with hypriot and rhasspy (port 12101) and nodered (port 1880) as docker Container at 192.168.0.54. A second raspberry is running mosquito mqtt broker at 192.168.0.30:1883. Can you explain to me what udp port and ip i have to choose in your App and in rhasspy to get the wake word function working? I use porcupine in rhasspy for wake word. In rhasspy i have german profile.

48

You can find more information in this post.

49

Hi there,

razzo04 thinks about an smartwatch App so we can use a Watch as Satellite. See here:
https://github.com/razzo04/rhasspy-mobile-app/issues/4

With this App we have a Satellite on our Wrist. Perhaps anyone has an Watch with Wear OS and Microphone, so we can help to develop. This time i have only a Pebble Time, but i will order a Wear OS Watch.

Greetings

50

I just installed the Mobile App but I realized the Settings are not loading, the wheel keeps spinning.
I both built it my self and also downloaded the app, I tried both.

51

What version of Android are you using?

52

Congratulations razzo04.

Your App is great!!!

I’m using it to control Domoticz with my Android Phone and a Node-Red structure to recognize the intents and execute commands via http responses to the Domoticz API.

For the moment is working fine. Now I want to to try with old Android phones placed in several rooms with the power supply connected and using wake up word.

My question is about the satellite siteId. I don’t know how to include the siteId in Rhasspy Mobile App to differenciatte bewteen them to send the audio from Rhasspy without active session to different mobiles.

Thanks in advace.

53

If you have any conflict with the siteId generated by the app you can change it manually by editing the text field “Siteid” and then you can click again on auto setup.

54

Thanks for the info razzo04. I’ve just realized you have released a new version and I don’t know why but now changing the siteid inside the app is working. With the previous releases, in order to get the recognition and the tts working I needed to have the same siteid name of the Rhasspy Host.

Is it possible to have some tts messages to some of the satellites and “a common” channel to be received by of them?
Thanks

55

The only way to play a tts message is to send to each satellite the message with the siteId set on the app, so there is no way to broadcast a message.

56

ok, so the broadcast alternative would be to create a nodered flow with the same msg to the satellites. Something like this:
imagen

Using the tts API and http Post Request is working fine:

http://192.168.1.204:12101/api/text-to-speech?siteId=Tablet
http://192.168.1.204:12101/api/text-to-speech?siteId=Mobile

But

when the tablet or the mobile are locked , the payload is received but there is no audio :frowning:

The tablet and the mobile will be connected to power so I don’t care about battery.

How could I have the App running all the time?

57

Android greatly limits the ability to run the app in the background and to access some resources like a microphone but theoretically if you enabled wake word should have appeared a notification warning that the app is running in the background so the app should operate even when the screen is off.

58

Hi @razzo04 I’m using stock MIUI ROM Global 12.0.5 stable and android version 10 QP1A.190711.020 ON xiaomi redmi note 8 pro

59

Very strange. I tested it on Xiaomi redmi note 9 pro and it worked. could you share the app logs should be in Android/data/com.example.rhasspy_mobile_app/files/logs.txt. Does this problem occur with previous versions?

60

Hi @razzo04 ! I just wanted to thank you for creating this app, it’s exactly what I’ve been looking for, and I’m eager to try it. If you don’t mind me asking, what are your future plans for the app? Are you planning on continuing development? Thanks!

1 Like
61

Now I’m focusing on developing a skill store for rhasspy so I don’t have much time to continue developing the app but once I have done I will try to improve the support for wear os.

62

Sounds good. If you have the time, I’d love some help on the initial setup for the app. Everything has gone great, except I can’t seem to access the satellite’s web UI as per the documentation. Any suggestions? Thanks!

63

Sorry @razzo04 , not sure if my last post sent a ping. I’ve gotten the app all up and running, however I can’t connect to the satellite’s web UI with http://IP_ADDRESS:12101 like the documentation advises. Any suggestions? Thanks!

64

I think you have looked at the official rhasspy documentation some settings are similar but the app doesn’t require a web UI because all available configurations can be edited in the app settings. If you want to get started I suggest following the guide here.