STT over MQTT Hermes and node red

[Gandalfand]

Hello,
i’m new to Rhasspy so first big thx for this solution.

So i try to manage an assisstant with node red (my domotic is managed with node-red to)

My question is about STT :

1. i use microphone under node-red dashboard
2. i transfert wave with mqtt hermes.
3. the answer is return with hermes mqtt

To do that under node-red i need to publish topic on mqtt to start :
hermes/asr/startListening
after send the audio frame : hermes/audioServer/default/audioFrame
and then stop the listing : hermes/asr/stopListening
its only after the STT answer with this topic : hermes/asr/textCaptured

1 question : Is there any solution to skip the start and stop listening and have an answer after sending the audio frame?

In fact node red wait the end of the captured soud to send buffer so i can’t anticipate the start.
2 question : Can i send TTS command with mqqt (because i send throught http request)

This is my node-red node screen :

This is my rhasspy conf :

MQTT : External
Audio recoding : Hermes MQTT
Wake Word : Disabled
Speech to TEXT : Kaldi
Intent Recognition : Disabled
Text to speech : PicoTTS
Audio Playing : Hermes MQTT

Thanks for your help!

[Gandalfand]

For the community this is my working flow (not optimised but working…)

[{“id”:“36326994.123996”,“type”:“tab”,“label”:“Flow 1”,“disabled”:false,“info”:""},{“id”:“15448192.81d52e”,“type”:“mqtt in”,“z”:“36326994.123996”,“name”:"",“topic”:“hermes/audioServer/default/playBytes/#”,“qos”:“0”,“datatype”:“auto”,“broker”:“75470753.1e7908”,“x”:430,“y”:80,“wires”:[[“57ae0eee.3a66c”,“dc7ae7a2.6762a8”]]},{“id”:“57ae0eee.3a66c”,“type”:“debug”,“z”:“36326994.123996”,“name”:"",“active”:true,“tosidebar”:true,“console”:false,“tostatus”:false,“complete”:“payload”,“targetType”:“msg”,“statusVal”:"",“statusType”:“auto”,“x”:950,“y”:140,“wires”:[]},{“id”:“dc7ae7a2.6762a8”,“type”:“ui_audio”,“z”:“36326994.123996”,“name”:"",“group”:“8e36eaa4.bca608”,“voice”:“fr-FR”,“always”:true,“x”:940,“y”:60,“wires”:[]},{“id”:“ed1f45fe.53c178”,“type”:“ui_microphone”,“z”:“36326994.123996”,“group”:“8e36eaa4.bca608”,“order”:0,“width”:0,“height”:0,“name”:"",“maxLength”:5,“timeslice”:0,“x”:150,“y”:420,“wires”:[[“9d0a9682.880f78”,“407f1d6d.81edc4”]]},{“id”:“42e0ea31.05f844”,“type”:“mqtt out”,“z”:“36326994.123996”,“name”:"",“topic”:“hermes/audioServer/default/audioFrame”,“qos”:“0”,“retain”:"",“broker”:“75470753.1e7908”,“x”:1020,“y”:520,“wires”:[]},{“id”:“1ba9e13d.b899cf”,“type”:“mqtt in”,“z”:“36326994.123996”,“name”:"",“topic”:“hermes/audioServer/default/audioFrame”,“qos”:“0”,“datatype”:“auto”,“broker”:“75470753.1e7908”,“x”:200,“y”:640,“wires”:[[“760e4da8.3a07d4”]]},{“id”:“a621c20e.e1832”,“type”:“mqtt in”,“z”:“36326994.123996”,“name”:"",“topic”:“hermes/asr/startListening”,“qos”:“0”,“datatype”:“auto”,“broker”:“75470753.1e7908”,“x”:150,“y”:540,“wires”:[[“220375ab.8c144a”]]},{“id”:“9d0a9682.880f78”,“type”:“change”,“z”:“36326994.123996”,“name”:"",“rules”:[{“t”:“set”,“p”:“payload”,“pt”:“msg”,“to”:"{“siteId”: “default”}",“tot”:“str”}],“action”:"",“property”:"",“from”:"",“to”:"",“reg”:false,“x”:320,“y”:360,“wires”:[[“d43a748e.26ea38”]]},{“id”:“d43a748e.26ea38”,“type”:“mqtt out”,“z”:“36326994.123996”,“name”:"",“topic”:“hermes/asr/startListening”,“qos”:“0”,“retain”:"",“broker”:“75470753.1e7908”,“x”:650,“y”:360,“wires”:[]},{“id”:“2bf374df.e9f78c”,“type”:“mqtt out”,“z”:“36326994.123996”,“name”:"",“topic”:“hermes/asr/stopListening”,“qos”:“0”,“retain”:"",“broker”:“75470753.1e7908”,“x”:910,“y”:640,“wires”:[]},{“id”:“407f1d6d.81edc4”,“type”:“change”,“z”:“36326994.123996”,“name”:"",“rules”:[{“t”:“set”,“p”:“payload”,“pt”:“flow”,“to”:“payload”,“tot”:“msg”}],“action”:"",“property”:"",“from”:"",“to”:"",“reg”:false,“x”:360,“y”:420,“wires”:[[]]},{“id”:“220375ab.8c144a”,“type”:“switch”,“z”:“36326994.123996”,“name”:"",“property”:“payload”,“propertyType”:“msg”,“rules”:[{“t”:“nnull”}],“checkall”:“true”,“repair”:false,“outputs”:1,“x”:370,“y”:540,“wires”:[[“9f98a014.98b96”]]},{“id”:“9f98a014.98b96”,“type”:“change”,“z”:“36326994.123996”,“name”:"",“rules”:[{“t”:“set”,“p”:“payload”,“pt”:“msg”,“to”:“payload”,“tot”:“flow”}],“action”:"",“property”:"",“from”:"",“to”:"",“reg”:false,“x”:560,“y”:520,“wires”:[[“42e0ea31.05f844”]]},{“id”:“760e4da8.3a07d4”,“type”:“switch”,“z”:“36326994.123996”,“name”:"",“property”:“payload”,“propertyType”:“msg”,“rules”:[{“t”:“nnull”}],“checkall”:“true”,“repair”:false,“outputs”:1,“x”:450,“y”:640,“wires”:[[“12ce7bbc.2aa554”]]},{“id”:“12ce7bbc.2aa554”,“type”:“change”,“z”:“36326994.123996”,“name”:"",“rules”:[{“t”:“set”,“p”:“payload”,“pt”:“msg”,“to”:"{“siteId”: “default”}",“tot”:“str”}],“action”:"",“property”:"",“from”:"",“to”:"",“reg”:false,“x”:620,“y”:640,“wires”:[[“2bf374df.e9f78c”]]},{“id”:“c936b95b.3f80d8”,“type”:“mqtt in”,“z”:“36326994.123996”,“name”:"",“topic”:“hermes/asr/textCaptured”,“qos”:“0”,“datatype”:“auto”,“broker”:“75470753.1e7908”,“x”:170,“y”:760,“wires”:[[“f765ab1c.b366d8”]]},{“id”:“ce918dca.9f245”,“type”:“http request”,“z”:“36326994.123996”,“name”:“http://localhost:12101/api/text-to-speech",“method”:“POST”,“ret”:“txt”,“paytoqs”:“ignore”,“url”:“http://10.27.1.1:12101/api/text-to-speech”,“tls”:"",“persist”:false,“proxy”:"",“authType”:"",“x”:1200,“y”:280,“wires”:[[]]},{“id”:“3cbee97f.4ce566”,“type”:“inject”,“z”:“36326994.123996”,“name”:"Welcome to the world of offline voice assistants.”,“props”:[{“p”:“payload”},{“p”:“topic”,“vt”:“str”}],“repeat”:"",“crontab”:"",“once”:false,“onceDelay”:0.1,“topic”:"",“payload”:“Test de synthèse vocale”,“payloadType”:“str”,“x”:460,“y”:280,“wires”:[[“ce918dca.9f245”]]},{“id”:“f765ab1c.b366d8”,“type”:“json”,“z”:“36326994.123996”,“name”:"",“property”:“payload”,“action”:"",“pretty”:false,“x”:750,“y”:760,“wires”:[[“5fef604d.b1083”]]},{“id”:“5fef604d.b1083”,“type”:“change”,“z”:“36326994.123996”,“name”:"",“rules”:[{“t”:“set”,“p”:“payload”,“pt”:“msg”,“to”:“payload.text”,“tot”:“msg”}],“action”:"",“property”:"",“from”:"",“to”:"",“reg”:false,“x”:960,“y”:760,“wires”:[[“ce918dca.9f245”,“a1ac660f.a785d8”]]},{“id”:“a1ac660f.a785d8”,“type”:“ui_text”,“z”:“36326994.123996”,“group”:“8e36eaa4.bca608”,“order”:2,“width”:0,“height”:0,“name”:"",“label”:“Recu : “,“format”:”{{msg.payload}}”,“layout”:“row-spread”,“x”:1190,“y”:780,“wires”:[]},{“id”:“75470753.1e7908”,“type”:“mqtt-broker”,“z”:"",“name”:"",“broker”:“10.27.1.1”,“port”:“1884”,“clientid”:"",“usetls”:false,“compatmode”:false,“keepalive”:“60”,“cleansession”:true,“birthTopic”:"",“birthQos”:“0”,“birthPayload”:"",“closeTopic”:"",“closeQos”:“0”,“closePayload”:"",“willTopic”:"",“willQos”:“0”,“willPayload”:""},{“id”:“8e36eaa4.bca608”,“type”:“ui_group”,“z”:"",“name”:“Local”,“tab”:“9460789b.1fdef8”,“order”:1,“disp”:true,“width”:“6”,“collapse”:false},{“id”:“9460789b.1fdef8”,“type”:“ui_tab”,“z”:"",“name”:“Local”,“icon”:“dashboard”,“disabled”:false,“hidden”:false}]

[moqart]

Answer to question 1:
From looking at the Reference the only other way is to use the http api under /api/speech-to-intent.

Answer to question 2:
You can do TTS over mqtt with this topic hermes/tts/say. For information on how to use this take a look at this:

Reference - Rhasspy

[Gandalfand]

Thanks a lot Moqart ill try it

[kaykoch]

Hi Gandalfand,
heres a simple example to send text to a satellite:

[{"id":"1e3a1976.3c2897","type":"function","z":"21f11887.571048","name":"","func":"siteId = \"wohnzimmer\"\ntext=\"Hallo. Ich bin ein Computer\"\n\n\nmsg.payload = {\"text\":text,\n               \"lang\":\"de\",\n               \"id\":\"\",\n               \"siteId\":siteId,\n               \"sessionId\":\"\"\n}\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":460,"y":1500,"wires":[["ccf3bb74.c920b8"]]},{"id":"ccf3bb74.c920b8","type":"mqtt out","z":"21f11887.571048","name":"-> Satellite","topic":"hermes/tts/say","qos":"","retain":"","broker":"1d34925b.6e4b5e","x":630,"y":1500,"wires":[]},{"id":"e8ae274b.dc21a8","type":"inject","z":"21f11887.571048","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":310,"y":1500,"wires":[["1e3a1976.3c2897"]]},{"id":"1d34925b.6e4b5e","type":"mqtt-broker","name":"MQTT_MOND","broker":"10.2.254.1","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]

[Alexey]

Hello!
I am a nwebie at Rhasspy. I want to use Rhasspy in my Openhab smart home. I want to send my voice through my browser microphone. Your example is very similar and good for my start. But I cannot import your code in Node Red. The first error is quotation marks. I’ve replaced all quotes with straight ones. Now the error is incomprehensible.
Help me run your code. Can you post code as kaykoch in preformatted text?

[Gandalfand]

Hello Alexey,

this is my code :

I have try the http request but its slower than mqtt with start and stop command

[{"id":"36326994.123996","type":"tab","label":"Flow 1","disabled":false,"info":""},{"id":"15448192.81d52e","type":"mqtt in","z":"36326994.123996","name":"","topic":"hermes/audioServer/default/playBytes/#","qos":"0","datatype":"auto","broker":"75470753.1e7908","x":430,"y":80,"wires":[["57ae0eee.3a66c","dc7ae7a2.6762a8"]]},{"id":"57ae0eee.3a66c","type":"debug","z":"36326994.123996","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":950,"y":140,"wires":[]},{"id":"dc7ae7a2.6762a8","type":"ui_audio","z":"36326994.123996","name":"","group":"8e36eaa4.bca608","voice":"fr-FR","always":true,"x":940,"y":60,"wires":[]},{"id":"ed1f45fe.53c178","type":"ui_microphone","z":"36326994.123996","group":"8e36eaa4.bca608","order":0,"width":0,"height":0,"name":"","maxLength":5,"timeslice":0,"x":110,"y":400,"wires":[["9934e86c.3b7e78","9d0a9682.880f78","407f1d6d.81edc4"]]},{"id":"42e0ea31.05f844","type":"mqtt out","z":"36326994.123996","name":"","topic":"hermes/audioServer/default/audioFrame","qos":"0","retain":"","broker":"75470753.1e7908","x":1020,"y":520,"wires":[]},{"id":"1ba9e13d.b899cf","type":"mqtt in","z":"36326994.123996","name":"","topic":"hermes/audioServer/default/audioFrame","qos":"0","datatype":"auto","broker":"75470753.1e7908","x":200,"y":640,"wires":[["760e4da8.3a07d4"]]},{"id":"a621c20e.e1832","type":"mqtt in","z":"36326994.123996","name":"","topic":"hermes/asr/startListening","qos":"0","datatype":"auto","broker":"75470753.1e7908","x":150,"y":540,"wires":[["220375ab.8c144a"]]},{"id":"9d0a9682.880f78","type":"change","z":"36326994.123996","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"{\"siteId\": \"default\"}","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":320,"y":360,"wires":[["d43a748e.26ea38"]]},{"id":"d43a748e.26ea38","type":"mqtt out","z":"36326994.123996","name":"","topic":"hermes/asr/startListening","qos":"0","retain":"","broker":"75470753.1e7908","x":650,"y":360,"wires":[]},{"id":"2bf374df.e9f78c","type":"mqtt out","z":"36326994.123996","name":"","topic":"hermes/asr/stopListening","qos":"0","retain":"","broker":"75470753.1e7908","x":910,"y":640,"wires":[]},{"id":"407f1d6d.81edc4","type":"change","z":"36326994.123996","name":"","rules":[{"t":"set","p":"payload","pt":"flow","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":360,"y":420,"wires":[[]]},{"id":"220375ab.8c144a","type":"switch","z":"36326994.123996","name":"","property":"payload","propertyType":"msg","rules":[{"t":"nnull"}],"checkall":"true","repair":false,"outputs":1,"x":370,"y":540,"wires":[["9f98a014.98b96"]]},{"id":"9f98a014.98b96","type":"change","z":"36326994.123996","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload","tot":"flow"}],"action":"","property":"","from":"","to":"","reg":false,"x":560,"y":520,"wires":[["42e0ea31.05f844"]]},{"id":"760e4da8.3a07d4","type":"switch","z":"36326994.123996","name":"","property":"payload","propertyType":"msg","rules":[{"t":"nnull"}],"checkall":"true","repair":false,"outputs":1,"x":450,"y":640,"wires":[["12ce7bbc.2aa554"]]},{"id":"12ce7bbc.2aa554","type":"change","z":"36326994.123996","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"{\"siteId\": \"default\"}","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":620,"y":640,"wires":[["2bf374df.e9f78c"]]},{"id":"c936b95b.3f80d8","type":"mqtt in","z":"36326994.123996","name":"","topic":"hermes/asr/textCaptured","qos":"0","datatype":"auto","broker":"75470753.1e7908","x":170,"y":760,"wires":[["f765ab1c.b366d8","fd38de12.910f2"]]},{"id":"f765ab1c.b366d8","type":"json","z":"36326994.123996","name":"","property":"payload","action":"","pretty":false,"x":650,"y":860,"wires":[["5fef604d.b1083"]]},{"id":"5fef604d.b1083","type":"change","z":"36326994.123996","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.text","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":920,"y":880,"wires":[["a1ac660f.a785d8"]]},{"id":"a1ac660f.a785d8","type":"ui_text","z":"36326994.123996","group":"8e36eaa4.bca608","order":2,"width":0,"height":0,"name":"","label":"Recu : ","format":"{{msg.payload}}","layout":"row-spread","x":1190,"y":780,"wires":[]},{"id":"9934e86c.3b7e78","type":"debug","z":"36326994.123996","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload2","targetType":"msg","statusVal":"","statusType":"auto","x":280,"y":480,"wires":[]},{"id":"fd38de12.910f2","type":"mqtt out","z":"36326994.123996","name":"","topic":"hermes/tts/say","qos":"0","retain":"","broker":"75470753.1e7908","x":1220,"y":680,"wires":[]},{"id":"e95d7ffc.ea383","type":"inject","z":"36326994.123996","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"{\"text\": \"Voici un test de synthèse vocale\"}","payloadType":"str","x":150,"y":880,"wires":[["fd38de12.910f2"]]},{"id":"75470753.1e7908","type":"mqtt-broker","z":"","name":"","broker":"10.27.1.1","port":"1884","clientid":"","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""},{"id":"8e36eaa4.bca608","type":"ui_group","z":"","name":"Local","tab":"9460789b.1fdef8","order":1,"disp":true,"width":"6","collapse":false},{"id":"9460789b.1fdef8","type":"ui_tab","z":"","name":"Local","icon":"dashboard","disabled":false,"hidden":false}]