Is it currently possible to use Rhasspy with dialogs using node-red?

1

Before I’ll search for help in a node-red forum, I’d like to be sure not to ask for things which are currently not possible with Rhasspy:
Is it currently possible to use Rhasspy with dialogs using node-red?

with “dialogs” I mean things like:
Me:Put the light on in the kitchen
Rhasspy: Should I put the light on in the kitchen
Me: Yes
or
Me: No

kind regards,
Hugo

2

It may be simpler to structure your sentences instead, as otherwise you would need to have rhasspy listen/command mode triggered after responding to emulate conversational interaction. Is there a particular use case you have in mind?

3

For some actions I just want to avoid the system misunderstood a request. I wonder if this can be done currently using node-red.

kind regards,
Hugo

4

Imagine you can, however I would be aiming for simpler, clearer sentences with different sounding key words as well as possibly revising hardware etc. Creating complexity with interaction scenarios would probably require an even higher success rate of speech to text processing. I have a simple node-red flow that simply responds with " I don’t understand, try again" when Rhasspy doesn’t understand a voice command.

5

You can use the dialogueManager/continueSession topic for this.
Use a Yes/No intent as intentFilter

https://rhasspy.readthedocs.io/en/latest/reference/#dialogue-manager

6

You can use the dialogueManager/continueSession topic for this.
Use a Yes/No intent as intentFilter

Reference - Rhasspy

Thanks for the answer.
Can this be implemented with node-red? Any idea of someone here did that?

kind regards,
Hugo

7

I also have a node-red flow like you mention above, but for some actions I really want that extra confirmation step, hence my question.

kind regards,
Hugo

8

Should be possible yes, since they are all mqtt messages.
I do not know if someone did that however.

1 Like
9

Yes, node-red is my core automation tool, and I usually start automations with rhasspy Mqtt inputs in parallel with node-red dashboard elements for dual voice and tablet control.

Their is an example of an Mqtt input listening for rhasspy topics. If you still need help for an example I’ll find and post a link.

10

philwent wrote:

Blockquote
Yes, node-red is my core automation tool, and I usually start automations with rhasspy Mqtt inputs in parallel with node-red dashboard elements for dual voice and tablet control.
Their is an example of an Mqtt input listening for rhasspy topics. If you still need help for an example I’ll find and post a link.

Thanks for answering, I’m still interested in an example in node-red which uses a dialog in the following format (the example I am using is not that important as such):

  1. User says: Snowboy (or another wake-up word)
  2. Rhasspy replies with beep
  3. User says: Put on the light in the kitchen
  4. Rhasspy answers: Do you want me to put on the light in the kitchen?
  5. User says: Yes
  6. Rhasspy answers: OK, I’ll put on the light in the kitchen
    or
    5’ User says: No
    6’ Rhasspy answers: OK, I won’t put on the light in the kitchen

I am especially interested in the confirmation steps: 4, 5, 6 or 4, 5’, 6’ because
it is always possible Rhasspy does not understand a request correctly and an action should be
avoided in that case.

kind regards,
Hugo

11

I imagine you could trigger a wake word via MQTT/websockets etc after a delay from successfully recognising an intent, however this would seem to complicate things which I think would make it even less reliable.

If the reasoning is about having accuracy with intent recognition, then maybe just play around with the minimum confidence setting in rhasspy until you are happy with the accuracy.

I think building a reliable interactive dialogue would in fact be more demanding of accuracy than single interactions, and would take sometime to build into a reliable interface. I don’t know of anyone who has examples of interactive dialogues, and it is not a rabbit hole I’d like to follow.

I use node-red and mqtt to tie everything together, so that the logic or automation can be triggered or responded to by voice, web interface or any other api.

Here is a node-red flow that handles failed intent recognition via the hermes/mqtt interface - update with your own rhasspy and mqtt server addresses;

[{"id":"500c313e.3ca7e","type":"http request","z":"6b7fb8b9.b3454","name":"Text-to-speech","method":"POST","ret":"txt","paytoqs":"ignore","url":"http://192.168.1.90:12101/api/text-to-speech?siteid={{{siteId}}}","tls":"","persist":false,"proxy":"","authType":"","x":700,"y":1260,"wires":[[]]},{"id":"77511436.e920a4","type":"mqtt in","z":"6b7fb8b9.b3454","name":"CommandNotRecognised","topic":"hermes/nlu/intentNotRecognized","qos":"2","datatype":"auto","broker":"3ecd7f11.4d3dd","x":130,"y":1220,"wires":[["1af49703.74e9d1","55fc3c09.945914","94295b27.7f997"]]},{"id":"1af49703.74e9d1","type":"debug","z":"6b7fb8b9.b3454","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":410,"y":1200,"wires":[]},{"id":"55fc3c09.945914","type":"exec","z":"6b7fb8b9.b3454","command":"python /home/pi/4mics_hat/pixels_demo.py","addpay":true,"append":"","useSpawn":"false","timer":"","oldrc":false,"name":"What? Voice response","x":460,"y":1320,"wires":[[],[],[]]},{"id":"94295b27.7f997","type":"function","z":"6b7fb8b9.b3454","name":"getsiteidforerrmsg","func":"var newMsg = {};\nnewMsg.payload = \"What did you say?\";\nif(msg.payload.siteId !=null && msg.payload.siteId.length>0) {\n    newMsg.payload.siteId = msg.payload.siteId\n}\nreturn newMsg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":490,"y":1260,"wires":[["500c313e.3ca7e"]]},{"id":"3ecd7f11.4d3dd","type":"mqtt-broker","name":"mqtt_pi_broker","broker":"192.168.1.90","port":"1883","clientid":"Node-Red-Client","usetls":false,"protocolVersion":4,"keepalive":"60","cleansession":false,"birthTopic":"","birthQos":"2","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]

And here is a sample flow that dims lights;

[{"id":"97ea04aa.c9be28","type":"mqtt in","z":"6b7fb8b9.b3454","name":"GetDimCommand","topic":"hermes/intent/SetBrightness","qos":"2","datatype":"json","broker":"3ecd7f11.4d3dd","x":110,"y":240,"wires":[["b9da5149.86661","93451b55.4bc328"]]},{"id":"b9da5149.86661","type":"debug","z":"6b7fb8b9.b3454","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":310,"y":220,"wires":[]},{"id":"93451b55.4bc328","type":"function","z":"6b7fb8b9.b3454","name":"Apply dim setting","func":"var switchName = \"not found\";\nvar switchState = 0;\nvar switchTopic = \"\";\n\nif(msg.payload.slots[0].slotName == \"name\"){\n    switchName = msg.payload.slots[0].rawValue;\n        switchState = msg.payload.slots[1].value.value;\n}else{\n    if(msg.payload.slots[1].slotName == \"name\"){\n        switchName = msg.payload.slots[1].rawValue;\n        switchState = msg.payload.slots[0].value.value;\n    }\n}\n//check which dimmer\nswitch(switchName){\n    case \"kitchen light\":\n        switchTopic = \"shellies/Kitchen/light/0/set\";\n        break;\n    default:\n        switchtopic = \"Not found\";\n        break;\n}\n\nvar setDimmerLevel = {payload:{brightness:switchState},topic:switchTopic};\nreturn setDimmerLevel;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":350,"y":280,"wires":[["46063abe.3e62cc","420d2efc.24d57"]]},{"id":"46063abe.3e62cc","type":"debug","z":"6b7fb8b9.b3454","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":570,"y":260,"wires":[]},{"id":"420d2efc.24d57","type":"mqtt out","z":"6b7fb8b9.b3454","name":"kitchendimmer","topic":"","qos":"2","retain":"","broker":"3ecd7f11.4d3dd","x":580,"y":300,"wires":[]},{"id":"3ecd7f11.4d3dd","type":"mqtt-broker","name":"mqtt_pi_broker","broker":"192.168.1.90","port":"1883","clientid":"Node-Red-Client","usetls":false,"protocolVersion":4,"keepalive":"60","cleansession":false,"birthTopic":"","birthQos":"2","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]```

This is a generic flow that uses an mqtt intent to then generate an mqtt message to switch on or off a selected light depending on the components in the sentence;

[{"id":"5ede97c0.a01e68","type":"exec","z":"6b7fb8b9.b3454","command":"python /home/pi/4mics_hat/pixels_demo.py","addpay":false,"append":"","useSpawn":"false","timer":"","oldrc":false,"name":"Light Switch Voice Response","x":340,"y":160,"wires":[[],[],[]]},{"id":"e3d765d8.5e38f","type":"mqtt in","z":"6b7fb8b9.b3454","name":"GetLightCommand","topic":"hermes/intent/ChangeSwitchState","qos":"2","datatype":"json","broker":"3ecd7f11.4d3dd","x":130,"y":80,"wires":[["b51b6321.be4848","37ece38c.30eaa4","5ede97c0.a01e68"]]},{"id":"b51b6321.be4848","type":"debug","z":"6b7fb8b9.b3454","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":330,"y":60,"wires":[]},{"id":"37ece38c.30eaa4","type":"function","z":"6b7fb8b9.b3454","name":"checkswitchname","func":"var switchName = \"not found\";\nvar switchState = \"not found\";\nvar siteId;\n\nif(msg.payload.slots[0].slotName == \"name\"){\n    switchName = msg.payload.slots[0].rawValue;\n    switchState = msg.payload.slots[1].rawValue;\n}else{\n    if(msg.payload.slots[1].slotName == \"name\"){\n        switchName = msg.payload.slots[1].rawValue;\n        switchState = msg.payload.slots[0].rawValue;\n    }\n}\n\nvar msgReturn = {\n    payload:{switch: switchName, state: switchState}\n}\n\nif(msg.payload.siteId !=null && msg.payload.siteId.length>0) {\n    msgReturn.payload.siteId = msg.payload.siteId\n}\n\nreturn msgReturn;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":370,"y":100,"wires":[["3819a049.d0f27","e1ca0e3a.2048e","cb3a4f5f.1f02c8"]]},{"id":"3819a049.d0f27","type":"function","z":"6b7fb8b9.b3454","name":"set topic and payload","func":"//Setup output vars\nvar switchPayLoad; //on or off\nvar switchTopic; //mqtt topic\nvar newMsg = {};\n//which switch are we??\nswitch(msg.payload.state) {\n    case \"on\":\n        switchPayLoad =\"on\";\n        break;\n    case \"off\":\n        switchPayLoad=\"off\";\n        break;\n    default:\n    switchPayLoad=\"off\";\n}\n\nif(msg.payload.switch == \"all lights\"){\n    \n        newMsg.payload = switchPayLoad;\n        newMsg.switchName = msg.payload.switch;\n        \n        newMsg.topic = \"shellies/Hallway/relay/0/command\";\n        node.send(newMsg);\n        newMsg.topic = \"shellies/FrontPorchLight/relay/0/command\";\n        node.send(newMsg);\n        newMsg.topic = \"shellies/BackPorchLight/relay/0/command\";\n        node.send(newMsg);\n        newMsg.topic = \"shellies/Kitchen/light/0/command\";\n        node.send(newMsg);\n        newMsg.topic = \"shellies/Laundry/relay/1/command\";\n        node.send(newMsg);\n        newMsg.topic = \"shellies/BedroomLight/relay/0/command\";\n        node.send(newMsg);\n        newMsg.topic = \"shellies/Laundry/relay/0/command\";\n        node.send(newMsg);\n        newMsg.topic = \"shellies/Bathroom/relay/1/command\";\n        node.send(newMsg);\n        newMsg.topic = \"shellies/Bathroom/relay/0/command\";\n        node.send(newMsg);\n        newMsg.topic = \"shellies/Toilet/relay/0/command\";\n        node.send(newMsg);\n        newMsg.topic = \"shellies/Office/relay/0/command\";\n        node.send(newMsg);\n        newMsg.topic = \"shellies/Office/relay/1/command\";\n        node.send(newMsg);\n        newMsg.topic = \"shellies/DiningRoom/relay/0/command\";\n        node.send(newMsg);\n        newMsg.topic = \"shellies/BalconyLight/relay/0/command\";\n        \n    return newMsg;\n}else{\n    switch(msg.payload.switch) {\n      case \"hallway light\":\n        switchTopic = \"shellies/Hallway/relay/0/command\";\n        break;\n      case \"front porch light\":\n        switchTopic = \"shellies/FrontPorchLight/relay/0/command\";\n        break;\n      case \"back porch light\":\n        switchTopic = \"shellies/BackPorchLight/relay/0/command\";\n        break;\n      case \"driveway light\":\n        switchTopic = \"shellies/Laundry/relay/1/command\";\n        break;\n      case \"kitchen light\":\n        switchTopic = \"shellies/Kitchen/light/0/command\";\n        break;\n      case \"bedroom light\":\n        switchTopic = \"shellies/BedroomLight/relay/0/command\";\n        break;\n      case \"laundry light\":\n        switchTopic = \"shellies/Laundry/relay/0/command\";\n        break;\n      case \"bathroom light\":\n        switchTopic = \"shellies/Bathroom/relay/1/command\";\n        break;\n      case \"bathroom fan\":\n        switchTopic = \"shellies/Bathroom/relay/0/command\";\n        break;\n      case \"toilet\":\n        switchTopic = \"shellies/Toilet/relay/0/command\";\n        break;\n      case \"office light\":\n        switchTopic = \"shellies/Office/relay/0/command\";\n        break;\n      case \"office wall light\":\n        switchTopic = \"shellies/Office/relay/1/command\";\n        break;\n      case \"dining room light\":\n        switchTopic = \"shellies/DiningRoom/relay/0/command\";\n        break;\n      case \"balcony light\":\n        switchTopic = \"shellies/BalconyLight/relay/0/command\";\n        break;\n    } \n\nnewMsg = {\n    payload: switchPayLoad, topic: switchTopic, switchName: msg.payload.switch\n}\nreturn newMsg;\n}","outputs":1,"noerr":0,"initialize":"","finalize":"","x":620,"y":120,"wires":[["93f047a1.4fa34","c73debe.6bacf98"]]},{"id":"93f047a1.4fa34","type":"debug","z":"6b7fb8b9.b3454","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":750,"y":60,"wires":[]},{"id":"e1ca0e3a.2048e","type":"debug","z":"6b7fb8b9.b3454","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":570,"y":60,"wires":[]},{"id":"c73debe.6bacf98","type":"mqtt out","z":"6b7fb8b9.b3454","name":"switch selected light","topic":"","qos":"2","retain":"","broker":"3ecd7f11.4d3dd","x":860,"y":120,"wires":[]},{"id":"cb3a4f5f.1f02c8","type":"function","z":"6b7fb8b9.b3454","name":"ConfirmationMsg","func":"//var confirmationText =\"Turning \" + msg.payload + \" \" + msg.switchName;\nvar confirmationText =\"Turning \" + msg.payload.switch + \" \" + msg.payload.state;\nvar newMsg = \n{\n    payload: confirmationText\n}\nnewMsg.siteId = msg.payload.siteId\nreturn newMsg;","outputs":1,"noerr":0,"initialize":"","finalize":"","x":610,"y":180,"wires":[["7b5f7ddb.52a854"]]},{"id":"7b5f7ddb.52a854","type":"http request","z":"6b7fb8b9.b3454","name":"Text to speech","method":"POST","ret":"txt","paytoqs":"ignore","url":"http://192.168.1.90:12101/api/text-to-speech?siteId={{{siteId}}}","tls":"","persist":false,"proxy":"","authType":"","x":880,"y":180,"wires":[[]]},{"id":"3ecd7f11.4d3dd","type":"mqtt-broker","name":"mqtt_pi_broker","broker":"192.168.1.90","port":"1883","clientid":"Node-Red-Client","usetls":false,"protocolVersion":4,"keepalive":"60","cleansession":false,"birthTopic":"","birthQos":"2","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]```
1 Like
12

Thanks for answering. Could you write down a sample dialog which your system performs when intent recognition fails?

thanks,
Hugo

13

I don’t have a sample dialogue, but have posted earlier in this thread a sample flow that responds when rhasspy doesn’t understand.

14

I don’t have a sample dialogue, but have posted earlier in this thread a sample flow that responds when rhasspy doesn’t understand.

Thanks, It was not totally clear to me what your nodered flow did, but now I’m convinced it not what I was looking for, I’m looking for a node-red flow which allows an extra confirmation dialog. e.g. When I have a question like: Snowboy, shut down Rhasspy 1, I want it to ask me first:
“do you really want to shut down Rhasspy 1?”
and according to my answer to that question (yes or no), to proceed with the shutdown or not

kind regards,
Hugo

15

Missed it earlier, but @romkabouter pointed to the document on the built-in dialogue function earlier in this thread where you can use an intent filter for confirmation; Reference - Rhasspy

You would need to play around with it and Mqtt to integrate it into node-red flows, not something I have done.

1 Like