One wake word/phrase followed by multiple commands?

Just wondering if this might be possible with the future 2.5 release?
Is it a concept many of you have considered?

Yes, the Snips dialogue messages are being supported in 2.5, so you can have MQTT services that use continueSession to do multi-turn dialogues with the user (with TTS feedback).

2 Likes

Thanks for that confirmation. One more question, the MQTT is built-in, not a separate server, right?

Yes. Rhasspy 2.5 runs an internal instances of mosquitto when you don’t have it connected to an external MQTT broker.

how do you implement the continue session in 2.5pre ? in homeassistant i made the following intent scripts:

Volume:
  action:
    - service: 'mqtt.publish'
      data_template:
        payload_template: '{"intentFilter": "Again", text: "ok" , sessionId": {{ session_id }}, "customData": {{ volume }} }'
        topic: 'hermes/dialogueManager/continueSession'
        retain: true
    - service: 'freebox_player.remote'
      data_template:
        code: '{{ volume }}'

Again:
  action:
    - service: 'persistent_notification.create'
      data_template:
        message: 'test {{custom_data}}'

the mqtt topic seems ok

hermes/dialogueManager/continueSession

{"intentFilter": "Again", text: "ok" , sessionId": salon-jarvis-a08864d4-b919-4b42-8e0c-3b960c910360, "customData": vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc }

but it seems rhasspy doesn t listen

Your JSON has syntax errors. Check it again.

indeed thank you for your answer indeed i had a few mistakes :stuck_out_tongue:
fixed it but still it’s not working :confused:

here’s the updated code:

Volume:
  action:
    - service: 'mqtt.publish'
      data_template:
        payload_template: '{"intentFilter": "Again", text: "ok" , "sessionId": "{{ session_id }}", "customData": "{{ volume }}" }'
        topic: 'hermes/dialogueManager/continueSession'
        retain: true
    - service: 'freebox_player.remote'
      data_template:
        code: '{{ volume }}'

Again:
  action:
    - service: 'persistent_notification.create'
      data_template:
        message: 'test {{custom_data}}'

hermes/dialogueManager/continueSession

{"intentFilter": "Again", text: "ok" , "sessionId": "salon-jarvis-012e3c9a-0649-4292-a06d-cca3edf9fa1b", "customData": "vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc" }

I’m assuming you set “Rhasspy” as dialogue manager in the settings.

I also recommend you use rhasspy-watch to watch what is happening on your instance (or you could just subscribe to the MQTT topics, but rhasspy-watch is cooler :sunglasses:).

yes on the server and Hermes MQTT on the satelite

ill install rhasspy-watch :slight_smile: thank you for the suggestion didn’t know about it :slight_smile:

[2020-05-11 13:01:42] [hotword] detected on site salon, for model jarvis
[2020-05-11 13:01:42] [hotword] was asked to toggle itself 'off' on site salon
[2020-05-11 13:01:42] [Asr] was asked to toggle itself 'off' on site salon
[2020-05-11 13:01:43] [hotword] was asked to toggle itself 'on' on site salon
[2020-05-11 13:01:43] [Asr] was asked to toggle itself 'on' on site salon
[2020-05-11 13:01:43] [Dialogue] session with id salon-jarvis-707bec44-3da7-4cae-8719-ed4268e589b9 was started on site salon.
[2020-05-11 13:01:43] [hotword] was asked to toggle itself 'off' on site salon
[2020-05-11 13:01:43] [Asr] was asked to listen on site salon
[2020-05-11 13:01:44] [Asr] captured text 'monte le son' in 1.3529847989993868s on site salon
[2020-05-11 13:01:44] [hotword] was asked to toggle itself 'off' on site salon
[2020-05-11 13:01:44] [Asr] was asked to toggle itself 'off' on site salon
[2020-05-11 13:01:45] [hotword] was asked to toggle itself 'on' on site salon
[2020-05-11 13:01:45] [Asr] was asked to toggle itself 'on' on site salon
[2020-05-11 13:01:45] [Asr] was asked to stop listening on site salon
[2020-05-11 13:01:45] [hotword] was asked to toggle itself 'on' on site salon
[2020-05-11 13:01:45] [Nlu] was asked to parse input 'monte le son'
[2020-05-11 13:01:45] [Nlu] Detected intent Volume with confidence score 1.0 for input 'vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc le son
[2020-05-11 13:01:45] [Asr] was asked to stop listening on site salon
[2020-05-11 13:01:45] [Dialogue] session with id salon-jarvis-707bec44-3da7-4cae-8719-ed4268e589b9 was ended on site salon. Reason: nominal
           with customData :
               jarvis
[2020-05-11 13:01:45] [hotword] was asked to toggle itself 'on' on site salon

indeed the session seems to be closed before i send the continuesession or / it s not detected at all

There is something strange in your log.
I dont’ understand why

[2020-05-11 13:01:45] [Nlu] Detected intent Volume with confidence score 1.0 for input 'vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc le son

Normally, input in this line has to be the capture text : monte le son

example :

[2020-05-10 13:31:40] [Nlu] was asked to parse input 'quelle heure est-il'
[2020-05-10 13:31:40] [Nlu] Detected intent AskTime with confidence score 1.0 for input 'quelle heure est-il
[2020-05-10 13:31:40] [Nlu] Intent AskTime with confidence score 1.0 on site bureau

And you have not send a continue session, otherwize, you’ll have :

[2020-05-10 13:31:40] [Nlu] was asked to parse input 'quelle heure est-il'
[2020-05-10 13:31:40] [Nlu] Detected intent AskTime with confidence score 1.0 for input 'quelle heure est-il
[2020-05-10 13:31:40] [Nlu] Intent AskTime with confidence score 1.0 on site bureau
[2020-05-10 13:31:40] [Dialogue] was ask to continue session with id bureau-jarvis-e72da4c0-d1ac-4c5e-bd0f-1c054e499851 by saying 'tu es sûr de vouloir l'heure ?. '
           with customData :
[...blablabla..]

there is a strange transfo between “monte le son” and “vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc le son”

Strange…

You can switch to json format by adding --outputFormat raw to rhasspy watch. You’ll see the messages themselves.

Ced

This line in my log means that something was sent on hermes/intent/AskTime
You don’t have this line in your log.

can you switch in raw mode and check if you have a line like this :

[2020-05-11 13:01:45] hermes/intent/Volume - {'input':  [blabla]...

that s because of my rhasspy sentence :wink:

[Volume]
(plus:vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc | monte:vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc | augmente:vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc | descend:vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec, | baisse:vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec | moins:vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec | coupe:mute | remet:mute ){volume} [le] ( fort | volume | son )

some words are replaced by values :slight_smile:

theres a lot of vol_inc because it increase volume by 1% and that not enough

in raw mode :wink:

[2020-05-11 21:10:29] hermes/hotword/jarvis/detected - {'modelId': 'jarvis', 'modelVersion': '', 'modelType': 'personal', 'currentSensitivity': 0.6, 'siteId': 'salon', 'sessionId': None, 'sendAudioCaptured': None}
[2020-05-11 21:10:29] hermes/hotword/toggleOff - {'siteId': 'salon', 'reason': 'playAudio'}
[2020-05-11 21:10:29] hermes/asr/toggleOff - {'siteId': 'salon', 'reason': 'playAudio'}
[2020-05-11 21:10:30] hermes/hotword/toggleOn - {'siteId': 'salon', 'reason': 'playAudio'}
[2020-05-11 21:10:30] hermes/asr/toggleOn - {'siteId': 'salon', 'reason': 'playAudio'}
[2020-05-11 21:10:30] hermes/dialogueManager/sessionStarted - {'sessionId': 'salon-jarvis-69e3ffc0-2fcf-4069-ae64-1c48f1544354', 'siteId': 'salon', 'customData': 'jarvis'}
[2020-05-11 21:10:30] hermes/hotword/toggleOff - {'siteId': 'salon', 'reason': 'dialogueSession'}
[2020-05-11 21:10:30] hermes/asr/startListening - {'siteId': 'salon', 'sessionId': 'salon-jarvis-69e3ffc0-2fcf-4069-ae64-1c48f1544354', 'stopOnSilence': True, 'sendAudioCaptured': True, 'wakewordId': 'jarvis', 'intentFilter': None}
[2020-05-11 21:10:32] hermes/asr/textCaptured - {'text': 'monte le son', 'likelihood': 1, 'seconds': 1.588414803991327, 'siteId': 'salon', 'sessionId': 'salon-jarvis-69e3ffc0-2fcf-4069-ae64-1c48f1544354', 'wakewordId': None, 'asrTokens': None}
[2020-05-11 21:10:32] hermes/hotword/toggleOff - {'siteId': 'salon', 'reason': 'playAudio'}
[2020-05-11 21:10:32] hermes/asr/toggleOff - {'siteId': 'salon', 'reason': 'playAudio'}
[2020-05-11 21:10:33] hermes/hotword/toggleOn - {'siteId': 'salon', 'reason': 'playAudio'}
[2020-05-11 21:10:33] hermes/asr/toggleOn - {'siteId': 'salon', 'reason': 'playAudio'}
[2020-05-11 21:10:33] hermes/asr/stopListening - {'siteId': 'salon', 'sessionId': 'salon-jarvis-69e3ffc0-2fcf-4069-ae64-1c48f1544354'}
[2020-05-11 21:10:33] hermes/hotword/toggleOn - {'siteId': 'salon', 'reason': 'dialogueSession'}
[2020-05-11 21:10:33] hermes/nlu/query - {'input': 'monte le son', 'siteId': 'salon', 'id': None, 'intentFilter': None, 'sessionId': 'salon-jarvis-69e3ffc0-2fcf-4069-ae64-1c48f1544354', 'wakewordId': 'jarvis'}
[2020-05-11 21:10:33] hermes/nlu/intentParsed - {'input': 'vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc le son', 'intent': {'intentName': 'Volume', 'confidenceScore': 1.0}, 'siteId': 'salon', 'id': None, 'slots': [{'entity': 'volume', 'value': {'kind': 'Unknown', 'value': 'vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc'}, 'slotName': 'volume', 'rawValue': 'monte', 'confidence': 1.0, 'range': {'start': 0, 'end': 79, 'rawStart': 0, 'rawEnd': 5}}], 'sessionId': 'salon-jarvis-69e3ffc0-2fcf-4069-ae64-1c48f1544354'}
[2020-05-11 21:10:33] hermes/intent/Volume - {'input': 'vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc le son', 'intent': {'intentName': 'Volume', 'confidenceScore': 1.0}, 'siteId': 'salon', 'id': None, 'slots': [{'entity': 'volume', 'value': {'kind': 'Unknown', 'value': 'vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc'}, 'slotName': 'volume', 'rawValue': 'monte', 'confidence': 1.0, 'range': {'start': 0, 'end': 79, 'rawStart': 0, 'rawEnd': 5}}], 'sessionId': 'salon-jarvis-69e3ffc0-2fcf-4069-ae64-1c48f1544354', 'customData': None, 'asrTokens': [[{'value': 'vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc', 'confidence': 1.0, 'rangeStart': 0, 'rangeEnd': 79, 'time': None}, {'value': 'le', 'confidence': 1.0, 'rangeStart': 80, 'rangeEnd': 82, 'time': None}, {'value': 'son', 'confidence': 1.0, 'rangeStart': 83, 'rangeEnd': 86, 'time': None}]], 'asrConfidence': None, 'rawInput': 'monte le son', 'wakewordId': 'jarvis'}
[2020-05-11 21:10:33] hermes/dialogueManager/endSession - {'sessionId': 'salon-jarvis-69e3ffc0-2fcf-4069-ae64-1c48f1544354'}
[2020-05-11 21:10:33] hermes/asr/stopListening - {'siteId': 'salon', 'sessionId': 'salon-jarvis-69e3ffc0-2fcf-4069-ae64-1c48f1544354'}
[2020-05-11 21:10:33] hermes/dialogueManager/sessionEnded - {'termination': {'reason': 'nominal'}, 'sessionId': 'salon-jarvis-69e3ffc0-2fcf-4069-ae64-1c48f1544354', 'siteId': 'salon', 'customData': 'jarvis'}
[2020-05-11 21:10:33] hermes/hotword/toggleOn - {'siteId': 'salon', 'reason': 'dialogueSession'}

Ok, so, first thing, it seems there’s a bug in human view of Rhasspy watch. I’ll have to check that. It’s strange that you don’t have this in human view mode :

 [Nlu] Intent Volume with confidence score 1.0 on site bureau with slots...

But the most important, it’s you don’t have continue session in your log.
You have only : hermes/dialogueManager/endSession

The endSession is sent by your handling intent app.

I don’t know home assistant enough to help you.

Ced

well since i never got the Intent handling setting (in rhasspysettings) working i m relying on the snips integration from home assistant to handle everything with hermes

that setting isn t working at all

but when i set to disabled and add “snips:” to my config in home assistant it works

Thanks to @lilbuh . The bug in Rhasspy-watch is solved.

But after analyse of MQTT message, there’s no continueSession sent.

I think it’s a problem between Rhasspy and HomeAssistant. Maybe a bad parameter somewhere ?

Ced

Hi,

Do any of you have any examples of successfully using continueSession with Rhasspy 2.5 to achieve the original poster’s aim or something similar? If so then could you please share an example here?
I have set up multiple intents and thought that I would be able to daisy chain them together however my sessionId is never passed between intents, it is always an empty string.

Thanks

If you’re using rhasspy-hermes-app, you can use this example for inspiration:


Note that there’s still a bug that prevents the custom_data field from being passed around (https://github.com/rhasspy/rhasspy-nlu-hermes/issues/3), but the basic principle of continuing a session and hence having multiple intents being recognized with the same session ID works.

Hi @koan,

Thanks for the heads up about the Hermes app, it looks really promising, nice work!

If I am honest, I am in a bit of a rabbit hole already with getting my head around how MQTT works in Rhasspy, the snips API reference, HA automations and templating. I am just kind of curious if what I am trying to do is even possible in Rhasspy or if I’m going to have to get my head around python and then use Hermes app instead.

Basically I have an intent called [ContinuousMode] which is as follows:

[ContinuousMode]
(enter|exit){action} continuous{mode} mode

The corresponding HA automation is:

- alias: 'Enter Continuous Mode'
    trigger:
      platform: event
      event_type: rhasspy_ContinuousMode
      event_data:
        action: 'enter'
        mode: 'continuous'
    action:
    - service: 'mqtt.publish'
      data_template:
        payload_template: '{"intentFilter": ["ContinuousMode,"ContinuousModeNavigation"], "sessionId": "{{ session_id }}", "siteId":"lounge" }'

If I open up MQTT Explorer and say “Grasshopper, enter continuous mode”, saying my wakeword starts a new session that contains a sessionId as follows:

{"sessionId": "lounge-grasshopper_raspberry-pi-2xxxxaff-xxxx-4f9f-xxxx-a7e5xxxxx186", "siteId": "lounge", "customData": "grasshopper_raspberry-pi", "lang": null}

Saying the intent phrase launches a separate session, this one contains no sessionId:

{"intentFilter": ["ContinuousMode","ContinuousModeNavigation"], "sessionId": "", "siteId":"lounge" }

and then a number of seconds later, the session that was triggered by the wakeword is terminated:

{"termination": {"reason": "timeout"}, "sessionId": "lounge-grasshopper_raspberry-pi-2xxxxaff-xxxx-4f9f-xxxx-a7e5xxxxx186", "siteId": "lounge", "customData": "grasshopper_raspberry-pi"}

So as far as I can tell, a new session is created by the wakeword (which has a sessionId), then another new separate session is started by my intent, but this has no sessionId and hence can’t be linked to the first.
Then the session started by the wakeword ends and the session started by my intent disappears to never be seen again.

I would have expected that saying the wakeword followed by an intent name would have linked those 2 sessions and then I could continue.

Does anyone know where I am going wrong?

Thanks