One wake word/phrase followed by multiple commands?

FredTheFrog · January 30, 2020, 7:30pm

Just wondering if this might be possible with the future 2.5 release?
Is it a concept many of you have considered?

synesthesiam · January 31, 2020, 3:21am

Yes, the Snips dialogue messages are being supported in 2.5, so you can have MQTT services that use continueSession to do multi-turn dialogues with the user (with TTS feedback).

FredTheFrog · January 31, 2020, 2:13pm

Thanks for that confirmation. One more question, the MQTT is built-in, not a separate server, right?

synesthesiam · February 3, 2020, 4:01am

Yes. Rhasspy 2.5 runs an internal instances of mosquitto when you don’t have it connected to an external MQTT broker.

lilbuh · May 10, 2020, 8:03pm

how do you implement the continue session in 2.5pre ? in homeassistant i made the following intent scripts:

Volume:
  action:
    - service: 'mqtt.publish'
      data_template:
        payload_template: '{"intentFilter": "Again", text: "ok" , sessionId": {{ session_id }}, "customData": {{ volume }} }'
        topic: 'hermes/dialogueManager/continueSession'
        retain: true
    - service: 'freebox_player.remote'
      data_template:
        code: '{{ volume }}'

Again:
  action:
    - service: 'persistent_notification.create'
      data_template:
        message: 'test {{custom_data}}'

the mqtt topic seems ok

hermes/dialogueManager/continueSession

{"intentFilter": "Again", text: "ok" , sessionId": salon-jarvis-a08864d4-b919-4b42-8e0c-3b960c910360, "customData": vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc }

but it seems rhasspy doesn t listen

daniele_athome · May 10, 2020, 10:59pm

Your JSON has syntax errors. Check it again.

lilbuh · May 10, 2020, 11:33pm

indeed thank you for your answer indeed i had a few mistakes
fixed it but still it’s not working

here’s the updated code:

Volume:
  action:
    - service: 'mqtt.publish'
      data_template:
        payload_template: '{"intentFilter": "Again", text: "ok" , "sessionId": "{{ session_id }}", "customData": "{{ volume }}" }'
        topic: 'hermes/dialogueManager/continueSession'
        retain: true
    - service: 'freebox_player.remote'
      data_template:
        code: '{{ volume }}'

Again:
  action:
    - service: 'persistent_notification.create'
      data_template:
        message: 'test {{custom_data}}'

hermes/dialogueManager/continueSession

{"intentFilter": "Again", text: "ok" , "sessionId": "salon-jarvis-012e3c9a-0649-4292-a06d-cca3edf9fa1b", "customData": "vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc" }

daniele_athome · May 11, 2020, 7:00am

I’m assuming you set “Rhasspy” as dialogue manager in the settings.

I also recommend you use rhasspy-watch to watch what is happening on your instance (or you could just subscribe to the MQTT topics, but rhasspy-watch is cooler ).

lilbuh · May 11, 2020, 11:52am

yes on the server and Hermes MQTT on the satelite

ill install rhasspy-watch thank you for the suggestion didn’t know about it

lilbuh · May 11, 2020, 1:03pm

[2020-05-11 13:01:42] [hotword] detected on site salon, for model jarvis
[2020-05-11 13:01:42] [hotword] was asked to toggle itself 'off' on site salon
[2020-05-11 13:01:42] [Asr] was asked to toggle itself 'off' on site salon
[2020-05-11 13:01:43] [hotword] was asked to toggle itself 'on' on site salon
[2020-05-11 13:01:43] [Asr] was asked to toggle itself 'on' on site salon
[2020-05-11 13:01:43] [Dialogue] session with id salon-jarvis-707bec44-3da7-4cae-8719-ed4268e589b9 was started on site salon.
[2020-05-11 13:01:43] [hotword] was asked to toggle itself 'off' on site salon
[2020-05-11 13:01:43] [Asr] was asked to listen on site salon
[2020-05-11 13:01:44] [Asr] captured text 'monte le son' in 1.3529847989993868s on site salon
[2020-05-11 13:01:44] [hotword] was asked to toggle itself 'off' on site salon
[2020-05-11 13:01:44] [Asr] was asked to toggle itself 'off' on site salon
[2020-05-11 13:01:45] [hotword] was asked to toggle itself 'on' on site salon
[2020-05-11 13:01:45] [Asr] was asked to toggle itself 'on' on site salon
[2020-05-11 13:01:45] [Asr] was asked to stop listening on site salon
[2020-05-11 13:01:45] [hotword] was asked to toggle itself 'on' on site salon
[2020-05-11 13:01:45] [Nlu] was asked to parse input 'monte le son'
[2020-05-11 13:01:45] [Nlu] Detected intent Volume with confidence score 1.0 for input 'vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc le son
[2020-05-11 13:01:45] [Asr] was asked to stop listening on site salon
[2020-05-11 13:01:45] [Dialogue] session with id salon-jarvis-707bec44-3da7-4cae-8719-ed4268e589b9 was ended on site salon. Reason: nominal
           with customData :
               jarvis
[2020-05-11 13:01:45] [hotword] was asked to toggle itself 'on' on site salon

indeed the session seems to be closed before i send the continuesession or / it s not detected at all

ced_cox · May 11, 2020, 8:15pm

There is something strange in your log.
I dont’ understand why

[2020-05-11 13:01:45] [Nlu] Detected intent Volume with confidence score 1.0 for input 'vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc le son

Normally, input in this line has to be the capture text : monte le son

example :

[2020-05-10 13:31:40] [Nlu] was asked to parse input 'quelle heure est-il'
[2020-05-10 13:31:40] [Nlu] Detected intent AskTime with confidence score 1.0 for input 'quelle heure est-il
[2020-05-10 13:31:40] [Nlu] Intent AskTime with confidence score 1.0 on site bureau

And you have not send a continue session, otherwize, you’ll have :

[2020-05-10 13:31:40] [Nlu] was asked to parse input 'quelle heure est-il'
[2020-05-10 13:31:40] [Nlu] Detected intent AskTime with confidence score 1.0 for input 'quelle heure est-il
[2020-05-10 13:31:40] [Nlu] Intent AskTime with confidence score 1.0 on site bureau
[2020-05-10 13:31:40] [Dialogue] was ask to continue session with id bureau-jarvis-e72da4c0-d1ac-4c5e-bd0f-1c054e499851 by saying 'tu es sûr de vouloir l'heure ?. '
           with customData :
[...blablabla..]

there is a strange transfo between “monte le son” and “vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc le son”

Strange…

You can switch to json format by adding --outputFormat raw to rhasspy watch. You’ll see the messages themselves.

Ced

ced_cox · May 11, 2020, 8:30pm

This line in my log means that something was sent on hermes/intent/AskTime
You don’t have this line in your log.

can you switch in raw mode and check if you have a line like this :

[2020-05-11 13:01:45] hermes/intent/Volume - {'input':  [blabla]...

lilbuh · May 11, 2020, 9:06pm

that s because of my rhasspy sentence

[Volume]
(plus:vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc | monte:vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc | augmente:vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc | descend:vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec, | baisse:vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec | moins:vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec,vol_dec | coupe:mute | remet:mute ){volume} [le] ( fort | volume | son )

some words are replaced by values

theres a lot of vol_inc because it increase volume by 1% and that not enough

lilbuh · May 11, 2020, 9:09pm

in raw mode

[2020-05-11 21:10:29] hermes/hotword/jarvis/detected - {'modelId': 'jarvis', 'modelVersion': '', 'modelType': 'personal', 'currentSensitivity': 0.6, 'siteId': 'salon', 'sessionId': None, 'sendAudioCaptured': None}
[2020-05-11 21:10:29] hermes/hotword/toggleOff - {'siteId': 'salon', 'reason': 'playAudio'}
[2020-05-11 21:10:29] hermes/asr/toggleOff - {'siteId': 'salon', 'reason': 'playAudio'}
[2020-05-11 21:10:30] hermes/hotword/toggleOn - {'siteId': 'salon', 'reason': 'playAudio'}
[2020-05-11 21:10:30] hermes/asr/toggleOn - {'siteId': 'salon', 'reason': 'playAudio'}
[2020-05-11 21:10:30] hermes/dialogueManager/sessionStarted - {'sessionId': 'salon-jarvis-69e3ffc0-2fcf-4069-ae64-1c48f1544354', 'siteId': 'salon', 'customData': 'jarvis'}
[2020-05-11 21:10:30] hermes/hotword/toggleOff - {'siteId': 'salon', 'reason': 'dialogueSession'}
[2020-05-11 21:10:30] hermes/asr/startListening - {'siteId': 'salon', 'sessionId': 'salon-jarvis-69e3ffc0-2fcf-4069-ae64-1c48f1544354', 'stopOnSilence': True, 'sendAudioCaptured': True, 'wakewordId': 'jarvis', 'intentFilter': None}
[2020-05-11 21:10:32] hermes/asr/textCaptured - {'text': 'monte le son', 'likelihood': 1, 'seconds': 1.588414803991327, 'siteId': 'salon', 'sessionId': 'salon-jarvis-69e3ffc0-2fcf-4069-ae64-1c48f1544354', 'wakewordId': None, 'asrTokens': None}
[2020-05-11 21:10:32] hermes/hotword/toggleOff - {'siteId': 'salon', 'reason': 'playAudio'}
[2020-05-11 21:10:32] hermes/asr/toggleOff - {'siteId': 'salon', 'reason': 'playAudio'}
[2020-05-11 21:10:33] hermes/hotword/toggleOn - {'siteId': 'salon', 'reason': 'playAudio'}
[2020-05-11 21:10:33] hermes/asr/toggleOn - {'siteId': 'salon', 'reason': 'playAudio'}
[2020-05-11 21:10:33] hermes/asr/stopListening - {'siteId': 'salon', 'sessionId': 'salon-jarvis-69e3ffc0-2fcf-4069-ae64-1c48f1544354'}
[2020-05-11 21:10:33] hermes/hotword/toggleOn - {'siteId': 'salon', 'reason': 'dialogueSession'}
[2020-05-11 21:10:33] hermes/nlu/query - {'input': 'monte le son', 'siteId': 'salon', 'id': None, 'intentFilter': None, 'sessionId': 'salon-jarvis-69e3ffc0-2fcf-4069-ae64-1c48f1544354', 'wakewordId': 'jarvis'}
[2020-05-11 21:10:33] hermes/nlu/intentParsed - {'input': 'vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc le son', 'intent': {'intentName': 'Volume', 'confidenceScore': 1.0}, 'siteId': 'salon', 'id': None, 'slots': [{'entity': 'volume', 'value': {'kind': 'Unknown', 'value': 'vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc'}, 'slotName': 'volume', 'rawValue': 'monte', 'confidence': 1.0, 'range': {'start': 0, 'end': 79, 'rawStart': 0, 'rawEnd': 5}}], 'sessionId': 'salon-jarvis-69e3ffc0-2fcf-4069-ae64-1c48f1544354'}
[2020-05-11 21:10:33] hermes/intent/Volume - {'input': 'vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc le son', 'intent': {'intentName': 'Volume', 'confidenceScore': 1.0}, 'siteId': 'salon', 'id': None, 'slots': [{'entity': 'volume', 'value': {'kind': 'Unknown', 'value': 'vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc'}, 'slotName': 'volume', 'rawValue': 'monte', 'confidence': 1.0, 'range': {'start': 0, 'end': 79, 'rawStart': 0, 'rawEnd': 5}}], 'sessionId': 'salon-jarvis-69e3ffc0-2fcf-4069-ae64-1c48f1544354', 'customData': None, 'asrTokens': [[{'value': 'vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc,vol_inc', 'confidence': 1.0, 'rangeStart': 0, 'rangeEnd': 79, 'time': None}, {'value': 'le', 'confidence': 1.0, 'rangeStart': 80, 'rangeEnd': 82, 'time': None}, {'value': 'son', 'confidence': 1.0, 'rangeStart': 83, 'rangeEnd': 86, 'time': None}]], 'asrConfidence': None, 'rawInput': 'monte le son', 'wakewordId': 'jarvis'}
[2020-05-11 21:10:33] hermes/dialogueManager/endSession - {'sessionId': 'salon-jarvis-69e3ffc0-2fcf-4069-ae64-1c48f1544354'}
[2020-05-11 21:10:33] hermes/asr/stopListening - {'siteId': 'salon', 'sessionId': 'salon-jarvis-69e3ffc0-2fcf-4069-ae64-1c48f1544354'}
[2020-05-11 21:10:33] hermes/dialogueManager/sessionEnded - {'termination': {'reason': 'nominal'}, 'sessionId': 'salon-jarvis-69e3ffc0-2fcf-4069-ae64-1c48f1544354', 'siteId': 'salon', 'customData': 'jarvis'}
[2020-05-11 21:10:33] hermes/hotword/toggleOn - {'siteId': 'salon', 'reason': 'dialogueSession'}

ced_cox · May 11, 2020, 9:21pm

Ok, so, first thing, it seems there’s a bug in human view of Rhasspy watch. I’ll have to check that. It’s strange that you don’t have this in human view mode :

 [Nlu] Intent Volume with confidence score 1.0 on site bureau with slots...

But the most important, it’s you don’t have continue session in your log.
You have only : hermes/dialogueManager/endSession

The endSession is sent by your handling intent app.

I don’t know home assistant enough to help you.

Ced

lilbuh · May 11, 2020, 9:27pm

well since i never got the Intent handling setting (in rhasspysettings) working i m relying on the snips integration from home assistant to handle everything with hermes

that setting isn t working at all

but when i set to disabled and add “snips:” to my config in home assistant it works

ced_cox · May 11, 2020, 10:19pm

Thanks to @lilbuh . The bug in Rhasspy-watch is solved.

But after analyse of MQTT message, there’s no continueSession sent.

I think it’s a problem between Rhasspy and HomeAssistant. Maybe a bad parameter somewhere ?

Ced

maleko · January 31, 2021, 12:59pm

Hi,

Do any of you have any examples of successfully using continueSession with Rhasspy 2.5 to achieve the original poster’s aim or something similar? If so then could you please share an example here?
I have set up multiple intents and thought that I would be able to daisy chain them together however my sessionId is never passed between intents, it is always an empty string.

Thanks

koan · January 31, 2021, 4:38pm

If you’re using rhasspy-hermes-app, you can use this example for inspiration:

github.com

rhasspy/rhasspy-hermes-app/blob/master/examples/continue_session.py

"""Example app that shows how to continue a session."""
import logging

from rhasspyhermes.nlu import NluIntent

from rhasspyhermes_app import ContinueSession, EndSession, HermesApp

_LOGGER = logging.getLogger("ContinueApp")

app = HermesApp("ContinueApp")


@app.on_intent("Yes")
async def yes(intent: NluIntent):
    """The user confirms."""
    if intent.custom_data == "TurnOffLight":
        response = "OK, turning off the light"
    elif intent.custom_data == "TurnOnLight":
        response = "OK, turning on the light"
    else:

This file has been truncated. show original

Note that there’s still a bug that prevents the custom_data field from being passed around (https://github.com/rhasspy/rhasspy-nlu-hermes/issues/3), but the basic principle of continuing a session and hence having multiple intents being recognized with the same session ID works.

maleko · January 31, 2021, 10:48pm

Hi @koan,

Thanks for the heads up about the Hermes app, it looks really promising, nice work!

If I am honest, I am in a bit of a rabbit hole already with getting my head around how MQTT works in Rhasspy, the snips API reference, HA automations and templating. I am just kind of curious if what I am trying to do is even possible in Rhasspy or if I’m going to have to get my head around python and then use Hermes app instead.

Basically I have an intent called [ContinuousMode] which is as follows:

[ContinuousMode]
(enter|exit){action} continuous{mode} mode

The corresponding HA automation is:

- alias: 'Enter Continuous Mode'
    trigger:
      platform: event
      event_type: rhasspy_ContinuousMode
      event_data:
        action: 'enter'
        mode: 'continuous'
    action:
    - service: 'mqtt.publish'
      data_template:
        payload_template: '{"intentFilter": ["ContinuousMode,"ContinuousModeNavigation"], "sessionId": "{{ session_id }}", "siteId":"lounge" }'

If I open up MQTT Explorer and say “Grasshopper, enter continuous mode”, saying my wakeword starts a new session that contains a sessionId as follows:

{"sessionId": "lounge-grasshopper_raspberry-pi-2xxxxaff-xxxx-4f9f-xxxx-a7e5xxxxx186", "siteId": "lounge", "customData": "grasshopper_raspberry-pi", "lang": null}

Saying the intent phrase launches a separate session, this one contains no sessionId:

{"intentFilter": ["ContinuousMode","ContinuousModeNavigation"], "sessionId": "", "siteId":"lounge" }

and then a number of seconds later, the session that was triggered by the wakeword is terminated:

{"termination": {"reason": "timeout"}, "sessionId": "lounge-grasshopper_raspberry-pi-2xxxxaff-xxxx-4f9f-xxxx-a7e5xxxxx186", "siteId": "lounge", "customData": "grasshopper_raspberry-pi"}

So as far as I can tell, a new session is created by the wakeword (which has a sessionId), then another new separate session is started by my intent, but this has no sessionId and hence can’t be linked to the first.
Then the session started by the wakeword ends and the session started by my intent disappears to never be seen again.

I would have expected that saying the wakeword followed by an intent name would have linked those 2 sessions and then I could continue.

Does anyone know where I am going wrong?

Thanks