How to Enable Continuous Mode

TLDR: How do I implement continuous mode in Rhasspy ie speak multiple commands one after the other without having to repeat the wakeword after each one?



I am trying to implement continuous mode ie:

I say “Porcupine, enter continuous mode” and then I can say as many commands as I require, without having to say the wakeword before each one and then once I am done I say “exit continuous mode”.

My use case is for example navigating Netflix. Once in the Netflix app on my TV, I want to be able to do something like say “right, right, down, right, down, left, select, play” and my selection will start playing (I can control my TV via a Broadlink RM 2 Pro).

I have the MQTT broker that I use for Home Assistant set as an external MQTT broker in Rhasspy. I have successfully created some other automations using hermes/dialogueManager/startSession and hermes/dialogueManager/endSession (to mute my speakers when saying a rhasspy command and to cancel a command respectively.)

From reading other posts on here I think I understand what I need to do and I have come up with the following.

Rhasspy Intents (note that I use Events to interact with Home Assistant):

[EnterContinuousMode]
enter{action} continuous{mode} mode
[ExitContinuousMode]
exit{action} continuous{mode} mode
[ContinuousModeNavigation]
actions = (right | left | up | down | select | play) {action}
<actions>

Home Assistant Automations:

  - alias: 'Enter Continuous Mode'
    trigger:
      platform: event
      event_type: rhasspy_EnterContinuousMode
      event_data:
        action: 'enter'
        mode: 'continuous'
    action:
    - service: 'mqtt.publish'
      data_template:
        payload_template: '{"intentFilter": ["enter continuous mode","exit continuous mode", "right", "left", "up", "down", "select", "play"], "sessionId": "{{ session_id }}", "siteId":"lounge" }'
        topic: 'hermes/dialogueManager/continueSession' 
  - alias: 'Exit Continuous Mode'
    trigger:
      platform: event
      event_type: rhasspy_ExitContinuousMode
      event_data:
        action: 'exit'
        mode: 'continuous'
    action:
    - service: 'mqtt.publish'
      data_template:
        payload_template: '{ "intentFilter": "exit continuous mode", "sessionId": "{{ session_id }}", "siteId":"lounge" }'
        topic: 'hermes/dialogueManager/endSession' 
  - alias: 'Continuous Mode Right'
    trigger:
      platform: event
      event_type: rhasspy_ContinuousModeNavigation
      event_data:
        action: 'right'
    action:
    - service: switch.toggle
      entity_id: switch.tv_right
    - service: 'mqtt.publish'
      data_template:
        payload_template: '{ "intentFilter": "right", "sessionId": "{{ session_id }}", "siteId":"lounge" }'
        topic: 'hermes/dialogueManager/continueSession'
  - alias: 'Continuous Mode Select'
    trigger:
      platform: event
      event_type: rhasspy_ContinuousModeNavigation
      event_data:
        action: 'select'
    action:
    - service: switch.toggle
      entity_id: switch.tv_ok
    - service: 'mqtt.publish'
      data_template:
        payload_template: '{ "intentFilter": "select", "sessionId": "{{ session_id }}", "siteId":"lounge" }'
        topic: 'hermes/dialogueManager/continueSession'

So if I understood the other posts correctly, I can limit the intents that work with the Enter Continuous Mode automation using intentFilter and as long as I end each of my navigation automations (eg Continuous Mode Select) with another continueSession command then I should be able to daisy chain any commands that I require together?

So with the above intents and automations in place, if I say “Porcupine, enter continuous mode”, Rhasspy is able to successfully recognise that specific intent and Home Assistant says that the Enter Continuous Mode automation has been successfully triggered in the logbook, however if I then say “right” or “select”, then the second command is not registered.

I opened up MQTT Explorer and when I do the above this is what I see under the dialogueManager:

sessionStarted:

{"sessionId": "lounge-grasshopper_raspberry-pi-7822xxxx-xxxx-41da-a670-xxxxxxx24d74", "siteId": "lounge", "customData": "grasshopper_raspberry-pi", "lang": null}

continueSession:

{"intentFilter": ["enter continuous mode","exit continuous mode", "right", "left", "up", "down", "play", "select"], "sessionId": "", "siteId":"lounge" }

and then after a few seconds I get this:
sessionEnded:

{"termination": {"reason": "timeout"}, "sessionId": "lounge-grasshopper_raspberry-pi-7822xxxx-xxxx-41da-a670-xxxxxxx24d74", "siteId": "lounge", "customData": "grasshopper_raspberry-pi"}

I think what the above is telling me is that I am not successfully passing the sessionId between my commands so basically when I say “Porcupine, enter continuous mode”, Rhasspy is looking for an intent with a matching sessionId and then times out when it does not receive one. I am very new to MQTT and automations in general so I may have misunderstood.

Can anyone please advise me how to get this working? How do I pass the sessionId between commands?

Is what I am doing logical or is there a better way of doing this?

Thanks

Can you try _sessionId instead of sessionId?

Hi @synesthesiam,

Thanks for responding.

I tried what you suggested but unfortunately it didn’t seem to make any difference. Is there anything else that I could try?

After reading more I realised that my above automations are wrong for the intentFilter part, I had incorrectly been putting the text to be spoken by the user to trigger the intent, not the intent name itself.

What I had was:

{"intentFilter": ["enter continuous mode","exit continuous mode", "right", "left", "up", "down", "ok", "select"] }

When actually it should be:

{"intentFilter": ["EnterContinuousMode","ExitContinuousMode", "ContinuousModeNavigation"]}

Unfortunately implementing this didn’t solve my issue.

From examining MQTT Explorer it appears that for all of my automations that use MQTT as a trigger or publish to MQTT as a service, I am never successfully passing the sessionId between commands, it always appears as an empty string in any subsequent message.

Does anyone have any examples of how they successfully pass a sessionId between intents/messages?

Thanks

Sorry, I forgot there is no _sessionId property in the Home Assistant event. There is, however, an _intent object that should have all the relevant information. I can’t remember off the top of my head if it’s _intent.sessionId or _intent.session_id, but I’m pretty sure it’s the first one.

Somthing to keep in mind for intent_filter: it will limit the possible intents that come out, but it doesn’t prevent others from being recognized yet. Ideally, Rhasspy would only be able to hear and recognize the intents that were part of the filter.

Hi @synesthesiam,

Thanks. It was a combination of that and not using the correct syntax for my HA template, it is now working great! Thank you for your help and for making Rhasspy, it really is an awesome addition to my setup!

Re your comment on the intents recognised with by the intentFilter, I have noticed this to be the case for some intents but not all.
Weirdly intents relating to my lights (controlled by a Broadlink rm pro or an rf bridge flashed with tasmota) all work both in continuous mode and as single commands (my continuous mode intent is identical to the single commands one but I just added “CM” to the end of the intent name.)
Whereas any intents to do with navigation (both on my tv using the rm pro to emulate button presses of the IR remote) and also controlling and navigating a media_player (piCoreplayer/squeezebox) seem to favour the continuous mode command, even if I make changes to the intent name and to the intent <name> or {tags} to differentiate from the non continuous command. Not sure what the reason is for the difference.

I really struggled to work out how to do this so in case it helps anyone else, here are some examples of intents and their corresponding HA automations. Commands 1 and 3 can be daisy chained together to achieve “continuous mode”. You then exit continuous mode using command 2 once you are finished.

  1. Enter Continuous Mode:
    intent:
[ContinuousMode]
enter{action} continuous{mode} mode

HA Automation:

- alias: 'Enter Continuous Mode'
    trigger:
      platform: event
      event_type: rhasspy_ContinuousMode
      event_data:
        action: 'enter'
        mode: 'continuous'
    action: 
    - service: switch.toggle
      entity_id: switch.soundbar_mute
    - service: 'mqtt.publish'
      data_template:
        payload_template: '{"intentFilter": ["CancelCommand","ContinuousMode","ExitContinuousMode","ControlLightsCM"], "sessionId": "{{ trigger.event.data._intent.sessionId }}", "siteId":"{{ trigger.event.data._intent.siteId }}" }'
        topic: 'hermes/dialogueManager/continueSession' 
  1. Exit Continuous Mode:
    Note that I deliberately separated this and intent 1 for greater flexability, I may later put them back as a single intent and use an alternative for (enter|exit).
    intent:
[ExitContinuousMode]
exit{action} continuous{mode} mode

HA Automation:

- alias: 'Exit Continuous Mode'
    trigger:
      platform: event
      event_type: rhasspy_ExitContinuousMode
      event_data:
        action: 'exit'
        mode: 'continuous'
    action:
    - service: switch.toggle
      entity_id: switch.soundbar_mute
    - service: 'mqtt.publish'
      data_template:
        payload_template: '{"intentFilter": ["CancelCommand","ContinuousMode","ExitContinuousMode","ControlLightsCM"], "sessionId": "{{ trigger.event.data._intent.sessionId }}", "siteId":"{{ trigger.event.data._intent.siteId }}" }'
        topic: 'hermes/dialogueManager/endSession'
  1. Turn Lights On and Off (Continuous Mode)
    intent:
[ControlLightsCM]
l_name = (light 1 | light 2 | lights) {name}
l_state = (on | off) {state}

turn <l_state> [the] <l_name>
turn [the] <l_name> <l_state> 

HA Automation:

- alias: 'Turn on light 1 Continuous'
    trigger: 
      platform: event
      event_type: rhasspy_ControlLightsCM
      event_data: 
        name: 'light 1'
        state: 'on'
    action: 
    - service: switch.turn_on
      entity_id: switch.light_1
    - service: 'mqtt.publish'
      data_template:
        payload_template: '{"intentFilter": ["CancelCommand","ContinuousMode","ExitContinuousMode","ControlLightsCM"], "sessionId": "{{ trigger.event.data._intent.sessionId }}", "siteId":"{{ trigger.event.data._intent.siteId }}" }'
        topic: 'hermes/dialogueManager/continueSession'

Please note that the mute / unmute step in the Enter/Exit Continuous Mode intents isn’t strictly necessary, I just found it made life easier than having to shout my commands over the sound of the TV. Alternatively when I am choosing music and navigating between tracks I have an alternative separate Enter/Exit Continuous Mode intent pair that does not mute/unmute as I needed to be able to hear what song or podcast was playing to decide what to do with it.

Hopefully this is helpful to someone.

Thanks

4 Likes

hi !
I’ve tried this mode but i’m in trouble.
My goal was to shutdown my PS4 and having a question about the ampli and another one for the TV.
In the continueSession i’ve put the question but it’s take too long time to be said (2 or 3 sec) and TTS send à “done” in 0 sec.
So during rhasspy is in listen mode, the question is said … and i cant answer :

[2022-05-22 16:55:12] [Dialogue] was ask to continue session with id maitre-cassiopėe-8e9af0a1-fd5c-413f-9758-e9db3716771b by saying ‘est-ce que je dois aussi arrêter la télé ?’
with customData :
cassiopėe
[2022-05-22 16:55:13] [Asr] was asked to stop listening on site maitre
[2022-05-22 16:55:13] [hotword] was asked to toggle itself ‘off’ on site maitre
[2022-05-22 16:55:13] [hotword] was asked to toggle itself ‘off’ on site maitre
[2022-05-22 16:55:13] [Asr] was asked to toggle itself ‘off’ on site maitre
[2022-05-22 16:55:13] [Tts] was asked to say ‘est-ce que je dois aussi arrêter la télé ?’ in None on site maitre
[2022-05-22 16:55:13] [Tts] finished speaking with id ‘maitre-cassiopėe-8e9af0a1-fd5c-413f-9758-e9db3716771b’
[2022-05-22 16:55:13] [hotword] was asked to toggle itself ‘on’ on site maitre
[2022-05-22 16:55:13] [Asr] was asked to toggle itself ‘on’ on site maitre
[2022-05-22 16:55:13] [Asr] was asked to listen on site maitre
[2022-05-22 16:55:16] [Asr] captured text ‘’ in 0s on site maitre
[2022-05-22 16:55:16] [hotword] was asked to toggle itself ‘off’ on site maitre
[2022-05-22 16:55:16] [Asr] was asked to toggle itself ‘off’ on site maitre
[2022-05-22 16:55:17] [hotword] was asked to toggle itself ‘on’ on site maitre
[2022-05-22 16:55:17] [Asr] was asked to toggle itself ‘on’ on site maitre
[2022-05-22 16:55:17] [Asr] was asked to stop listening on site maitre
[2022-05-22 16:55:17] [hotword] was asked to toggle itself ‘on’ on site maitre
[2022-05-22 16:55:17] [Nlu] Intent not recognized for

I will try to move the question in a separated TTS before the continueSession.

To be continued …

So … i’ve put the question in a TTS and have to put a big delay before the continueSession to let rhasspy hear my response.
The automation in HASS look like this :

  • service: rest_command.tts_rhasspy
    data:
    site: ‘{% set site = trigger.event.data._site_id %} {{ site }}’
    lemessage: est-ce que je dois aussi arrêter la télé ?
    • delay: ‘00:00:06’
    • service: input_select.select_option
      target:
      entity_id:
      - input_select.objet
      data:
      option: tv
    • service: ‘mqtt.publish’
      data_template:
      payload_template: ‘{“intentFilter”:[“Negatif”,“Affirmatif”], “sendIntentNotRecognized”:“false”, “customData”:“cassiopėe”, “sessionId”:"{{ trigger.event.data._intent.sessionId }}", “siteId”:"{{ trigger.event.data._intent.siteId }}" }’
      topic: ‘hermes/dialogueManager/continueSession’

I’ve one intent for YES and another for NO.
And i set an input_select.objet to TV or AMPLI before sending the continueSession.
So i’ve only one automation for YES that can work on each element i want to stop (or another action, it’s depend on the question rhasspy asked to me).

I didn’t close the session : i can’t estimate the time taken by all the Q/R
I let rhasppy close it : after 26 seconds, it’s automatically closed