Incorrect/unexpected behaviour when intent is not recognized

Help
1

I have Rhasspy running in combination with node-red in docker-containers. Everything works fine except for one thing: when the intent is not recognized I want Rhasspy to speak out an “intent-not-recognized-phrase”: “I didn’t understand you, what did you say” (in Dutch: ik begreep jou niet, wat zei je"). When I enter a nonsensical sentence “e.g. blablablie” in the field “Recognize” (http://myrpiaddress:12101/), the system reacts correctly. However, when I say the same thing the system reacts incorrectly. I made a record of the session so you can hear what is happening:
https://www.dropbox.com/s/t4hmflfe0g6xzjk/intentnotrecognized.mp3?dl=1

These are the events happening:

  1. I speak out the activation word “snowboy” followed by the nonsensical phrase: blablablie
  2. The system reacts with two beeps but does not react by speaking out the “intent-not-recognized-phrase”.
  3. Then, I speak out the activation word “snowboy” once more
  4. The system reacts unexpectedly by speaking out the “intent-not-recognized-phrase” twice!

I have also made the following node-red flows available:

  1. The main flow
    https://www.dropbox.com/s/l5vvuxo6ronfjbg/mainflow.json?dl=1
  2. The unknown intent flow
    https://www.dropbox.com/s/awmuz2shmbmrp3e/unknownintentflow.json?dl=1
  3. One extra flow: the GetTime-flow
    https://www.dropbox.com/s/a6imtl4x6csg24o/gettimeflow.json?dl=1

Is this a bug? Is there a workaround?

kind regards,
Hugo

2

Can you share the log of this?
I suspect the session is not ended after the intent not recognized.
Then, the second wake also might end the first session or something like that.

But we need the log files.

3

Can you share the log of this?
I suspect the session is not ended after the intent not recognized.
Then, the second wake also might end the first session or something like that.

But we need the log files.

These are logs from Rhasspy:

Log van snowboy, blablablie
[DEBUG:2020-08-31 14:46:25,486] rhasspyserver_hermes: Handling TtsSayFinished (topic=hermes/tts/sayFinished, id=708183e8-3691-47be-951b-27d96da0936d)
[DEBUG:2020-08-31 14:46:22,916] rhasspyserver_hermes: Handling AudioPlayBytes (topic=hermes/audioServer/default/playBytes/780953c9-2790-4f9b-b42f-220a1343df1b, id=708183e8-3691-47be-951b-27d96da0936d)
[DEBUG:2020-08-31 14:46:22,837] rhasspyserver_hermes: Publishing 142 bytes(s) to hermes/tts/say
[DEBUG:2020-08-31 14:46:22,835] rhasspyserver_hermes: -> TtsSay(text=‘Ik begreep jou niet. Wat zei je?’, site_id=‘default’, lang=None, id=‘780953c9-2790-4f9b-b42f-220a1343df1b’, session_id=’’)
[DEBUG:2020-08-31 14:46:22,782] rhasspyserver_hermes: Sent 292 char(s) to websocket
[DEBUG:2020-08-31 14:46:22,779] rhasspyserver_hermes: Sent 292 char(s) to websocket
[DEBUG:2020-08-31 14:46:22,776] rhasspyserver_hermes: Sent 292 char(s) to websocket
[DEBUG:2020-08-31 14:46:22,774] rhasspyserver_hermes: Sent 292 char(s) to websocket
[DEBUG:2020-08-31 14:46:19,359] rhasspyserver_hermes: <- NluIntentNotRecognized(input=‘hoe’, site_id=‘default’, id=None, custom_data=None, session_id=‘default-snowboy-d067bdb1-d13c-4f48-9424-5002ea0aac92’)
[DEBUG:2020-08-31 14:46:17,729] rhasspyserver_hermes: Handling TtsSayFinished (topic=hermes/tts/sayFinished, id=20202d47-caf9-4da2-b777-1a71c90d93fc)
[DEBUG:2020-08-31 14:46:15,073] rhasspyserver_hermes: Handling AudioPlayBytes (topic=hermes/audioServer/default/playBytes/b91e9b0d-e10e-4b09-bcae-639a21c44009, id=20202d47-caf9-4da2-b777-1a71c90d93fc)
[DEBUG:2020-08-31 14:46:14,992] rhasspyserver_hermes: Publishing 142 bytes(s) to hermes/tts/say
[DEBUG:2020-08-31 14:46:14,991] rhasspyserver_hermes: -> TtsSay(text=‘Ik begreep jou niet. Wat zei je?’, site_id=‘default’, lang=None, id=‘b91e9b0d-e10e-4b09-bcae-639a21c44009’, session_id=’’)
[DEBUG:2020-08-31 14:46:14,951] rhasspyserver_hermes: Sent 336 char(s) to websocket
[DEBUG:2020-08-31 14:46:14,949] rhasspyserver_hermes: Sent 336 char(s) to websocket
[DEBUG:2020-08-31 14:46:14,947] rhasspyserver_hermes: Sent 336 char(s) to websocket
[DEBUG:2020-08-31 14:46:14,944] rhasspyserver_hermes: Sent 336 char(s) to websocket
[DEBUG:2020-08-31 14:46:14,939] rhasspyserver_hermes: <- HotwordDetected(model_id=‘snowboy’, model_version=’’, model_type=‘personal’, current_sensitivity=0.5, site_id=‘default’, session_id=None, send_audio_captured=None, lang=None)
[DEBUG:2020-08-31 14:46:11,242] rhasspyserver_hermes: <- NluIntentNotRecognized(input=‘hoe warm is’, site_id=‘default’, id=None, custom_data=None, session_id=‘default-snowboy-ad9a44f1-b10a-4f1b-b623-81a2336b6d61’)
[DEBUG:2020-08-31 14:46:06,308] rhasspyserver_hermes: <- HotwordDetected(model_id=‘snowboy’, model_version=’’, model_type=‘personal’, current_sensitivity=0.5, site_id=‘default’, session_id=None, send_audio_captured=None, lang=None)

Log van snowboy, abacadabra:
[DEBUG:2020-08-31 14:50:46,279] rhasspyserver_hermes: Handling TtsSayFinished (topic=hermes/tts/sayFinished, id=70421102-316c-499b-9f79-c434b069162e)
[DEBUG:2020-08-31 14:50:43,713] rhasspyserver_hermes: Handling AudioPlayBytes (topic=hermes/audioServer/default/playBytes/1baca2e2-a540-4c30-9446-8b82d00ed578, id=70421102-316c-499b-9f79-c434b069162e)
[DEBUG:2020-08-31 14:50:43,633] rhasspyserver_hermes: Publishing 142 bytes(s) to hermes/tts/say
[DEBUG:2020-08-31 14:50:43,632] rhasspyserver_hermes: -> TtsSay(text=‘Ik begreep jou niet. Wat zei je?’, site_id=‘default’, lang=None, id=‘1baca2e2-a540-4c30-9446-8b82d00ed578’, session_id=’’)
[DEBUG:2020-08-31 14:50:43,607] rhasspyserver_hermes: Sent 292 char(s) to websocket
[DEBUG:2020-08-31 14:50:43,606] rhasspyserver_hermes: Sent 292 char(s) to websocket
[DEBUG:2020-08-31 14:50:43,604] rhasspyserver_hermes: Sent 292 char(s) to websocket
[DEBUG:2020-08-31 14:50:43,603] rhasspyserver_hermes: Sent 292 char(s) to websocket
[DEBUG:2020-08-31 14:50:40,922] rhasspyserver_hermes: <- NluIntentNotRecognized(input=‘hoe’, site_id=‘default’, id=None, custom_data=None, session_id=‘default-snowboy-ff41b0e2-dff7-41bf-ac7e-31ef94844dec’)
[DEBUG:2020-08-31 14:50:38,573] rhasspyserver_hermes: Handling TtsSayFinished (topic=hermes/tts/sayFinished, id=7dd301fc-6828-4daf-943b-2b32b5de7125)
[DEBUG:2020-08-31 14:50:35,931] rhasspyserver_hermes: Handling AudioPlayBytes (topic=hermes/audioServer/default/playBytes/53f28481-a1f8-4a60-b925-1afe66f30f8f, id=7dd301fc-6828-4daf-943b-2b32b5de7125)
[DEBUG:2020-08-31 14:50:35,860] rhasspyserver_hermes: Publishing 142 bytes(s) to hermes/tts/say
[DEBUG:2020-08-31 14:50:35,859] rhasspyserver_hermes: -> TtsSay(text=‘Ik begreep jou niet. Wat zei je?’, site_id=‘default’, lang=None, id=‘53f28481-a1f8-4a60-b925-1afe66f30f8f’, session_id=’’)
[DEBUG:2020-08-31 14:50:35,822] rhasspyserver_hermes: Sent 318 char(s) to websocket
[DEBUG:2020-08-31 14:50:35,817] rhasspyserver_hermes: Sent 318 char(s) to websocket
[DEBUG:2020-08-31 14:50:35,812] rhasspyserver_hermes: Sent 318 char(s) to websocket
[DEBUG:2020-08-31 14:50:35,810] rhasspyserver_hermes: Sent 318 char(s) to websocket
[DEBUG:2020-08-31 14:50:35,805] rhasspyserver_hermes: <- HotwordDetected(model_id=‘snowboy’, model_version=’’, model_type=‘personal’, current_sensitivity=0.5, site_id=‘default’, session_id=None, send_audio_captured=None, lang=None)
[DEBUG:2020-08-31 14:50:29,792] rhasspyserver_hermes: <- NluIntentNotRecognized(input=‘hoe koud’, site_id=‘default’, id=None, custom_data=None, session_id=‘default-snowboy-9a97c617-5ab0-4138-bc6e-ad65af3aaffb’)
[DEBUG:2020-08-31 14:50:24,543] rhasspyserver_hermes: <- HotwordDetected(model_id=‘snowboy’, model_version=’’, model_type=‘personal’, current_sensitivity=0.5, site_id=‘default’, session_id=None, send_audio_captured=None, lang=None)

I noticed some parts of my sentences are visible in the log e.g. “hoe”, "hoe warm is"n “hoe koud” (wat ik niet gezegd heb)

kind regards,
hugo

4

Can you show the output of:

mosquitto_sub -t 'hermes/#' -T 'hermes/audioServer/+/playBytes/+' -T 'hermes/audioServer/+/audioFrame' -v

while these events are happening?

I find this much easier to parse than the raw logs of Rhasspy.

5

I don’t see any output giving this command on my raspberry

hugo

6

Ok you may have to add the connection settings for the MQTT broker use for Rhasspy.

7

Ok you may have to add the connection settings for the MQTT broker use for Rhasspy.

Can you tell me how to do that?
You can see the contents of my docker-compose.yml here below:

---
version: '3.7'

services:
 pigpiod:
  image: corbosman/pigpiod
  container_name: pigpiod
  restart: always
  privileged: true
  command: -t 0
 rhasspy:
  image: rhasspy/rhasspy
  container_name: rhasspy
  restart: always
  volumes:
   - /home/pi/containers/rhasspy/profiles:/profiles
  ports:
   - 12101:12101
  devices:
   - /dev/snd:/dev/snd
  command: --user-profiles /profiles --profile nl
 node-red:
  image: nodered/node-red
  container_name: node-red
  restart: always
  volumes:
   - ./containers/node-red:/data
   - ./containers/certificates:/etc/ssl/private:ro
   - /etc/localtime:/etc/localtime:ro
  ports:
   - 1880:1880
  environment:
   - TZ=Europe/Brussels
 mosquitto:
  image: eclipse-mosquitto
  container_name: mosquitto
  restart: always
  ports:
   - "1883:1883"
   - "9001:9001"
  volumes:
   - ./containers/mosquitto/config:/mosquitto/config
   - ./containers/mosquitto/data:/mosquitto/data
   - ./containers/mosquitto/log:/mosquitto/log
   - /etc/localtime:/etc/localtime:ro
  user: "1000:1000"

kind regards,
hugo

8

What’s your Mosquitto setup? With or without TLS? With or withour authentication?

9

What’s your Mosquitto setup? With or without TLS? With or withour authentication?

it’s currently without TLS without authentication (I first want to get the system working, then I’ll add the security measures)

hugo

10

Then on the same machine you should be able to run the command I gave. This only shows output of course when Rhasspy is sending MQTT messages.

On a different machine you need to add --host HOSTNAME with HOSTNAME the hostname of your Raspberry Pi running the Mosquitto container.

This all assuming you have configured Rhasspy to use this external MQTT broker running in the Mosquitto container.

11

I’m running the command on the same machine.
You can see how I configured a mqtt-in node in this picture:
https://www.dropbox.com/s/kp6utg5czzyl297/mqtt_in_node.png?dl=1
and a mqtt-out node
https://www.dropbox.com/s/wz0jmnvj3pp138o/mqtt_out_node.png?dl=1

And a screenshot of my rhasspy settings:
https://www.dropbox.com/s/55epp8dn2541del/rhasspysettings.png?dl=1

The important flows themselves were posted in the original message here above

kind regards,
hugo

12

Apparently my Rhasspy was all the time using the internal MQTT-broker. I changed the setting to external (host mosquitto)
and now I see the output you were asking for Koan. B.t.w. the unexpected behaviour didn’t change.
Here is the output you asked for (snowboy, blablablie, snowboy):

hermes/hotword/snowboy/detected {“modelId”: “snowboy”, “modelVersion”: “”, “modelType”: “personal”, “currentSensitivity”: 0.5, “siteId”: “default”, “sessionId”: null, “sendAudioCaptured”: null, “lang”: null}
hermes/hotword/toggleOff {“siteId”: “default”, “reason”: “playAudio”}
hermes/asr/toggleOff {“siteId”: “default”, “reason”: “playAudio”}
hermes/audioServer/default/playFinished {“id”: “fb1357cf-1c7a-4d2a-a871-96b2ae6c4c10”, “sessionId”: “”}
hermes/hotword/toggleOn {“siteId”: “default”, “reason”: “playAudio”}
hermes/asr/toggleOn {“siteId”: “default”, “reason”: “playAudio”}
hermes/dialogueManager/sessionStarted {“sessionId”: “default-snowboy-04c8f25e-e1d3-4578-b7db-5b2fe74fb611”, “siteId”: “default”, “customData”: “snowboy”, “lang”: null}
hermes/hotword/toggleOff {“siteId”: “default”, “reason”: “dialogueSession”}
hermes/asr/startListening {“siteId”: “default”, “sessionId”: “default-snowboy-04c8f25e-e1d3-4578-b7db-5b2fe74fb611”, “lang”: null, “stopOnSilence”: true, “sendAudioCaptured”: true, “wakewordId”: “snowboy”, “intentFilter”: null}
hermes/asr/textCaptured {“text”: “drie”, “likelihood”: 1, “seconds”: 2.2260784999998577, “siteId”: “default”, “sessionId”: “default-snowboy-04c8f25e-e1d3-4578-b7db-5b2fe74fb611”, “wakewordId”: null, “asrTokens”: null, “lang”: null}
hermes/hotword/toggleOff {“siteId”: “default”, “reason”: “playAudio”}
hermes/asr/toggleOff {“siteId”: “default”, “reason”: “playAudio”}
hermes/audioServer/default/playFinished {“id”: “8794a60a-c373-4b84-b4ca-f6c0ce69697f”, “sessionId”: “”}
hermes/hotword/toggleOn {“siteId”: “default”, “reason”: “playAudio”}
hermes/asr/toggleOn {“siteId”: “default”, “reason”: “playAudio”}
hermes/asr/stopListening {“siteId”: “default”, “sessionId”: “default-snowboy-04c8f25e-e1d3-4578-b7db-5b2fe74fb611”}
hermes/hotword/toggleOn {“siteId”: “default”, “reason”: “dialogueSession”}
hermes/nlu/query {“input”: “drie”, “siteId”: “default”, “id”: null, “intentFilter”: null, “sessionId”: “default-snowboy-04c8f25e-e1d3-4578-b7db-5b2fe74fb611”, “wakewordId”: “snowboy”, “lang”: null}
hermes/nlu/intentNotRecognized {“input”: “drie”, “siteId”: “default”, “id”: null, “customData”: null, “sessionId”: “default-snowboy-04c8f25e-e1d3-4578-b7db-5b2fe74fb611”}
hermes/hotword/toggleOff {“siteId”: “default”, “reason”: “playAudio”}
hermes/asr/toggleOff {“siteId”: “default”, “reason”: “playAudio”}
hermes/audioServer/default/playFinished {“id”: “8bccbafb-c65d-42e1-a280-3dfa95059930”, “sessionId”: “”}
hermes/hotword/toggleOn {“siteId”: “default”, “reason”: “playAudio”}
hermes/asr/toggleOn {“siteId”: “default”, “reason”: “playAudio”}
hermes/asr/stopListening {“siteId”: “default”, “sessionId”: “default-snowboy-04c8f25e-e1d3-4578-b7db-5b2fe74fb611”}
hermes/dialogueManager/sessionEnded {“termination”: {“reason”: “intentNotRecognized”}, “sessionId”: “default-snowboy-04c8f25e-e1d3-4578-b7db-5b2fe74fb611”, “siteId”: “default”, “customData”: “snowboy”}
hermes/hotword/toggleOn {“siteId”: “default”, “reason”: “dialogueSession”}
hermes/hotword/snowboy/detected {“modelId”: “snowboy”, “modelVersion”: “”, “modelType”: “personal”, “currentSensitivity”: 0.5, “siteId”: “default”, “sessionId”: null, “sendAudioCaptured”: null, “lang”: null}
hermes/hotword/toggleOff {“siteId”: “default”, “reason”: “playAudio”}
hermes/asr/toggleOff {“siteId”: “default”, “reason”: “playAudio”}
hermes/tts/say {“text”: “Ik begreep jou niet. Wat zei je?”, “siteId”: “default”, “lang”: null, “id”: “0127d5d7-f6fa-47dd-8bfc-96ff8f891811”, “sessionId”: “”}
hermes/audioServer/default/playFinished {“id”: “0bdc4d61-52b9-462f-9eef-b2af3195e395”, “sessionId”: “”}
hermes/hotword/toggleOn {“siteId”: “default”, “reason”: “playAudio”}
hermes/asr/toggleOn {“siteId”: “default”, “reason”: “playAudio”}
hermes/dialogueManager/sessionStarted {“sessionId”: “default-snowboy-8d5c7f77-169b-4515-916a-31b0d6a552f7”, “siteId”: “default”, “customData”: “snowboy”, “lang”: null}
hermes/hotword/toggleOff {“siteId”: “default”, “reason”: “dialogueSession”}
hermes/asr/startListening {“siteId”: “default”, “sessionId”: “default-snowboy-8d5c7f77-169b-4515-916a-31b0d6a552f7”, “lang”: null, “stopOnSilence”: true, “sendAudioCaptured”: true, “wakewordId”: “snowboy”, “intentFilter”: null}
hermes/tts/sayFinished {“siteId”: “default”, “id”: “0127d5d7-f6fa-47dd-8bfc-96ff8f891811”, “sessionId”: “”}
hermes/audioServer/default/playFinished {“id”: “0127d5d7-f6fa-47dd-8bfc-96ff8f891811”, “sessionId”: “”}
hermes/asr/textCaptured {“text”: “op”, “likelihood”: 1, “seconds”: 2.1021692829999665, “siteId”: “default”, “sessionId”: “default-snowboy-8d5c7f77-169b-4515-916a-31b0d6a552f7”, “wakewordId”: null, “asrTokens”: null, “lang”: null}
hermes/hotword/toggleOff {“siteId”: “default”, “reason”: “playAudio”}
hermes/asr/toggleOff {“siteId”: “default”, “reason”: “playAudio”}
hermes/audioServer/default/playFinished {“id”: “c7d0853c-270f-4815-b5e9-7349d2ec3bdc”, “sessionId”: “”}
hermes/hotword/toggleOn {“siteId”: “default”, “reason”: “playAudio”}
hermes/asr/toggleOn {“siteId”: “default”, “reason”: “playAudio”}
hermes/asr/stopListening {“siteId”: “default”, “sessionId”: “default-snowboy-8d5c7f77-169b-4515-916a-31b0d6a552f7”}
hermes/hotword/toggleOn {“siteId”: “default”, “reason”: “dialogueSession”}
hermes/nlu/query {“input”: “op”, “siteId”: “default”, “id”: null, “intentFilter”: null, “sessionId”: “default-snowboy-8d5c7f77-169b-4515-916a-31b0d6a552f7”, “wakewordId”: “snowboy”, “lang”: null}
hermes/nlu/intentNotRecognized {“input”: “op”, “siteId”: “default”, “id”: null, “customData”: null, “sessionId”: “default-snowboy-8d5c7f77-169b-4515-916a-31b0d6a552f7”}
hermes/hotword/toggleOff {“siteId”: “default”, “reason”: “playAudio”}
hermes/asr/toggleOff {“siteId”: “default”, “reason”: “playAudio”}
hermes/audioServer/default/playFinished {“id”: “49299fcf-b5b1-46fc-8e2d-8f0da97b61e3”, “sessionId”: “”}
hermes/hotword/toggleOn {“siteId”: “default”, “reason”: “playAudio”}
hermes/asr/toggleOn {“siteId”: “default”, “reason”: “playAudio”}
hermes/asr/stopListening {“siteId”: “default”, “sessionId”: “default-snowboy-8d5c7f77-169b-4515-916a-31b0d6a552f7”}
hermes/dialogueManager/sessionEnded {“termination”: {“reason”: “intentNotRecognized”}, “sessionId”: “default-snowboy-8d5c7f77-169b-4515-916a-31b0d6a552f7”, “siteId”: “default”, “customData”: “snowboy”}
hermes/hotword/toggleOn {“siteId”: “default”, “reason”: “dialogueSession”}
hermes/tts/say {“text”: “Ik begreep jou niet. Wat zei je?”, “siteId”: “default”, “lang”: null, “id”: “b42c1201-d653-497a-ac48-0205e6bce4b6”, “sessionId”: “”}
hermes/audioServer/default/playFinished {“id”: “b42c1201-d653-497a-ac48-0205e6bce4b6”, “sessionId”: “”}
hermes/tts/sayFinished {“siteId”: “default”, “id”: “b42c1201-d653-497a-ac48-0205e6bce4b6”, “sessionId”: “”}

kind regards,
hugo

13

I think the unexpected behaviour is because this simple example is not using the dialogue manager’s messages (to end a session), but the simple TtsSay message. There have been some improvements in dialogue management in Rhasspy in recent months.

14

I think the unexpected behaviour is because this simple example is not using the dialogue manager’s messages (to end a session), but the simple TtsSay message. There have been some improvements in dialogue management in Rhasspy in recent months.

Is it a matter of changing something in the rhasspy-settings or is it more involved (I guess it will be )

kind regards,
hugo

15

It’s slightly more involved. Instead of sending a TtsSay message, you reply with a DialogueEndSession message that has the same session ID as the session ID of your IntentNotRecognized message.

Don’t ask me for a flow, I don’t have one at hand :slight_smile: Maybe you can find some examples on the forum here. I prefer Python for this anyway.

16

I wish I could, but I didn’t find anything until now, it’s difficult to look for something you don’t know

hugo

17

May I conclude the following:

  1. the behaviour I showed is indeed due to a bug
  2. the bug might dissappear in a future version of Rhasspy

kind regards,
Hugo

18

I think so. Can you open an issue and link to this forum topic?

20

OK, I just did that
kind regards,
Hugo