Training issue with Rhasspy 2.5: string index out of range

First of all. I try to clear whats your problem:

  • You are using iobroker to control whatever
  • You want to use rhasspy to get inputs by voice
  • You want to “read” the intents and slots with mqtt in iobroker to control things.
  • You still have problems to use the intern mqtt-Broker (Server) of ioBroker

I repeat:

  • You need a mqtt-Broker (Server), which can be reached from ALL your devices (ioBroker, Rhasspy-Server, Rhasspy-Clients)
  • All of them must be able to publish and subscribe messages
    If this works with ioBroker you got it

I did not get this running. So I use my Docker MQTT-Server.
If you need another Broker (outsite ioBroker) your new Problem is to “read” the mqtt-messages insite ioBroker and use them to control your stuff.
I’m not sure, so don’t be upset. I will explain you how mqtt works.

Using MQTT is like watching videos on YOUTUBE

  • There is Microsoft, who has the YOUTUBE-Server (lets say, its only one)
    There is ONE MQTT-Server ( 192.168.13.152, In Docker or in ioBroker)

  • There are many users who upload videos to YOUTUBE server
    Everybody can upload messages to the MQTT server.

    mosquitto_pub -h 192.168.13.152:1883 -t “test” -m “test-message”
    
  • There are many users who watches videos. The video is send to them, when they are online.
    Everybody can subscribe to the MQTT-Server and will receive messages as soon as they are send

    mosquitto_sub -h 192.168.13.152:1883 -t “test”
    

When You want to receive messages in ioBroker, you need a Tool, that can subscribe to the MQTT-Broker. This can be:

  • a program in Python (Python Skill Code)
  • a node in nodered (thats what I’m using)
  • a program in javascript (not my business)
  • (a oBroker MQTT-Client (I’m not sure. I think there is one)

I still not know, what is working in your home

Test ioBroker as MQTT Broker (SERVER):

  • let it run (SERVER !!!)
  • subscripe from rhasspy-sat and publish from rhasspy-Server
  • If it works, try it the other way
  • if its not working, try to install another mqtt-Broker,
    (Docker is the easiest way, or try it on a raspberry 3)
  • If THIS WORKS, ioBrokers MQTT is the evil.
  • If not, you need an exorcist :japanese_ogre:
  • If it works, anyway which solution, let ALL your rhasspys point to the IP-Adress of the MQTT-Broker.

If you have an MQTT-Server working and the rhasspys are working with it, then let’s talk about using mqtt-messages.
As I said, I use nodered. And only nodered because all devices, that I used in ioBroker are also supported by nodered.
Awaiting your reports

Kay

Hi Kay,
I am just at the beginning of a conference call. So just briefly:

I have just installed the broker. It’s up and running. Until now and because of my call: currently no further experiences.

Will come back later

Hi Kay,
I’m coming back for a short report. Of course I haven’t been able to perform a complete test in the meantime. However I have some impressions I would like to share.

Let me start with your considerations:

In my case its Javascript, and for quick drafts and testing blockly and nodered

In a first step I have used ioBroker’s MQTT adapter to mediate between Rhasspy and ioBroker. The MQTT adapter was able to identify the intents spread by Rhasspy. Simultaneously, Rhasspy was not able to connect to this broker and consequencially, intent recognition failed.

As I have reported briefly, I have installed an eclipse mqtt broker on IP …155 with port 1883. Rhasspy sat and master are both pointing to this address. In order to make mqtt intents visible I have also installed a mqtt client on iobroker, that also points to this address. Still I have not redirected my Javascripts to the adresses of this client. But this is not an issue.

Now I have got this situation:

  1. I was able to see in the master’s and in the satellite’s terminal that they have connected to the new mqtt broker
  2. the mqtt client on iobroker is showing all intents published by the new mqtt broker
  3. However both, the master and the satellite, are reporting the lookup error on the mosquitto_pub and …-sub command
  4. netstat on the satellite is reporting:

pi@Rhasspy-Sat1:~ $ netstat -tln | grep 1883
tcp 0 0 0.0.0.0:1883 0.0.0.0:* LISTEN

  1. netstat on the master is reporting:

pi@ioBroker-Master:/opt/mqtt$ netstat -tln | grep 1883
tcp6 0 0 :::1883 :::* LISTEN

Thus no IP V4!!!

  1. netmap on sat:

pi@Rhasspy-Sat1:~ $ nmap 192.168.13.155 -p 1883
Starting Nmap 7.70 ( https://nmap.org ) at 2020-04-16 23:40 CEST
Nmap scan report for ioBroker-Master.fritz.box (192.168.13.155)
Host is up (0.0015s latency).
PORT STATE SERVICE
1883/tcp open mqtt
Nmap done: 1 IP address (1 host up) scanned in 0.13 seconds

  1. nmap on master:

pi@ioBroker-Master:/opt/mqtt$ nmap 192.168.13.155 -p 1883
Starting Nmap 7.60 ( https://nmap.org ) at 2020-04-16 23:42 CEST
Nmap scan report for ioBroker-Master (192.168.13.155)
Host is up (0.00019s latency).
PORT STATE SERVICE
1883/tcp open mqtt
Nmap done: 1 IP address (1 host up) scanned in 0.08 seconds

So, besides the missing TCP IP V4 on the master side, everything seems to be OK

However, the texts in the terminals of master and satellite are continoulsy running, showing error messages at the end. Due to these activities the entire system (both Rhasspy and ioBroker) are being lamed. (Or is this caused by the new mqtt broker? I still haven’t find out yet.)

Error message on Master:

[ERROR:2020-04-16 21:52:19,131] rhasspyasr_kaldi_hermes: parse_mqtt_message (topic=hermes/asr/startListening)
Traceback (most recent call last):
File “/usr/lib/rhasspy-voltron/rhasspy-hermes/rhasspyhermes/client.py”, line 303, in parse_mqtt_message
json_payload = json.loads(payload)
File “/usr/lib/python3.7/json/init.py”, line 348, in loads
return _default_decoder.decode(s)
File “/usr/lib/python3.7/json/decoder.py”, line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File “/usr/lib/python3.7/json/decoder.py”, line 355, in raw_decode
raise JSONDecodeError(“Expecting value”, s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Error message on satellite:

RuntimeError
[ERROR:2020-04-16 21:52:45,596] rhasspyremote_http_hermes: handle_audio_frame
Traceback (most recent call last):
File “/usr/lib/rhasspy-voltron/rhasspy-remote-http-hermes/rhasspyremote_http_hermes/init.py”, line 465, in handle_audio_frame
with wave.open(in_io) as in_wav:
File “/usr/lib/python3.7/wave.py”, line 510, in open
return Wave_read(f)
File “/usr/lib/python3.7/wave.py”, line 164, in init
self.initfp(f)
File “/usr/lib/python3.7/wave.py”, line 153, in initfp
chunk.skip()
File “/usr/lib/python3.7/chunk.py”, line 160, in skip
self.file.seek(n, 1)
File “/usr/lib/python3.7/chunk.py”, line 113, in seek
raise RuntimeError
RuntimeError

These error messages do not occur, when the Rhasspy master is using it internal mqtt broker and the satellite is pointing to this.

So, still there must be something different between your and my configuration. I don’t think it’s a big story.

In order to sort out, I would like to ask the following questions:

  1. What user and group are you using for mqtt. My system looks like this:

pi@ioBroker-Master:/opt$ ls -l
total 16
drwx–x--x 4 root root 4096 Feb 15 14:29 containerd
drwxrwxr-x+ 10 iobroker iobroker 4096 Apr 16 17:24 iobroker
drwxrwxr-x+ 11 iobroker iobroker 4096 Apr 10 11:17 iobrokerold
drwxr-xr-x 3 root root 4096 Apr 16 18:23 mqtt
pi@ioBroker-Master:/opt/mqtt$ ls -l
total 12
-rw-r–r-- 1 root root 315 Apr 16 18:22 docker-compose.yml
-rw-r–r-- 1 root root 1644 Apr 16 18:17 mosquitto.conf
drwxr-xr-x 2 root root 4096 Apr 16 18:23 pwfile

  1. On my satellite I have configured site-ID = Satellite1 and Satellite site-ID: Satellite1 in each section. Is this correct or do you have it different? Because in section “Audio Recording” the first and only time “Output siteId” appears, while in all other sections it is called “Satellite site-ID”.
  2. Finally: I have nothing configured regarding UDP. Is this correct?

My conclusion: With the installation of an new mqtt broker and a mqtt client located on iobroker, most mqtt issues seem to be resolved. Still there are some issues with this fgrxzqyc Version 2.5

How can we sort them out?

Nevertheless, you put me on the right track, which was very helpfull. Thank you very much again for your kind help.

Best Thomas

Hi Thomas,
I tried something and the result is strange:
NOT Working:

mosquitto_pub -h 10.2.254.1:1883 -t “test” -m “test-message”
mosquitto_sub -h 10.2.254.1:1883 -t “test”

WORKING !

mosquitto_pub -h 10.2.254.1 -t “test” -m “test-message”
mosquitto_sub -h 10.2.254.1 -t “test”

Also WORKING !

mosquitto_pub -h 10.2.254.1 -p 1883 -t “test” -m “test-message”
mosquitto_sub -h 10.2.254.1 -p 1883 -t “test”

So please try it. I guess its working also with you and everything (mqtt) is OK.

–> Correct !!
-> Everywhere in client-places must be the ID of the client
-> Everywhere in master-places (if used by mqtt) must be ALL IDs of the clients

–> You can use it or leave it.
If you leave it, you will have a lot of traffic. I use it.

Here you can compare the configs:

My Satellite-IDs: wohnzimmer,badezimmer,küche,schlafzimmer
(Wohnzimmer has its own wakeword)
My Master-IP: 10.2.254.1
My maryTTS-IP: 10.2.254.1

Master:

{
    "dialogue": {
        "satellite_site_ids": "wohnzimmer,badezimmer,küche,schlafzimmer",
        "system": "rhasspy"
    },
    "intent": {
        "satellite_site_ids": "wohnzimmer,badezimmer,küche,schlafzimmer",
        "system": "fsticuffs"
    },
    "mqtt": {
        "enabled": "true",
        "host": "10.2.254.1",
        "site_id": "master"
    },
    "sounds": {
        "error": "${RHASSPY_PROFILE_DIR}/wav/beep.wav",
        "recorded": "${RHASSPY_PROFILE_DIR}/wav/answer.wav",
        "wake": "${RHASSPY_PROFILE_DIR}/wav/question.wav"
    },
    "speech_to_text": {
        "satellite_site_ids": "wohnzimmer,badezimmer,küche,schlafzimmer",
        "system": "kaldi"
    },
    "text_to_speech": {
        "marytts": {
            "locale": "de",
            "url": "http://10.2.254.1:59125/process",
            "voice": "bits1-hsmm"
        },
        "satellite_site_ids": "badezimmer,küche,schlafzimmer",
        "system": "marytts"
    }
}

Satellite:

{
    "intent": {
        "system": "hermes"
    },
    "microphone": {
        "arecord": {
            "device": "default:CARD=seeed2micvoicec",
            "siteId": "schlafzimmer",
            "udp_audio_port": "12202"
        },
        "system": "arecord"
    },
    "mqtt": {
        "enabled": "true",
        "host": "10.2.254.1",
        "site_id": "schlafzimmer"
    },
    "sounds": {
        "aplay": {
            "device": "sysdefault:CARD=seeed2micvoicec"
        },
        "recorded": "${RHASSPY_PROFILE_DIR}/wav/answer.wav",
        "system": "aplay",
        "wake": "${RHASSPY_PROFILE_DIR}/wav/question.wav"
    },
    "speech_to_text": {
        "system": "hermes"
    },
    "text_to_speech": {
        "system": "hermes"
    },
    "wake": {
        "snowboy": {
            "model": "computer_2mic.pmdl",
            "sensitivity": "0.45",
            "udp_audio_port": "12202"
        },
        "system": "snowboy"
    }
}

Hi Kay,

sorry, I had some missions for my wife to complete and had no time to attend to Rhasspy.

Thank you very much for your extensive explanations, which are very helpful. I will start tomorrow to look after the Rhasspy configuration. Using UDP in order to reduce the amount of traffic on the network seems extremely valuable to me. Unless you could explain to me, what needless and surplus traffic is good for :joy:

OK, have a good week
Thomas

Hello Kay,

I apologize to have to contact you again, but Rhasspy 2.5 is very much different to it’s previous version. I am still not familiar with it.

OK: In the meantime I set up the system as you recommended it. Intent recognition works fine with the new MQTT broker . Switching back to ioBroker’s MQTT adapter confirmed what you said: This combination with Rhasspy does not work.

On my MQTT client I am now able to see the data model and the intents and I recognised some differences to Rhasspy 2.4.19.

In 2.4.19 Rhasspy’s all intents were included in mqtt.hermes.intent with their complete JSON string and also in mqtt.rhasspy.intent with the relevant information (e.g. device, state, level etc. as it is also shown in the rhasspy home screen).

My Javascripts picked up intents under mqtt.rhasspy.intent to trigger the devices and applications.

Opposed to this in 2.5 only one intent under mqtt.rhasspy.intent has been created automatically and with no content. I created another intent manually, however this also does not receive any data.

How did you manage it to forward the intents correctly?

Furthermore I found out, that two intents obviously do not exist any more. These are “mqtt.rhasspy.de.transition.SnowboyWakeListener” and “…WebrtcvadCommandListener”, which we used to send back voice messages (confirmations) to Rhasspy’s speakers.

I assume, this has been resolved differently now. Do you know how?

Thank you very much in advance

Best
Thomas

Hi Thomas,
no problem. I stay at home.

reading intents : hermes/intent/#
sesion startet: hermes/dialogueManager/sessionStarted
finish session: hermes/dialogueManager/endSession
say something: hermes/tts/say

Here’s something for nodeRed (version 1):

I save the siteId and in second example the sessionId, cause in some nodes all payload are destroyed.

[{"id":"341f829c.eef3ee","type":"mqtt in","z":"27a1dcd5.ab7ad4","name":"intentIsRecognized","topic":"hermes/intent/#","qos":"2","datatype":"json","broker":"6f747297.64151c","x":110,"y":520,"wires":[["bce45bd4.746cd8"]]},{"id":"185bb0bc.bae1df","type":"mqtt out","z":"27a1dcd5.ab7ad4","name":"saySomething","topic":"hermes/tts/say","qos":"","retain":"","broker":"6f747297.64151c","x":980,"y":520,"wires":[]},{"id":"bd10ced8.e046","type":"change","z":"27a1dcd5.ab7ad4","name":"doSomething","rules":[{"t":"set","p":"payload","pt":"msg","to":"Etwas wurde erledigt","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":470,"y":460,"wires":[["237a18ca.addcf8"]]},{"id":"bce45bd4.746cd8","type":"change","z":"27a1dcd5.ab7ad4","name":"save siteId","rules":[{"t":"set","p":"mySiteId","pt":"global","to":"payload.siteId","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":310,"y":520,"wires":[["bd10ced8.e046"]]},{"id":"5ffd1b8a.cd8534","type":"change","z":"27a1dcd5.ab7ad4","name":"prepare Answer","rules":[{"t":"set","p":"payload","pt":"msg","to":"{\t \"text\":msg.payload,\t \"lang\":\"de\",\t \"id\":\"\",\t \"siteId\":msg.siteId,\t \"sessionId\":\"\"\t \t}","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":800,"y":520,"wires":[["185bb0bc.bae1df"]]},{"id":"237a18ca.addcf8","type":"change","z":"27a1dcd5.ab7ad4","name":"read siteId","rules":[{"t":"set","p":"siteId","pt":"msg","to":"mySiteId","tot":"global"}],"action":"","property":"","from":"","to":"","reg":false,"x":630,"y":520,"wires":[["5ffd1b8a.cd8534"]]},{"id":"6f747297.64151c","type":"mqtt-broker","z":"","name":"MQTT","broker":"10.2.254.1","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]

Here’s something for nodeRed (version 2):

now I also save the sessionId to end the session after answering

[{"id":"71ad8e3c.aa46c","type":"mqtt in","z":"27a1dcd5.ab7ad4","name":"sessionStarted","topic":"hermes/dialogueManager/sessionStarted","qos":"2","datatype":"json","broker":"6f747297.64151c","x":100,"y":580,"wires":[["5ef9514c.863a5"]]},{"id":"341f829c.eef3ee","type":"mqtt in","z":"27a1dcd5.ab7ad4","name":"intentIsRecognized","topic":"hermes/intent/#","qos":"2","datatype":"json","broker":"6f747297.64151c","x":110,"y":520,"wires":[["bce45bd4.746cd8"]]},{"id":"185bb0bc.bae1df","type":"mqtt out","z":"27a1dcd5.ab7ad4","name":"saySomething","topic":"hermes/tts/say","qos":"","retain":"","broker":"6f747297.64151c","x":980,"y":520,"wires":[]},{"id":"e350059e.44f328","type":"mqtt out","z":"27a1dcd5.ab7ad4","name":"endSession","topic":"hermes/dialogueManager/endSession","qos":"","retain":"","broker":"6f747297.64151c","x":970,"y":580,"wires":[]},{"id":"5ef9514c.863a5","type":"change","z":"27a1dcd5.ab7ad4","name":"saveSession","rules":[{"t":"set","p":"session","pt":"global","to":"{ \"sessionId\":msg.payload.sessionId, \"siteId\": msg.payload.siteId }","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":310,"y":580,"wires":[[]]},{"id":"bd10ced8.e046","type":"change","z":"27a1dcd5.ab7ad4","name":"doSomething","rules":[{"t":"set","p":"payload","pt":"msg","to":"Etwas wurde erledigt","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":470,"y":460,"wires":[["237a18ca.addcf8"]]},{"id":"3f2d8464.acdbdc","type":"change","z":"27a1dcd5.ab7ad4","name":"readSession","rules":[{"t":"set","p":"payload","pt":"msg","to":"session","tot":"global"}],"action":"","property":"","from":"","to":"","reg":false,"x":790,"y":580,"wires":[["e350059e.44f328"]]},{"id":"5eca4b1.2c03ab4","type":"delay","z":"27a1dcd5.ab7ad4","name":"wait 1''","pauseType":"delay","timeout":"1","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":610,"y":580,"wires":[["3f2d8464.acdbdc"]]},{"id":"bce45bd4.746cd8","type":"change","z":"27a1dcd5.ab7ad4","name":"save siteId","rules":[{"t":"set","p":"mySiteId","pt":"global","to":"payload.siteId","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":310,"y":520,"wires":[["bd10ced8.e046"]]},{"id":"5ffd1b8a.cd8534","type":"change","z":"27a1dcd5.ab7ad4","name":"prepare Answer","rules":[{"t":"set","p":"payload","pt":"msg","to":"{\t \"text\":msg.payload,\t \"lang\":\"de\",\t \"id\":\"\",\t \"siteId\":msg.siteId,\t \"sessionId\":\"\"\t \t}","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":800,"y":520,"wires":[["185bb0bc.bae1df","5eca4b1.2c03ab4"]]},{"id":"237a18ca.addcf8","type":"change","z":"27a1dcd5.ab7ad4","name":"read siteId","rules":[{"t":"set","p":"siteId","pt":"msg","to":"mySiteId","tot":"global"}],"action":"","property":"","from":"","to":"","reg":false,"x":630,"y":520,"wires":[["5ffd1b8a.cd8534"]]},{"id":"1472dd78.aa2e53","type":"mqtt in","z":"27a1dcd5.ab7ad4","name":"sessionEnded__","topic":"hermes/dialogueManager/sessionEnded","qos":"2","datatype":"json","broker":"6f747297.64151c","x":100,"y":640,"wires":[["2fbaa744.027eb8"]]},{"id":"2fbaa744.027eb8","type":"debug","z":"27a1dcd5.ab7ad4","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":290,"y":640,"wires":[]},{"id":"6f747297.64151c","type":"mqtt-broker","z":"","name":"MQTT","broker":"10.2.254.1","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]

Here’s something for nodeRed (version 3):

I was always not happy with the numbered array of the slots. So I changed them to own variables. Also, if now room is said ithe the intent, the actual sideId is taken. Both is in the function node in javascript:
mqtt_example_4
Heres the nodeRed nodes:
mqtt_example_3

[{"id":"9d9044a8.bb4108","type":"mqtt in","z":"27a1dcd5.ab7ad4","name":"FromRhasspy","topic":"hermes/intent/#","qos":"2","datatype":"json","broker":"6f747297.64151c","x":410,"y":680,"wires":[["be3c58cb.88e6a8"]]},{"id":"be3c58cb.88e6a8","type":"function","z":"27a1dcd5.ab7ad4","name":"createNewSlots","func":"var slots = {\"room\":msg.payload.siteId}\nfor (var i in msg.payload.slots){\n slots[msg.payload.slots[i].slotName] = msg.payload.slots[i].value.value\n}\nmsg.slots = slots\n \nreturn msg","outputs":1,"noerr":0,"x":600,"y":680,"wires":[["a7e5d908.ebc348"]]},{"id":"a7e5d908.ebc348","type":"debug","z":"27a1dcd5.ab7ad4","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":770,"y":680,"wires":[]},{"id":"6f747297.64151c","type":"mqtt-broker","z":"","name":"MQTT","broker":"10.2.254.1","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]