Trying matrix satellite killed my matrix voice

modernhistorian · June 10, 2022, 11:37pm

I’ve been trying to follow the instructions for setting up my matrix voice as a satellite. I got as far as using VS Code to try to install OTA updating, but it would not compile. Now I find that the matrix is dead when I try to use it with rhasspy. I suspect that the instuction to reset the matrix’s flash memory:

esptool.py --chip esp32 --port /dev/ttyS0 --baud 115200 --before default_reset --after hard_reset erase_flash

has killed its functioning. Please, how can I flash the ESP32 from the pi so it will function as before?

modernhistorian · June 11, 2022, 12:21am

After giving up on the OTA part of this setup, I returned to the git instructions and cloned the repository:

Once your environment is set up go into the PlatformIO directory
Copy and configure the settings.ini.example to settings.ini and configure the proper parameters. Any $ in the settings.ini will be threated as an environment variable, did not find a way to escape it yet, fixes or workarounds are welcome
Set the device_type to 1 in the settings.ini
For the first flash, set the deployhost to the IP address of the attached pi and set method to ‘matrix’
Build and flash the project with PlatformIO, for example run pio run --target upload from the commandline

I set the device type to 1 and the deployhost to the pi’s ip address. Then I click the checkmark to compile. Again, I get this error:

> Executing task in folder PlatformIO: d:\Documents\Making\PlatformIO\Projects\ESP32-Rhasspy-Satellite\PlatformIO\platformio.exe run <

The terminal process failed to launch: Path to shell executable "d:\Documents\Making\PlatformIO\Projects\ESP32-Rhasspy-Satellite\PlatformIO\platformio.exe" does not exist.

Terminal will be reused by tasks, press any key to close it.

Why can’t I compile this? What is wrong here?

modernhistorian · June 11, 2022, 1:32am

Here is a screenshot

. Can anyone tell me why this is happening?

romkabouter · June 11, 2022, 11:44am

First:
sudo voice_esp32_enable
Then
esptool.py --chip esp32 --port /dev/ttyS0 --baud 115200 --before default_reset --after hard_reset erase_flash

This will erase what was flashed

Your platformIO cannot be found, but I am sorry I do not know a solution for it. I do not use windows.

modernhistorian · June 11, 2022, 3:45pm

I gave up on VScode in windows and installed it, along with platformIO, on the rpi4 that the matrix is plugged into. I cloned your repository, opened it with code, copied settings example to settings.ini with changes, and hit the check mark. It began to build, but failed with errors. See attached screenshot. Can you see what caused this? Thanks again.

romkabouter · June 12, 2022, 6:53am

Hmm, that is the same error the builds on github give atm. It is outside my code, but I cannot reproduce it on my localmachine.

A declaration of TAG is missing in one of the used libraries, but it has to be fixed in there. I will check on those libraries, but if you want to compile you have to change the code. By the looks of the error, the source file for the library is WM9978.cpp

modernhistorian · June 12, 2022, 1:28pm

Thanks, I will look at those but I’m not much of a c++ programmer. The flash reset has my matrix voice working again.

romkabouter · June 15, 2022, 6:53pm

Please pull master from github, I have pinned espressif32 to version 4.2.0.
This will fix the compile error

modernhistorian · June 15, 2022, 8:48pm

Yes, Visual Code running on the pi now creates an esp32 image (in 505 seconds). I have not tried to flash the matrix voice yet, as I think I’d better study how to use it first. Looking back at the topic title, I want to make clear that whatever happened to make my matrix not work happened during the first steps, the ESP32 setup page on the matrix site. Thanks again for your help here.

romkabouter · June 15, 2022, 9:38pm

If you followed the guide you mentioned earlier, you have flashed https://github.com/matrix-io/esp32-platformio
This is NOT the esp32 satellite streamer.

Check this guide for that: https://github.com/Romkabouter/ESP32-Rhasspy-Satellite/blob/master/matrixvoice.md

modernhistorian · June 16, 2022, 1:56am

I’d not seen the first link; it’s the second one I’ve been following.

I compiled the ESP32 image and attempted upload to the matrix voice on the pi. I got a password denied failure, which is odd, then realized that deploy.sh assumes the username is pi. I changed that to “voice,” the correct name, and the image uploaded. I unplugged the matrix, powered it, and found the ip address in my router. The config page loaded on a browser, and subscribing to the audioframe topic in a terminal brought a stream of RIFF, WAVE fmt characters.

OK, so the satellite seems to be working, but now I’m struggling with the setting in Rhasspy. It will not reapond to the wake word, and when waken manually, does not hear commands spoken into the matrix voice. The site id is satellite.

I’ve attached png’s of the settings. Can you see anything wrong with them? Thanks.

romkabouter · June 16, 2022, 9:31am

The stream of wave’s is correct, is it indeed hermes/audioServer/satellite/audioFrame?
That is what it should be if you have set the name te salellite.

I assume you also have satellite set on the Id’s textbox in WakeWord?
Also, set Audio Playing to Hermes MQTT. Otherwise the output will never reach the satellite

Can you post your whole json profile? Is it under Settings->Advanced in Rhasspy

modernhistorian · June 16, 2022, 1:49pm

In a terminal, the command `mosquitto_sub -t hermes//audioServer/satellite/audioFrame’ does generate a stream of characters, which stop if I power down the matrix and start again with power. For now, I’m leaving audio playing to the pi with aplay (one problem at a time). Here is my json profile:

{
    "dialogue": {
        "satellite_site_ids": "satellite",
        "system": "rhasspy",
        "volume": "0.5"
    },
    "handle": {
        "satellite_site_ids": "satellite",
        "system": "hass"
    },
    "home_assistant": {
        "access_token": "notMyActualToken",
        "handle_type": "event",
        "url": "http://192.168.0.114:8123"
    },
    "intent": {
        "satellite_site_ids": "satellite",
        "system": "fsticuffs"
    },
    "microphone": {
        "pyaudio": {
            "device": "3"
        }
    },
    "mqtt": {
        "enabled": "true",
        "host": "192.168.0.114",
        "site_id": "base"
    },
    "sounds": {
        "aplay": {
            "device": "default:CARD=Headphones",
            "volume": "0.25"
        },
        "system": "aplay"
    },
    "speech_to_text": {
        "satellite_site_ids": "satellite",
        "system": "pocketsphinx"
    },
    "text_to_speech": {
        "satellite_site_ids": "satellite",
        "system": "nanotts"
    },
    "wake": {
        "porcupine": {
            "keyword_path": "jarvis_raspberry-pi.ppn"
        },
        "raven": {
            "keywords": {
                "Daedalus": {
                    "enabled": true
                }
            },
            "probability_threshold": "0.4"
        },
        "satellite_site_ids": "satellite",
        "system": "porcupine"
    }
}

Audio Recording is set to disabled.

romkabouter · June 16, 2022, 2:00pm

That is good, it means the satellite is working properly.
I cannot spot an error in your profile I must say. Maybe check the log if you see entries with satellite popping up

modernhistorian · June 16, 2022, 2:23pm

I cleared the log, woke rhasspy, and spoke into the matrix. Rhasspy timed out and gave this log entry:

[ERROR:2022-06-16 08:11:20,940] rhasspyserver_hermes: 
Traceback (most recent call last):
  File "/usr/lib/rhasspy/.venv/lib/python3.7/site-packages/quart/app.py", line 1821, in full_dispatch_request
    result = await self.dispatch_request(request_context)
  File "/usr/lib/rhasspy/.venv/lib/python3.7/site-packages/quart/app.py", line 1869, in dispatch_request
    return await handler(**request_.view_args)
  File "/usr/lib/rhasspy/rhasspy-server-hermes/rhasspyserver_hermes/__main__.py", line 943, in api_listen_for_command
    async for response in core.publish_wait(handle_intent(), [], message_types):
  File "/usr/lib/rhasspy/rhasspy-server-hermes/rhasspyserver_hermes/__init__.py", line 995, in publish_wait
    result_awaitable, timeout=timeout_seconds
  File "/usr/lib/python3.7/asyncio/tasks.py", line 423, in wait_for
    raise futures.TimeoutError()
concurrent.futures._base.TimeoutError
[DEBUG:2022-06-16 08:10:50,916] rhasspyserver_hermes: <- HotwordDetected(model_id='default', model_version='', model_type='personal', current_sensitivity=1.0, site_id='base', session_id=None, send_audio_captured=None, lang=None, custom_entities=None)
[DEBUG:2022-06-16 08:10:50,907] rhasspyserver_hermes: Waiting for intent (session_id=None)
[DEBUG:2022-06-16 08:10:50,906] rhasspyserver_hermes: Publishing 196 bytes(s) to hermes/hotword/default/detected
[DEBUG:2022-06-16 08:10:50,905] rhasspyserver_hermes: -> HotwordDetected(model_id='default', model_version='', model_type='personal', current_sensitivity=1.0, site_id='base', session_id=None, send_audio_captured=None, lang=None, custom_entities=None)

No reference to satellite at all. When I click the wake button, rhasspy publishes to hermes/hotword/default/detected, but captures no audio.

I am running rhasspy in docker with this run command:

 docker run -d -p 12101:12101 \
      --name rhasspy \
      --restart unless-stopped \
      -v "$HOME/.config/rhasspy/profiles:/profiles" \
      -v "/etc/localtime:/etc/localtime:ro" \
      --device /dev/snd:/dev/snd \
      rhasspy/rhasspy \
      --user-profiles /profiles \
      --profile en

Do I need a different device, or a port option for streaming from the matrix?

romkabouter · June 16, 2022, 5:54pm

The timeout is strange, but these errors do not have anything to do with the satellite. All what that does is send the audio, which it does. When you click the wake button, you wake the base (also shows in the logs)

Command for docker looks good to me, maybe you have a mike available so you can try it on the base?

modernhistorian · June 16, 2022, 7:33pm

I’m using the matrix on the base and it works fine there, even with the newly flashed image on the ESP32. Is there a program to run in the terminal that will decode the audio frames sent by the matrix satellite?

romkabouter · June 16, 2022, 8:17pm

Yes, there is actually.

In the voco branch I have created a python script

github.com

Romkabouter/ESP32-Rhasspy-Satellite/blob/voco/record.py

#!/usr/bin/python

import time
import sys
import os
import paho.mqtt.client as paho
import wave
import io
import pyaudio
broker = "192.168.43.79"

# folder = './tmp/'
# file_name = folder + 'wav_'

output = io.BytesIO()
p = pyaudio.PyAudio()
streamOut = p.open(format=pyaudio.paInt16,
                    channels=1,
                    rate=16000,
                    output=True)

This file has been truncated. show original

Change your broker and others tike the topic, and then run it.
It will run for 4 seconds and save a file test.wav. This will hold the audio recorded from the audiostream

modernhistorian · June 17, 2022, 2:07pm

The script produces a clear WAV file of my voice, so the problem is definitely not in the ESP32 programming. I will study again how rhasspy handles mqtt messages and, if I can’t find an answer, post a new topic. Thanks again for your in-depth help.

modernhistorian · June 17, 2022, 3:44pm

Jeez, all it took was to add a username and password to the MQTT settings in rhasspy. It doesn’t like these being blank, I guess. Headphone output (to powered speaker) works well also.

I like the green LEDs animation. Is there a way to have them blink red on the error beep from rhasspy?