Rebranded the Matrix Voice to esp32-rhasspy-satellite

Show Us
21

@romkabouter
Thank you so much for the rewrite. I have successfully compiled the code and flashed it on my M5 Echo. I can trigger recording via the button and Rhasspy successfully recognizes and handles the intent :slight_smile:
Somehow I cannot make the remote hotword detection work (I understood that local hotwork detection was removed) but remote should work, right?

In the Rhasspy log I have:
1611951144: New client connected from 192.168.x.x as satellite_kitchenAudio (c1, k15, u’pip’).,
1611951144: New client connected from 192.168.x.x as satellite_kitchen (c1, k15, u’pip’).

I made sure the the M5 is set to remote hotword via the webinterface

Rhasspy itself:
AudioRecording: Hermes MQTT
WakeWord: Porcupine - Satellite id “satellite_kitchen” is listed (I have also tried to add satellite_kitchenAudio as well)

I have a second satellite set up using the Android App which works fine (Hotword via UDP Audio).

Do you have any idea why it does not work with the M5 Echo?

/edit
After reboot hotword detection does not work at all for my setup anymore, hmm

22

Ok, does recognition etc still work?
Because then the AudioSettings are correct for the M5.
Only satellite_kitchen is needed by the way.

I believe satellite_kitchen should be on all settings as well

23

yes, with button everything works fine with the M5 (leds, speech to text, recognition,…) . I have disabled Audio Recording, now Hotword works again for Android satellite, but for M5 only button works. Both satellites are listed in all active Rhasspy server settings (all but audio recording). Hmm, I will experiment a little bit… maybe sth related with UPD streaming from other satellite :slight_smile:

for hotword detection only hotword would be needed, right? Audio Recording on Server is not needed?

24

Well, the software publishes audio to Hermes MQTT, so Audio Recording on your server should be set to Hermes MQTT.
Can you post some screenshots from your settings?

I think it has to do with the UDP settings somehow.

25

okay, that’s what I thought.
Here is my profile.json
{ "command": { "webrtcvad": { "before_sec": "0.5", "max_sec": "7", "min_sec": "1", "silence_sec": "0.5" } }, "dialogue": { "satellite_site_ids": "satellite_cell,satellite_kitchen", "system": "rhasspy" }, "handle": { "satellite_site_ids": "satellite_cell,satellite_kitchen", }, "intent": { "satellite_site_ids": "satellite_cell,satellite_kitchen", "system": "fsticuffs" }, "microphone": { "system": "hermes" }, "mqtt": { "site_id": "Central" }, "sounds": { "command": { "play_arguments": "", "play_program": "pulse_tts.sh" }, "recorded": "", "system": "command" }, "speech_to_text": { "satellite_site_ids": "satellite_cell,satellite_kitchen", "system": "kaldi" }, "text_to_speech": { "nanotts": { "language": "en-US" }, "satellite_site_ids": "satellite_cell,satellite_kitchen", "system": "nanotts", "wavenet": { "sample_rate": "44100" } }, "wake": { "porcupine": { "keyword_path": "computer_linux.ppn", "sensitivity": "1.0", "udp_audio": "172.17.0.2:20000:satellite_cell" }, "satellite_site_ids": "satellite_cell,satellite_kitchen", "system": "porcupine" } }

satellite_cell works as intended and M5 (satellite_kitchen) only via button

/edit
Hmm, maybe it works afterall but only very bad (maybe due to noise or sth). I just managed to trigger via the hotword on M5 once.

26

Did you try with version 7.1 are earlier?
I suggest using 22050 as google wavenet by the way. Higher than that will probably cause static sound (hissing) on the M5

Also on wakeword I see udp_audio set, might also be an issue. I do not know, never tried.

27

yes, version 7.1. I just managed to use the wakeword on the M5. It seems to work but quite bad. I had to put the M5 in a box to kind of isolate it and then after a few tries it picked up the wakeword. Only wakeword is that bad, the comand itself is picked up without any problems (when using the button).
Anyways, in general everything seems to work. I will try to find out whats interfering.
Thanks for your help and your great work on the code, I really appreciate it.

28

Maybe try some different wakeword and/or systems.
Since the commands are picked up ok that might be the issue.

The device does nothing more than stream audio.

29

Hi,

i have some questions for the ip addresses in the settings.ini

[General]
hostname=192.168.0.140 (is this is the ip which the m5echo should get ???)
deployhost=192.168.0.54 (is this the ip from the rhasspy server?)

30

hostname: the IP of the device you want to use for OTA.
So, in case of the m5echo, use that IP if you want OTA. The ip is not forced, it might get a different ip on your network. Change it to the ip the device gets.

deployhost: only used for matrix voice. In that case, it is the IP of the Raspberry Pi the Matrix Voice is attached to.

31

i never used plattform.io before so i dont understand how to build and upload the whole project to the m5echo.

i installed plattform.io and open your project. when i then click on build i get this error message.

Executing task in folder PlatformIO: C:\Users\sebastian.platformio\penv\Scripts\pio.exe run --target upload <

Processing esp32dev (platform: espressif32; board: esp32dev; framework: arduino)

Verbose mode can be enabled via -v, --verbose option

warning: Ignoring attempt to set reserved variable `$TARGETS’
File “C:\Users\sebastian\Downloads\ESP32-Rhasspy-Satellite-master\PlatformIO\load_settings.py”, line 36, in
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/esp32dev.html
PLATFORM: Espressif 32 (3.0.0) > Espressif ESP32 Dev Module
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (esp-prog) External (esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES:

  • framework-arduinoespressif32 3.10004.210126 (1.0.4)
  • tool-esptoolpy 1.30000.201119 (3.0.0)
  • tool-mkspiffs 2.230.0 (2.30)
  • toolchain-xtensa32 2.50200.80 (5.2.0)
    LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
    LDF Modes: Finder ~ chain, Compatibility ~ soft
    Library Manager: Installing git+https://github.com/matrix-io/matrixio_hal_esp32.git
    FileNotFoundError: [WinError 2] Das System kann die angegebene Datei nicht finden:
    File “C:\Users\sebastian.platformio\penv\lib\site-packages\platformio\builder\main.py”, line 177:
    env.SConscript("$BUILD_SCRIPT")
    File “C:\Users\sebastian.platformio\packages\tool-scons\scons-local-4.1.0\SCons\Script\SConscript.py”, line 591:
    return _SConscript(self.fs, *files, **subst_kw)
    File “C:\Users\sebastian.platformio\packages\tool-scons\scons-local-4.1.0\SCons\Script\SConscript.py”, line 280:
    exec(compile(scriptdata, scriptname, ‘exec’), call_stack[-1].globals)
    File “C:\Users\sebastian.platformio\platforms\espressif32\builder\main.py”, line 225:
    target_elf = env.BuildProgram()
    File “C:\Users\sebastian.platformio\packages\tool-scons\scons-local-4.1.0\SCons\Util.py”, line 658:
    return self.method(*nargs, **kwargs)
    File “C:\Users\sebastian.platformio\penv\lib\site-packages\platformio\builder\tools\platformio.py”, line 62:
    env.ProcessProjectDeps()
    File “C:\Users\sebastian.platformio\packages\tool-scons\scons-local-4.1.0\SCons\Util.py”, line 658:
    return self.method(*nargs, **kwargs)
    File “C:\Users\sebastian.platformio\penv\lib\site-packages\platformio\builder\tools\platformio.py”, line 140:
    project_lib_builder = env.ConfigureProjectLibBuilder()
    File “C:\Users\sebastian.platformio\packages\tool-scons\scons-local-4.1.0\SCons\Util.py”, line 658:
    return self.method(*nargs, **kwargs)
    File “C:\Users\sebastian.platformio\penv\lib\site-packages\platformio\builder\tools\piolib.py”, line 1062:
    project.install_dependencies()
    File “C:\Users\sebastian.platformio\penv\lib\site-packages\platformio\builder\tools\piolib.py”, line 898:
    lm.install(spec)
    File “c:\users\sebastian.platformio\penv\lib\site-packages\platformio\package\manager_install.py”, line 49:
    spec, silent=silent, skip_dependencies=skip_dependencies, force=force
    File “c:\users\sebastian.platformio\penv\lib\site-packages\platformio\package\manager\library.py”, line 86:
    force=force,
    File “c:\users\sebastian.platformio\penv\lib\site-packages\platformio\package\manager_install.py”, line 97:
    pkg = self.install_from_url(spec.url, spec, silent=silent)
    File “c:\users\sebastian.platformio\penv\lib\site-packages\platformio\package\manager_install.py”, line 141:
    vcs = VCSClientFactory.new(tmp_dir, url)
    File “c:\users\sebastian.platformio\penv\lib\site-packages\platformio\package\vcsclient.py”, line 55:
    src_dir, remote_url, tag, silent
    File “c:\users\sebastian.platformio\penv\lib\site-packages\platformio\package\vcsclient.py”, line 137:
    self.configure()
    File “c:\users\sebastian.platformio\penv\lib\site-packages\platformio\package\vcsclient.py”, line 146:
    result = proc.exec_command([cls.command, “–exec-path”])
    File “c:\users\sebastian.platformio\penv\lib\site-packages\platformio\proc.py”, line 117:
    p = subprocess.Popen(*args, **kwargs)
    File “C:\Users\sebastian.platformio\python3\lib\subprocess.py”, line 800:
    restore_signals, start_new_session)
    File “C:\Users\sebastian.platformio\python3\lib\subprocess.py”, line 1207:
    startupinfo)
    ================================================================================================== [FAILED] Took 1.51 seconds ==================================================================================================
    The terminal process “C:\Users\sebastian.platformio\penv\Scripts\pio.exe ‘run’, ‘–target’, ‘upload’” terminated with exit code: 1.
32

Sorry, I do not use windows. Looks like a configuration error to me

33

Looks like it can not find git.

34

The solution is to reinstall Platform.io extension.
Something under Windows is weird with path to git.

Described in this forum
https://community.platformio.org/t/git-library-download/17000/10

1 Like
35

I now build and upload your code to a m5echo. plattform.io shows several errors or warnings while uploading but at the end it says “success”.
Is there an easy way for testing m5echo that it is working correctly?

I can access the webserver on the m5.
I have an external mqtt server, configured in the m5 and rhasspy server.
I have wakeword detection working with the android app.
I get no response from the m5.
When it is connected to my WiFi, the LED is blue.
When i push the button, nothing changes.
The rhasspy log shows nothing when pushing the m5 Button or speaking hot word.

what is the usual way using the m5? wake word in m5 settings is on remote. when it is connected to WiFi what should be the usual behavior? should i see something in the rhasspy log when pushing the button? do i have to speak something to get a log entry? can i look somewhere at the external mqtt server for some action? is there an topic to subscribe to get some information? at the moment i dont know where to start debugging…

36

Yes, check the broker on the hermes/audioServer/<siteID>/audioFrame topic.
Where siteID is what you called it.

If you can acces the webserver, the flashing worked :slight_smile:
Also, if you have it connected to the pc, open up the terminal and press the reset button.
Several messages should be printed there, like which state is being entered.

1 Like
37

That is the usual way, local hotword is not working.
What are your settings in Rhasspy?

38

Start by checking the terminal and check the hermes/audioServer/<siteID>/audioFrame topic.
That way, you can verify that audio stream is working or not.

39

Oh, you fast in responding :wink:
5 minutes ago i got it working. The problem was the wifi connection over my repeater downstairs. I dont know why i can arrive the website on the m5 but the udp stream was not working. When it connects over the access point in my Living room everything works fantastic, it is realy great, very fast response.

When i use the same siteId for the rhasspy app and the m5, i can use the app for looking at the intent recognition. This is great for testing.

This is my profile in rhasspy, maybe it helps someone…

{
“dialogue”: {
“satellite_site_ids”: “default”,
“system”: “rhasspy”
},
“intent”: {
“satellite_site_ids”: “default”,
“system”: “fsticuffs”
},
“microphone”: {
“system”: “hermes”
},
“mqtt”: {
“enabled”: “true”,
“host”: “192.168.0.30”,
“password”: “your password”,
“username”: “your username”
},
“sounds”: {
“system”: “hermes”
},
“speech_to_text”: {
“satellite_site_ids”: “default”,
“system”: “pocketsphinx”
},
“text_to_speech”: {
“satellite_site_ids”: “default”,
“system”: “espeak”
},
“wake”: {
“precise”: {
“sensitivity”: “0.9”,
“udp_audio”: “172.17.0.4:20000:default”
},
“satellite_site_ids”: “default”,
“system”: “precise”
}
}

40

Ok great.

If you want Rhasspy to act as a server and the M5 as the one and only satellite, you can set the siteid of Rhasspy and the M5 the same

If you need more satellites, Satellite SiteID’;s should be a comma separated list of siteID’s in all Rhasspy server fields.
Rhasspy siteID should be called different, for which I suggest “server” or something like that.

1 Like