Making rhasspy 2.5 venv run as a service on pi zero

Adibou · April 28, 2020, 9:30am

Hi everyone

I have the latest 2.5-pre on a docker NAS as a server and a pi zero with respeaker running as a venv (also latest 2.5 pre), both speaking through MQTT.

Everything works quite fine when I manually start rhasspy on the pi zero, hotword is recognised, intents are passed as hass events, etc.
BUT I can’t make it run as a service.

Here is what I did:

created /home/pi/.config/systemd/user directory
created a rhasspy.service file as follow:

[Unit]
Description=Rhasspy
After=syslog.target network.target
[Service]
Type=simple
WorkingDirectory=/home/pi/rhasspy-satellite
ExecStart=‘/home/pi/rhasspy-satellite/bin/rhasspy-satellite --profile fr’
RestartSec=1
Restart=on-failure
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=rhasspy
[Install]
WantedBy=multi-user.target

run the following commands
systemctl --user daemon-reload
systemctl --user start rhasspy
systemctl --user enable --now rhasspy

I have the following issues :

it doesnt start at boot. I tried to replace WantedBy=multi-user.target by WantedBy=default.target as found in google and that seems to do the job
I can get the web UI on the pi zero,
manual recognition of text works (so both server and satellite seems to communicate) but intent is not passed to hass (which works when manually started)
hotword detection works but the asr/nlu/toggleOff is broken, basically it starts listening and that’s it, the pi zero doesn’t get back to iddle state

Any help would be amazing, I feel so close !

Thanks a lot

rolyan_trauts · April 29, 2020, 8:52am

try after=dhcpcd as that is always really late in the boot order also I always get that wrong but that service spelt correct.
The wants=graphical.target is later and doesn’t have to occur so try that with dhcpd

prob need systemctl status rhasspy and journalctl -b of what it fails on

Prob a lot happening at start and get things to start later?

ExecStartPre=/bin/sleep 10
TimeoutStartSec=0 Or maybe set bigger than default?

Adibou · April 29, 2020, 3:27pm

Thanks for the suggestions @rolyan_trauts but still no luck.

journalctl gives me these erros, errors I dont have when manually running rhasspy-satellite/bin/rhasspy-satellite --profile fr

avril 29 17:22:00 respeaker rhasspy[483]: Starting services...
avril 29 17:22:09 respeaker rhasspy[483]: 2020-04-29 17:22:09,545 INFO supervisord started with pid 700
avril 29 17:22:10 respeaker rhasspy[483]: 2020-04-29 17:22:10,619 INFO spawnerr: unknown error making dispatchers for 'microphone': ENXIO
avril 29 17:22:10 respeaker rhasspy[483]: 2020-04-29 17:22:10,663 INFO spawnerr: unknown error making dispatchers for 'intent_handling': ENXIO
avril 29 17:22:10 respeaker rhasspy[483]: 2020-04-29 17:22:10,678 INFO spawnerr: unknown error making dispatchers for 'wake_word': ENXIO
avril 29 17:22:11 respeaker rhasspy[483]: 2020-04-29 17:22:11,761 INFO spawnerr: unknown error making dispatchers for 'microphone': ENXIO
avril 29 17:22:11 respeaker rhasspy[483]: 2020-04-29 17:22:11,833 INFO spawnerr: unknown error making dispatchers for 'intent_handling': ENXIO
avril 29 17:22:11 respeaker rhasspy[483]: 2020-04-29 17:22:11,880 INFO spawnerr: unknown error making dispatchers for 'wake_word': ENXIO
avril 29 17:22:13 respeaker rhasspy[483]: 2020-04-29 17:22:13,943 INFO spawnerr: unknown error making dispatchers for 'microphone': ENXIO
avril 29 17:22:14 respeaker rhasspy[483]: 2020-04-29 17:22:14,002 INFO spawnerr: unknown error making dispatchers for 'intent_handling': ENXIO
avril 29 17:22:14 respeaker rhasspy[483]: 2020-04-29 17:22:14,018 INFO spawnerr: unknown error making dispatchers for 'wake_word': ENXIO
avril 29 17:22:17 respeaker rhasspy[483]: 2020-04-29 17:22:17,087 INFO spawnerr: unknown error making dispatchers for 'microphone': ENXIO
avril 29 17:22:17 respeaker rhasspy[483]: 2020-04-29 17:22:17,099 INFO gave up: microphone entered FATAL state, too many start retries too quickly
avril 29 17:22:17 respeaker rhasspy[483]: 2020-04-29 17:22:17,135 INFO spawnerr: unknown error making dispatchers for 'intent_handling': ENXIO
avril 29 17:22:17 respeaker rhasspy[483]: 2020-04-29 17:22:17,159 INFO gave up: intent_handling entered FATAL state, too many start retries too quickly
avril 29 17:22:17 respeaker rhasspy[483]: 2020-04-29 17:22:17,203 INFO spawnerr: unknown error making dispatchers for 'wake_word': ENXIO
avril 29 17:22:17 respeaker rhasspy[483]: 2020-04-29 17:22:17,206 INFO gave up: wake_word entered FATAL state, too many start retries too quickly
avril 29 17:22:26 respeaker rhasspy[483]: [DEBUG:2020-04-29 17:22:26,237] rhasspyserver_hermes: Namespace(certfile=None, host='0.0.0.0', keyfile=None, local_mqtt_port=12183, log_fo
avril 29 17:22:28 respeaker rhasspy[483]: [DEBUG:2020-04-29 17:22:28,299] rhasspyprofile.profile: Loading /home/pi/rhasspy-satellite/rhasspy-profile/rhasspyprofile/profiles/fr/prof
avril 29 17:22:28 respeaker rhasspy[483]: [DEBUG:2020-04-29 17:22:28,315] rhasspyprofile.profile: Loading /home/pi/.config/rhasspy/profiles/fr/profile.json

Thanks for any help !

rolyan_trauts · April 29, 2020, 6:03pm

At a guess there is prob just far too much happening at one time for a Pi Zero is capable of.

Guess you will have to have a look at systemd-analyze -h and see what is happening, but I think there is so much load things are taking such a long time they are restarting, retaining load until they fail.
In the scripts to start you prob need to add some delays.

Its prob a pointless exercise though as it is a Pi Zero.

Adibou · April 29, 2020, 9:48pm

Am I the only one trying to have a rhasspy-zero satellite starting at boot ??

rolyan_trauts · April 29, 2020, 11:03pm

Think you might be the only one trying to run the full 2.5 venv on a pi zero.

But maybe others will comment have you done a Top or Htop and had a look at your load?
What is the load through boot and for the first couple of minutes?

Adibou · April 30, 2020, 12:07am

Thanks for your help @rolyan_trauts, unfortunately your last two messages are far beyond my pi knowledge

What I don’t understand is how it can fully work when manually launched and fail when it’s supposedly just asked to run a command at boot

rolyan_trauts · April 30, 2020, 3:00am

systemd-analyze -h was me telling you to have a look of the options for systemd-analyze which has some great tools.

systemd-analyze blame is one I often do as it lists by slowest services first and gives time to init.
systemd-analyze exit-status is another.

Logging into the pi with ssh or console and simple running top or htop can tell you much.

Prob because it doesn’t have services to time out and you are running the services in the correct order as everything else has definately finnished.

I am also getting to grips with Rhasspy but isn’t

Better for the zero.

ced_cox · April 30, 2020, 7:32am

hi,

I found a workaround solution few days ago but as it’s not clean, i did not share it, thinking i’ll have time to analyze what. But not
So here is quick and dirty solution.

nano /home/pi/rhasspy-satellite/bin/rhasspy-voltron

replace line 120 :

# trap finish EXIT

# Run assistant
log_path="${profile_dir}/supervisord.log"

by

# trap finish EXIT

sed -i 's/\/dev\/stdout/\/home\/pi\/rhasspy.log/g' "${conf_path}"

# Run assistant
log_path="${profile_dir}/supervisord.log"

When started as service, supervisord seems to not be able access /dev/std/out

Have a good day

Adibou · April 30, 2020, 11:16am

Amazing ! It works thanks a lot.

Just for future references if needed, the file to edit on the rhasspy-satellite package is this one

nano rhasspy-satellite/bin/rhasspy-satellite

Thanks to both of you

daniele_athome · May 10, 2020, 8:01pm

I’m afraid the trick doesn’t survive restarts

As I understand it, systemd takes control of standard output and error because it redirects them to the journal. However supervisord uses those to control the processes AFAICU. Anyway this needs further study. Maybe I’ll open an issue on GitHub.

ulno · May 19, 2020, 7:21pm

Describing same problem in 2.5, venv, run with systemd impossible on raspberry pi 3, I assume this makes running rhasspy 2.5 impossible in venv on any systemd linux system (basically all of them).

Can’t find the github issue - is there any new development here?

ulno · May 19, 2020, 7:36pm

I applied the suggested patch here in a copy of rhasspy-voltron start script (copied to rhasspy-voltron-systemd). It seems to survive restarts - I still see lots of messages in journalctl.

Does it make sense to play with just setting StandardOutput=null in the unit file?

ced_cox · May 20, 2020, 7:10am

yes, the trick supports restart but not support an update (the file rhasspy-voltron or rhasspy-satellite is replaced).

So, if you don’t update rhasspy, the trick works (even with restart). If you update rhasspy, you’ll have to add the sed.

I don’t think that an issue is opened. And i would like to understand why supervisord has a problem with “/dev/stdout” in his config file.

Cedric

koan · May 20, 2020, 8:10am

Can someone create an issue about this for the rhasspy-voltron project?

ced_cox · May 20, 2020, 8:48am

I did it : https://github.com/rhasspy/rhasspy-voltron/issues/42

I never open ticket before on git, so, i hope it’s ok

Ced

koan · May 20, 2020, 8:55am

Thanks, looks fine, we can follow up there!