My system:
A Master in docker on a i5 box
5 satellites (2 Pi 3B+, 2 Pi 4B, 1 Pi ZeroW) each with a google AIY voice hat.
It has been running since early January 2023.
All Intent handling in HTTP remote to the Master which sends intents to Home Assistant.
Most intents are handled by Node-Red.
I have a node that records the last command so it can be repeated.
Everything seems to have been working fine until this morning.
Around 7:30 AM I asked Rhasspy what time it was. It told me.
Around and hour later I asked Rhasspy another question. I got the wake word acknowledge sound and the “Recorded WAV” beep, but no reply.
No intents have been executed from any of the satellites since.
The Master has been restarted.
I tried deleting the Home Assistant Access Token and generating a new one with no change.
I get these errors on the Master no matter what command I give issue:
[ERROR:2023-07-04 11:15:59,296] rhasspyserver_hermes: MethodNotAllowed(405)
Traceback (most recent call last):
File "/usr/lib/rhasspy/.venv/lib/python3.7/site-packages/quart/app.py", line 1821, in full_dispatch_request
result = await self.dispatch_request(request_context)
File "/usr/lib/rhasspy/.venv/lib/python3.7/site-packages/quart/app.py", line 1863, in dispatch_request
raise request_.routing_exception
quart.exceptions.MethodNotAllowed: MethodNotAllowed(405)
Those are always tough to run down. The 405 is generally (not Rhasspy specific) error of a client trying to talk to a server api endpoint using a not-allowed method (i.e. client making a GET request to a POST only api endpoint). The kicker is it resolved itself after 5 hours but you are still seeing the error. I would check the logs of any outside systems that Rhasspy is issuing REST commands to. Rhasspy also has an extensive HTTP API with a number of endpoints (check here for them) that a system could call, but it looks like this is a call from Rhasspy out to something (could even be a call internally to its self). Lines before/after the error in the logs might provide more context on what is being called and who is doing the calling.
As for Rhasspy failing to run commands, if it happens again, here is what I would do:
Add a debug node and connect it to the HTTP / Websocket / MQTT In node that is responsible for handling request from Rhasspy. This will let you verify NodeRed is actually getting the request.
Add another debug node to any response nodes. For example I have a MQTT node listening to hermes/intent/# which is where Rhasspy post any recognized intents. Many times the flow ends in a MQTT out to hermes/dialogueManager/startSession in order to send back a TTS command to Rhasspy. Attaching a debug node to that out node confirms that NodeRed is processing the command and responding.
Once the debug nodes are in place, log into the main Rhasspy instance and attempt to issue a command from there.
If it works, then you issue is most likely with the satellite to main communication. This could be a network issue, but check your satellite logs.
If commands from your main instance to NodeRed fail then you have a Rhasspy-to-NodeRed issue. Again, could be a network issue but I have also seen NodeRed freeze up, crash, and then reboot. This is especially true if you are running the NodeRed as a Home Assistant add on with Watchdog enabled. Check your NodeRed logs for indications of a hung flow and/or restart. I will say 5 hours is a long time for a NodeRed crash and reboot though…
If satellite-to-main works and main-to-NodeRed works then you would have to dive deeper into the Home Assistant side of things. One test would be to create a simple intent in Rhasspy (i.e. “HealthCheck”) and then in NodeRed create a flow that responds to “HealthCheck” with a simple TTS response. This bypasses any processing from HA and confirms that not only NodeRed works, but components such as your TTS engine (Rhasspy, MaryTTS, Larynx, etc) also work. If that simple “HealthCheck” fails then that is where NodeRed debug nodes and logs are going to be your best bet.
Intents are all sent to Home Assistant via HTTP /api/intent/handle built into Rhasspy and all are processed in node-red where response messages are routed to node-red functions via the intent.name in each message. Responses from intents are sent back the the proper Rhasspy satellite via Rhasspy’s api/text-to-speech.
I have some announcements that start from Node-Red and a few from Home Assistant automations, but those would not have been active.
I did and do have debugs on the node-red inputs and outputs. I never saw any errors. It looked like the intent never got to Home Assistant.
I was looking at the logs on the satellites and the master.
It appeared that satellites were talking to the master because I saw the intent recognition, which is done on the master.
My guess from the errors was there was a problem with the intent api into Home Assistant, but it guess it could have been in the TTS. Someone who knows the code could tell.
I don’t think any commands from the Rhasspy (blue button) Recognize function were recognized.
There has been a node-red update since then. I no longer see the errors in the Rhasspy log, but I can’t say for sure when they went away.
I didn’t have to wait too long to collect more data.
It happened again in mid July. I collected all of the data you suggested and began to analyze the logs.
I had to leave town for a few weeks and left it alone.
When I returned the errors were unchanged.
I deleted and created a new long-lived access token.
I did other system upgrades.
it started working again.
Then, last night, it stopped working. Same errors.
I tried changing the Long-lived token again.
I checked a few hours later and it still had the errors.
Then I restarted node-red and everything started working!
I still don’t know what’s going on, but I’m sure I’ll get to see it again.