Fully support the Hermes or Hermod protocol

[fastjack]

This will allow for a more modular approach and ease the separation of concern between Rhasspy components.

This should also ease the base/satellite configuration.

The communication layer can be either a builtin websocket or HTTP2 endpoint or a external MQTT server.

The Hermes/Hermod protocol has some flaws but it should be a good starting point for a complete Rhasspy communication protocol.

Plus it will allow all Snipsters to move over to Rhasspy easily and for interaction with projects like Alice.

What do you think?

[romkabouter]

I agree, there is a lot more to the protocol then used now

[thinker]

I agree, too. I like to see MQTT (Hermes/Hermod) for the complete internal and external (Skills) communication.

[koan]

I agree. I have started documenting the current state of Rhasspy’s Hermes implementation on the reference page. This collects information that was already there but scattered around Rhasspy’s documentation. So now you can easily see what has been implemented already and what not yet

[romkabouter]

Great work, I have made a small comment on the playBytes topic.

[synesthesiam]

Here’s my plan for this so far:

I’ve been adding messages to the rhasspy-hermes library. This is just a set of classes that, when JSON-ified, will match the Hermes messages. We can extend Hermes by adding extra fields to these classes (or new classes!).

A library like rhasspy-nlu will build on rhasspy-hermes and become a service like rhasspy-nlu-hermes. These Hermes services should be standalone MQTT services with a command-line interface, and can be installed via source, Debian package, and Docker image.

Here’s the state of things so far:

• NLU
  • rhasspy-nlu-hermes built on rhasspy-nlu
• ASR
  • rhasspy-asr-kaldi-hermes built on rhasspy-asr-kaldi and rhasspy-silence

TODO:

• ASR
  • rhasspy-asr-pocketsphinx and rhasspy-asr-pocketsphinx-hermes
• NLU
  • Need to handle entity injection
• Wake
  • rhasspy-wake-*
• TTS
  • rhasspy-tts-*
• Audio Server
  • Service for MQTT -> speakers
  • Service for microphone -> MQTT (like @koan’s)
• Dialogue Manager
  • rhasspy-dialogue-hermes service that will manage sessions and coordinate other components

I’m not sure if the Rhasspy web interface should be combined with the Dialogue Manager, or if the two should be separate services.

Thoughts?

[koan]

I’m not sure if the Rhasspy web interface should be combined with the Dialogue Manager, or if the two should be separate services.

I would definitely separate the web interface, which is an interface, and the dialogue manager, which is a core behavioural component.

[romkabouter]

I think separation is better. I see the webinterface more as a settings manager (+some extras)
The dialogue manager is, like koan said, a core component.

[fastjack]

I agree with @koan and @romkabouter
Separation of concern is instrumental for future improvement and maintenance.

[thinker]

That’s the way it should be. With the planned steps of @synesthesiam we come very close to a Snips replacement. But Snips always lacked a management web console like Rhasspy has one . If the individual components run as separate services, communicate with each other via the MQTT protocol and there is a seperate management web console it will be very pleasant to work with the “new” Rhasspy.

[synesthesiam]

OK, I have a basic set of Hermes compatible services:

• ASR
  • rhasspy-asr-pocketsphinx-hermes (Docker image)
  • Uses acoustic_model, dictionary.txt, and language_model.txt from Rhasspy profile
• NLU
  • rhasspy-nlu-hermes (Docker image)
  • Uses intent.json from Rhasspy profile
• Dialogue Manager
  • rhasspy-dialogue-hermes (Docker image)
  • Handles sessions and listens for wake word detected event
• TTS
  • rhasspy-tts-cli-hermes (Docker image)
  • Calls external program for text to speech (optionally plays audio too)
• Microphone
  • rhasspy-microphone-cli-hermes (Docker image)
  • Streams raw audio from external program (e.g., arecord)
• Hotword
  • rhasspy-wake-porcupine-hermes (Docker image)
  • Listens for MQTT audio and does wake word detection

All of these services take --host, --port, and --siteId parameters to configure MQTT. I’d also recommend passing --debug to see what’s going on.

Next Steps

Going forward, I need help to figure out how we should deploy these services and how to incorporate them into the main Rhasspy interface.

As it stands, I’ve been able to get each of the services above to build with PyInstaller, so they can be deployed as Debian packages, Docker images, or source code (virtual environment). I still need to add builds for ARM, of course.

Questions

• Should Rhasspy become just a call out to supervisord or Docker compose? We can always bundle mosquitto if they don’t want to install an MQTT broker.
• How do we handle profile files and training? Is this just a separate service? What if different services run on different machines, but need access to specific profile files?
• What can we do to make it easier for people to understand and submit pull requests for individual services (or add new ones)?

[koan]

What can we do to make it easier for people to understand and submit pull requests for individual services (or add new ones)?

Rhasspy’s main documentation and main repository could have links to all repositories of the services. General issues can still be opened on the main repository.

[koan]

Should Rhasspy become just a call out to supervisord or Docker compose?

If all services run in their own container and the Rhasspy interface too, starting Rhasspy can come down to running Docker Compose indeed. We can even publish Docker Compose files for satellites, for specific combinations of services, and so on.

[fastjack]

This is awesome!

The rhasspy-hotword service should not listen to MQTT streamed audio frames as in a base/satellite configuration this will spam the MQTT broker all the time.

I think there should be a rhasspy-satellite service that handle the audio input AND output and do the wake word detection. This is pretty much what Amazon, Google and Snips are doing and is the single requirement for a satellite to work.

Only when the wake word is detected this service will notify the MQTT broker and start streaming audio frames.

This service should also output audio (feedback sounds, tts generated wav files, etc) from MQTT when asked.

The base services like ASR and NLU will have to use the same profile files as they are pretty closely related.

The rest of the services are not related to each other as long as they use the Hermes protocol so they can be easily separated.

Installing a base or satellite via a docker compose is perfect.

How can we choose which ASR, NLU and TTS we want for the base then? Should the base include multiple systems like today (kaldi, pocketsphinx, deepspeech, etc)? If they are separated (which sounds better) maybe there can be a simple docker compose generator script to select these?

What do you think?

[romkabouter]

The rhasspy-hotword service should not listen to MQTT streamed audio frames as in a base/satellite configuration this will spam the MQTT broker all the time.

I think this is not correct if we see the hermes services as snips replacement.
The mqtt hotword service should listen to the streamed audio, because that is what it’s for.

The audio stream itselfs comes from the microphone service, which is fed via the internal mic or other sources. In snips, all of this is used with an internal broker so we should definitely deploy mosquitto with rhasspy if we want this.
The audio service indeed spams the broker, but this does only affect network if an external broker is used or audio coming from other sources (like my streamer)

The sattelite service which snips used, is basically a hotword and audioserver. Some code is in there that when a hotword is detected on the interal mqtt audiostream, the service starts relaying the stream to the base broker.

For Snips, the main component is the DialogueManager. This service is the one putting the different services to work via mqtt messages.
A nice picture about the interacttions can be found here: https://snips.gitbook.io/tutorials/t/technical-guides/listening-to-intents-over-mqtt-using-python

This is a little outdated probably, but in it’s basis still correct

[fastjack]

I haven’t though of an internal MQTT broker between the wake word and the audio server…

That’ll work but might consume additional resources (CPU and memory) to constantly pack, unpack and push/pull the audio frames to/from this internal broker and increase installation complexity… maybe this can be simpler if these services are in the same process and use the extracted audio frames from record input, pass them to the wake word engine and when detected push them to the base MQTT broker… Although I really like the idea of having separated input handler and output handler…

Both have pros and cons I guess…

As I think that modularity should be one of the main objectives of Rhasspy then I’ll rally the internal MQTT broker proposition. The resources part can be addressed later if required (premature optimization…).

So we’ll have a base with:

• the main MQTT broker
• ASR of choice
• NLU of choice
• TTS of choice
• DialogueManager/MainManager

And a satellite with:

• an internal MQTT broker
• Input of choice
• Output of choice
• Wakeword of choice
• SatelliteManager (can allow to handle satellite registering and bridge both MQTT brokers to avoid maintaining many connections)

And somewhere:

• the web UI that connects to the main manager via HTTP rest API?

Does this look correct?

It seems Rhasspy is heading the MQTT road which is pretty cool .

[koan]

This sounds like a pretty good overview of the architecture, @fastjack and @romkabouter!

[KiboOst]

I will let the vocal specialists on this, but would just be sure that wakeword detection is on the satellite. No network flooding when no wakeword is detected please. Maybe a ping here and there so the base know the satellite is still up or updating some settings for example but no continual audio streaming between the two.

What I understand from snip is that the satellite listen, detect the wakeword, and only then stream the audio to base so the base can do asr then nlu then intent recognition

Also the satellite may have a minimal api http to handle minimal interface to have http access so we can see how it goes, restart it (even reboot the pi) and send TTS speech to it. Or we send TTS speech to the base with siteid and the base forward it to the right satellite. Like this anything can send speech command to any device via http without having mqtt itself.

And related to this : https://github.com/synesthesiam/rhasspy/issues/102

We would need /api/devices, even only on the base, to get all the rhasspy devices via http: all base and satellites with siteid, ip, port, so we can just do an http request and know the setup and all base/satellites to later know which siteid to send speech commands and such (or send to the base with the right siteid).

I also think siteid should be in settings apart from mqtt settings. Actually I don’t have mqtt, don’t need it to get notify in jeedom plugin for intent recognition and send speech commad. This would allow, even if base/satellites talks via mqtt, to control all this with another external device without mqtt. This would allow powerfull lightweight control for entire setup, and allow any smarthome solution to handle entire rhasspy setup with all its satellites without having mqtt itself.

[koan]

No network flooding when no wakeword is detected please.

That’s what the ‘internal’ MQTT broker @romkabouter and @fastjack are talking about is for: the hotword detector and audio service on the satellite device run on the same device together with a MQTT broker and satellite manager, and these four services each run in a separate Docker container and are connected to an internal (virtual) network. So no network flooding is happening, because the ‘network’ is contained on the device.

At the same time, these containers are also connected to the ‘real’ network, so when the hotword detector detects a hotword, a rhasspy-satellite service starts relaying the audio messages received on the internal MQTT broker to the external MQTT broker on your home network, and your Rhasspy server starts receiving audio for command handling. When the text is captured, the satellite manager stops relaying the audio to the external MQTT broker.

[synesthesiam]

This sounds great, @fastjack, @romkabouter, and @koan!

For the non-Docker case, we could have the hotword service listen to local (internal) MQTT broker and publish to a remote one. We could even create a special topic for the internal broker that is meant for raw audio chunks instead of tiny WAV files to avoid overhead. Another option is to accept raw audio directly from an external program or over UDP with gstreamer.

It seems like the web UI could be split into the part that lets you test and train, and the configuration portion. I’m not sure what configuration becomes when Rhasspy is split into services. Does the web server generate a Docker compose file or a supervisord.conf file?

[koan]

Does the web server generate a Docker compose file or a supervisord.conf file?

I hadn’t thought of that, awesome idea!

[romkabouter]

Maybe we can have a common shared space on the host that every docker service used to read/write configuration files.
HassIO does that as well, folders like config/share/ssl and such.
I use the proximanager for certicate auto-renewal and the files are placed in /share
My HassIO instance is used these same ssl files.
That way, you could have 1 UI controlling the configuration files, while the services use each their own specific config just like now.

As much as I like the MQTT support, I still believe that the excellent ways of using Rhasspy should be kept so that MQTT is always optional.

[fastjack]

The underlying communication layer can be either websocket events on a single endpoint or a MQTT broker. Both work pretty much the same way. They can be secured using tls and credentials so it should be fine.

As long as the websocket events are the same as the MQTT topic/message it should be easy.

The base DialogueManager/MainManager can provide a default websocket endpoint for all the other services (asr, nlu, tts, satellite) to subscribe (like MQTT).

The satellite manager can provide the same for local communication.

An additional MQTT bridge service can eventually also forward websocket events to an MQTT broker and relay MQTT message from this broker as websocket events for Hermes protocol over MQTT compatibility if required.

That way Rhasspy does not depend on another piece of software to work out of the box.

I also like MQTT a lot but it does not seem like a really required dependency.

[fastjack]

The web UI main objective should indeed be intents/slots management (creation, edition, etc), training and testing (unit-tests à la Snips would be awesome).

[banderson]

New user here…been a lurker for awhile. I like how this is proceeding! I am not a domain expert in voice systems, but have been interested in this technology for quite some time. I have a Matrix Voice and ReSpeaker gathering dust that I hope to dust off and use soon! Especially the Matrix Voice.

Some comments about the architecture…

It seems that things are moving in the direction of a “distributed” system. That is, one (or more?) “base” stations whose responsibility is to consume audio from a satellite, perform ASR, NLU, TTS (sending the audio back to a satellite for output) and possibly dialog management whilst the satellite responsibility is to produce audio (for input to ASR) and consume audio (presumably from TTS). From an architectural point of view, It seems important to keep in mind that the satellite might (in the future) be a (standalone) device with limited resources (such as a Matrix Voice).

With this in mind, whilst it might be the path of least resistance for an initial implementation to implement a satellite node using an “internal” MQTT broker this certainly seems like overkill in the long run for a “local” service to manage the satellite. As long as the internal MQTT broker is an implementation detail of the satellite and does not architecturally “leak” outside the boundaries of the satellite, I don’t see a problem. We should keep in mind the possibility of implementing a satellite on a standalone device with more limited resources such as the Matrix Voice ESP32. As such, as long as the satellite implements the correct “interfaces” that the base station requires, then we keep the option open to reimplement the satellite functionality in a more compact and resource efficient manner.

It occurs to me that exposing wake word, led control, etc. outside of the satellite is really unnecessary and inappropriate as this is arguably an implementation detail of a “particular type of satellite” and need not be globally exposed. One might envision other types of satellites that don’t operate the same way (with a wake word) such as activation via face recognition, a button, or presence detection. I’m not saying that these necessarily make sense right now, but the architecture should not preclude other types of satellites. Thus, there should be no need to expose these details outside of the satellite…its just a producer of recorded audio and a consumer of audio for playback. One might envision a sort of “plugin” architecture for the satellite that lets one plug in various “local” functionality in order to operate some arbitrary satellite device. Home Assistants plugin and event architecture come to mind as one possible architectural example. A purpose built software for (e.g.) a Matrix Voice is another example of how one might implement a satellite.

There was also a question about how to “configure” all of the various “nodes” in the system. Why not via the system wide “global” MQTT broker? The broker is easily accessable to all nodes in the system (base and satellite) as well as providing a simple form of persistence for the configuration. That means that each satellite (and base station) only needs a small “bootstrap” configuration to operate, i.e. its assigned ID (node ID/site ID?) and the URL and credentials necessary to access the global MQTT broker. Using the MQTT broker would also allow the web interface to easily author and view the configuration for each satellite (and base station) and publish it the an appropriate MQTT topic. If the satellite is listening to that topic, it can easily reload the config dynamically. Same for the web interface, all it needs is the URL and credentials to access the global MQTT broker to configure satellites and the base station.

I know that some of the above assumes a global MQTT broker is part of the core architecture. A pub/sub broker such as this is pretty useful in a distributed system, and MQTT is a proven service. Maybe there are ways of doing the same thing using HTTP and/or websockets, but I’m not familiar with how these might be used in the same manner. This is not to say that some HTTP or websocket APIs might also be useful, but it does seem as if MQTT is a reasonable choice to base the architecture on.

My thoughts. Keep up the good work! This is pretty cool!

ba

[synesthesiam]

Thanks for the feedback!

As long as the internal MQTT broker is an implementation detail of the satellite and does not architecturally “leak” outside the boundaries of the satellite, I don’t see a problem.

I agree that the details of how the satellite is implemented need not be a concern of the base station(s), but it may make some sense when those services are mixed and matched on different devices. If microphone audio is recorded on a satellite, but wake word detection is done elsewhere, the how the audio gets to the wake word detector matters.

My instinct is to have services support MQTT by default, but have additional options for services that deal with streamed audio (e.g., raw udp).

It occurs to me that exposing wake word, led control, etc. outside of the satellite is really unnecessary and inappropriate…

Yes, except for the case where you need to do something different with an intent depending on which wakeword/face/etc activated the assistant. Luckily, Hermes seems to have a customData field that could be used for this purpose!

I know that some of the above assumes a global MQTT broker is part of the core architecture.

Some kind of hub will be necessary. I think the Rhasspy web server could be used in a pinch, but we should probably default to a global broker to avoid multiple hops for messages between services. A lot of this is going to depend on the user’s exact configuration. Ideally, the same set of services could be used in the “all-in-one” scenario as well as a multi-base, multi-satellite setup.

[banderson]

I agree that the details of how the satellite is implemented need not be a concern of the base station(s), but it may make some sense when those services are mixed and matched on different devices. If microphone audio is recorded on a satellite, but wake word detection is done elsewhere, the how the audio gets to the wake word detector matters.

Ah, yes. I see. Sort of like an external “trigger” to tell the satellite to start listening for a voice command. That and some “events” issued by the satellite (via MQTT or whatever) to inform interested parties that the satellite was:

• triggered via an e.g. internal to the satellite wake word (with ID), external trigger (again with an ID? such as presence, face, button, HA event, etc)
• is listening for a voice command
• is recording a voice command,

Any thoughts about configuring base and satellites via MQTT?

[FredTheFrog]

After reading the entire thread, I only have one personal preference to express. I don’t have an MQTT broker installed or in use, and I don’t really care to set one up. If there’s going to be an ‘internal’ MQTT server for Rhasspy, please please please make it as transparent as possible, with as little configuration necessary as possible.

[synesthesiam]

Any thoughts about configuring base and satellites via MQTT?

The HTTP API seems better suited for configuration, since it may involve downloading files from a central Rhasspy hub. I could see MQTT being used to send URLs around, but probably not the data itself.

[synesthesiam]

If there’s going to be an ‘internal’ MQTT server for Rhasspy, please please please make it as transparent as possible, with as little configuration necessary as possible.

My plan thus far is to write the core Rhasspy services in a way that they can function as standalone Python modules or as MQTT services. I’ve also come across hbmqtt recently, which is an MQTT broker that can be installed via pip. If we absolutely need MQTT, I’ll probably use hbmqtt (internally configured).

[banderson]

The HTTP API seems better suited for configuration, since it may involve downloading files from a central Rhasspy hub. I could see MQTT being used to send URLs around, but probably not the data itself.

Hmmm. Let me try to explain my idea again. The web interface can use whatever it wants to communicate with the “hub” to “author” (create/edit) the configuration for a satellite. At some point, it “publishes/applies” this configuration after editing is complete. At this point, the hub “publishes” this new (json) configuration to the global MQTT broker. The satellite, having previously subscribed to its configuration topic gets a change notification at which point it (the satellite) can apply the new configuration. The only “file” necessary on the satellite is the bootstrap configuration necessary to communicate with the MQTT broker (URL, credentials) as well as its “assigned” ID. No other configuration file(s) necessary.

I have to say that I am a bit concerned about the proliferation of HTTP URLs in this architecture, especially for event notification (PUSH). This does not necessarily scale well. Websockets seem like a better solution (for both event delivery and command issuing). It was designed as a bi-directional, real-time communication channel between client (satellite) and server (hub). As a case study, consider HASS itself which migrated from HTTP to websockets for event delivery to clients and command issuing from clients. Also, AppDaemon is an interesting example for “hosting” services (apps) that consume HASS events and product HASS commands. In fact, I believe AppDaemon is architected in a way that it could be used for something other than HASS to host apps. Hmmm

My 2c/p.

[fastjack]

@banderson Loving the configuration distribution and we can throw in the satellite registration and even satellite/intent control.

I agree that a centralized broker (builtin or standalone, MQTT or websocket) is the way to go for a distributed system. As MQTT is a IoT standard nowadays I’m rooting for it

[KiboOst]

Reading the doc on mqtt published topic:

hermes/intent/<INTENT_NAME>

Rhasspy publishes a message to this topic on recognition of an intent.
The payload is a JSON object with the recognized intent, entities and text.

hermes/nlu/intentNotRecognized

Rhasspy publishes a message to this topic when it doesn't recognize an intent.

hermes/hotword/<WAKEWORD_ID>/detected

Rhasspy wakes up when a message is received on this topic.

Publishing to hermes/intent/<INTENT_NAME> seems strange. Does it seems mqtt should susbribe to one topic per intent_name ??
Why not publishing all intents on hermes/intent/intentRecognized topic with intent_name in the payload ? Would be a lot more efficient no ?
Same for wakeword, publish payload with wakeword id and site id on hermes/hotword/wakewordDetected ?

Also, didn’t find how to install and set mqtt on the pi. Checking mqtt in interface settings doesn’t work, I guess we have to manually install it.

[fastjack]

I’ve asked myself the same question… It would be easier indeed. I do not know if there is a specific reason for this topic per intent, wakeword specification.

I suspect this is due to the MQTT way of doing things (one topic per device/metric for instance)

Apart from Hermes compatibility with third party systems, this should possible to implement alongside the existing topics.

Probably using a topic like hermes/nlu/intentRecognized to avoid collision.

Maybe Rhasspy can send both?

What do you guys think?

[KiboOst]

Here are the topic the jeedom snips plugin subscribe to:

const TOP_INTENTS = 'hermes/intent/#';
const TOP_SESSION_STARTED = 'hermes/dialogueManager/sessionStarted';
const TOP_SESSION_ENDED = 'hermes/dialogueManager/sessionEnded';
const TOP_HOTWORD_DETECTED = 'hermes/hotword/default/detected';

const TOP_START_SESSION = 'hermes/dialogueManager/startSession';
const TOP_CONTINUE_SESSION = 'hermes/dialogueManager/continueSession';
const TOP_END_SESSION = 'hermes/dialogueManager/endSession';

So it subscribe to ‘hermes/intent/#’ then get the name like this : $payload->{‘intent’}->{‘intentName’}

This work for sure, but dunno the impact on performance of publishing to lot of different topics and subscribe to all of them with # (wildcard sub)

[koan]

This is just the way MQTT works, and I find it very well designed:

• If you’re interested in a specific intent, subscribe to the intent’s topic hermes/intent/<INTENT_NAME>.
• If you’re interested in all intents, subscribe to the intent wildcard topic hermes/intent/#.

Note: you may think you’re interested in all intents, but in many situations you’re not. Because of the distributed nature of MQTT, other programs can send intents too to the MQTT broker that your program is not aware of (and shouldn’t be). So if you implement something like hermes/intent/intentRecognized with an intent name in the payload, you’ll end up having to parse the payload to filter intents you’re not interested in. That seems much more cumbersome and error-prone than just subscribing to the intents you’re interested in and being sure in the rest of your code you only get events for these intents.

I don’t think we have to worry about performance now. Premature optimization is the root of all evil.

[maxbachmann]

since the messages are all posted to hermes/intent/<INTENT_NAME>/<message> the correct way to subscribe would be the wildcard hermes/intent/+, so it really only receives the intent messages and when someone decides for whatever reason to add the topic hermes/intent/<INTENT_NAME>/<NEW_FEATURE>/<message> you do not receive this message.

In terms of speed I am not even sure whether subsribing to one topic like hermes/intent/intentRecognized is in any way faster, since when subscribing to it you would receive all the messages even when your not interested in them -> you need to filter them yourself, which most likely is not faster than the implementation paho-mqtt is using (but then again yes this is definetly premature optimization)

[fastjack]

I agree that it is not a priority for now. Let’s stick to the protocol and we can extend it if required/useful in the future when Rhasspy is fully compliant and all the services are done and installable easily.

[KiboOst]

Yes sure, and this is how snips works and was not a problem !

[synesthesiam]

This is just the way MQTT works, and I find it very well designed

I agree. The “#” and “+” wildcards allow for a lot of flexibility and, with the right implementation, the performance impact is minimal. It’s much better to subscribe to a lot of topics than to JSON parse everything that comes across the wire.

[wisrou]

Hi here!

I just discover this project after I eard of the end of Snips solution.
I’m so excited !
I always want to put some “magic” into my home automation (voice control) and I’m really concerned by privacy.

So I’ve read a lot of topics (from Snips forum, to HA then here), documentation and so on.

Today, I have an Hass.io installation into a raspberry 3, with MQTT addon installed.
My next step would be to build a “satelitte” (rpi 0 + ReSpeaker ? Matrix Voice ?) for 3 differents rooms and control my home automation from voice.

So I have some questions reading this topic :

1. When we except have this new architecture in place ?
2. Did you have some milestone in mind ?
3. When the pi zero will be 100% support ?
4. Can we already used the Matrix-Voice-ESP32-MQTT-Audio-Streamer with a Rhasspy server ?

Thanks folks for all the great job already done and thanks to Snips to having boost this project

[synesthesiam]

When we except have this new architecture in place ?

The new architecture is mostly complete, but is missing the ability to re-train and download profiles. It currently works if you have an existing Rhasspy profile, but is still rough around the edges. I’m anticipating an alpha version with all the pieces in place in the next two weeks.

Did you have some milestone in mind ?

I’m making some modifications to Rhasspy’s training system so that it no longer depends on pre-compiled C++ libraries (opengrm, openfst, and phonetisaurus). Once that’s done, the only thing that needs to be pre-compiled is Kaldi, which will significantly simplify Rhasspy’s deployment. This is the milestone I’m looking at for the transition.

When the pi zero will be 100% support ?

Kaldi is the only piece that I haven’t gotten working on the Pi Zero. I need to double check that Porcupine and Snowboy are working there, though. If someone could figure out how to fix Kaldi, it would really speed things up.

Can we already used the Matrix-Voice-ESP32-MQTT-Audio-Streamer with a Rhasspy server ?

I believe that’s what @romkabouter is using here.

[romkabouter]

I use the streamer with the addon for Hassio, but that is basically the same

[wisrou]

Love those answers

[wisrou]

Good to know
I guess I’ll do the same in my house.

Or use a pi zero + respeaker 2-mic when the “satellite” architecture will be 100% support on this platform.

My main “problem” with the Matrix Voice is the price compare to the Pi Zero

[romkabouter]

Yes, this is an issue indeed.

[synesthesiam]

I’m looking for some feedback on training in the MQTT/Hermes architecture for Rhasspy. With slot programs and converters, things are a little more complicated.

I’m thinking of having the main Rhasspy server initiate the training process when you click Train or do a POST to /api/train. At that point, the server could gather your various sentences.ini files, run any slot programs, and then packages up sentences + slots to send over MQTT. Here’s how I’m imagining the flow:

1. User requests training from Rhasspy server
2. Server sends sentences + slots over MQTT (JSON? gzip?)
3. ASR service generates dictionary, language model, etc. from sentences + slots (using rhasspy-nlu for parsing)
4. NLU service generates graph, etc. from sentences + slots

When the NLU service receives a query, it will do recognition and then run converters locally before sending out a Hermes intent message. So, in this view:

• Rhasspy web server is responsible for running slot programs
• NLU service is responsible for running converters

Anyone see any issues with this?

[CrankyCoder]

Maybe a ping here and there so the base know the satellite is still up or updating some settings for example but no continual audio streaming between the two.

Since there seems to be a big push to mqtt (which I love) to even lower this to not even having to do the pings. We could do something similar to what the tasmota code does on my light switches. It creates a LWT (Last Will and Testament) message.

When the satellite starts, it sends a “Status” message that says “Im Online” with a LWT = “Offline” if the satellite code holds subscription to the master open, then that state will always = Im Online. But if it disconnects (pi restarts, network, power ect) the LWT will kick in on the master updating Status to “Offline”

Just a thought.

[synesthesiam]

@koan or @fastjack, do you happen to know which service is responsible in Snips for detecting silence and the end of a voice command?

I’m trying to decide when to send out a textCaptured event. It seems like the ASR service should be detecting silence, but then should the hermes/asr/stopListening message force a textCaptured if no silence has been detected yet?

[fastjack]

I was just thinking about this yesterday. They use Kaldi online endpointing feature in the ASR to detect that the utterance has ended. This avoids using an additional VAD system.

The stopListening topic is to explicitly tell the ASR to stop listening on a specific siteId. So I think that this topic should force a textCaptured message of what was transcribed between startListening and stopListening.

This is probably used as a timeout system by the dialogue manager in case the endpointing logic is not able to detect silence (noisy environment, tv background voices, etc.). I think I remember that there was a lot of people on the Snips forum that complained about this timeout happening too soon (5sec if I remember correctly).

[koan]

I don’t know the specifics of the Snips services, but @fastjack’s interpretation seems sensible to me: I agree that a hermes/asr/stopListening message should trigger a hermes/asr/textCaptured message, and this is corroborated by the Hermes reference section of textCaptured:

When the ASR is listening, it transcribes voice to text in real time. This process stops when a longer period of silence is detected, after which the transcription results are posted, as described here.

However, I’m not sure whether the ASR has a timeout system. I thought that in Snips, the timeouts are in the dialogue manager. For instance, if you look at the Snips configuration file, the only timeout values you can configure are:

• lambda_timeout (default value: 5 seconds): timeout between dialogue and lambda for endSession, in seconds
• session_timeout (default value: 15 seconds): internal timeout if one component doesn’t answer, in seconds

Both values are used by the dialogue manager, not by the ASR. I’m not sure what the lambda means in this context, by the way.

So it seems to me the dialogue manager has the responsibility to send hermes/asr/stopListening to the ASR when there’s no silence detected after a configured time (session_timeout or lambda_timeout?), and after that the ASR sends the hermes/asr/textCaptured message.

Or maybe the ASR does have a timeout in Snips, but it’s just not configurable? Maybe it’s good to have a look at what Hermod and Project Alice are doing?

[fastjack]

This is probably used as a timeout system by the dialogue manager

As I said the timeout should not be in the ASR but in the dialogue manager to ensure the platform does not become unresponsive. The ASR only stop listening by itself when it detect a silence via endpointing.

[koan]

Ok, then we’re saying the same I just wasn’t familiar with Kaldi’s endpointing feature.

[fastjack]

The dialogue manager looks like a central piece in the architecture (orchestrating the other services).

Looks like a « master » or « base » to me…

[synesthesiam]

I still need to add session timeouts to the dialogue manager. I don’t have the lambda timeout in the ASR either, just the silence detection (using webrtcvad).

I’ll leave the silence detection as an ASR responsibility, so we can do something different with Kaldi, Pocketsphinx, etc. For now, both use the rhasspy-silence library, but we can change that in the future.

Thanks!

[bruwozniak]

This is really great, I have struggled a little to have a good separation of concerns between base and satellite.
Is there a target version for this functionality? Or any other way I can check if it’s released yet? I am considering pausing my tinkering until those changes are rolled out.

[duch]

Sorry if i’m mistaken guys but i tried to publish the hermes/asr/startListening topic and Rhasspy did not wake, are this topics really available in Rhasspsy core or should i wait for an update.

The fact that they are mentionned in the docs confuses me.

[synesthesiam]

This only works currently with the Hermes Command Listener. In the near future, it will always work since Hermes will be baked into Rhasspy.