Rhasspy 3 Developer Preview

Announcements
1

At long last, a developer preview of Rhasspy 3 is finally ready :partying_face:
Check out the tutorial to get started!

I’m calling this a “developer preview” because this is not Rhasspy 3’s final form. It’s missing a lot of pieces, including a user-friendly Docker image. But here are some exciting things that do work:

Pipelines

A pipeline in Rhasspy 3 is basically the configuration for an entire Rhasspy 2 system: the mic, wake, asr, vad, etc. systems and their settings. You can have as many pipelines as you want in Rhasspy 3, and they can be run continuously or on-demand in response to HTTP/Websocket calls.

Pipelines can share access to small speech to text or text to speech servers too, so you don’t have to keep separate copies of models in memory.

Satellites

Rhasspy 3 has been designed for satellites from the ground up. Once you have the HTTP server running on your base station, setting up a satellite is pretty easy.

You don’t have to use Rhasspy for your satellites, though! The Websocket API lets you stream raw audio after your wake word is detected, and receive raw audio back with a text to speech response. In fact, you can just create a pipeline on the server for each satellite with the “mic” (audio in) and “snd” (audio out) programs being something like GStreamer.

Where is sentences.ini?

This is currently missing from Rhasspy 3, but for good reason. With the release of faster-whisper, it’s now possible to run the “tiny” model on a Raspberry Pi 4 with decent performance and accuracy. And with the Assist feature in Home Assistant (which I wrote :wink:), you can send the transcript from Whisper directly in without a separate intent recognizer.

Custom sentences and intents will be possible in the future for Rhasspy 3, but I’ve obviously focused on the use case that’s centered around my job at Nabu Casa so I can kind of have a life :laughing:

Feedback and Contributions

I’m looking for feedback, mostly from developers at this stage. The pipeline system is powerful, but lacks some features that I’d like to get design ideas for. For example, there needs to be a way for each stage of the pipeline to pass custom data to the next.

Contributions are welcome, but I’d caution anyone with spending too much time implementing stuff when the API is still in flux. Bug fixes and discussion about architecture limitations would be best :slight_smile:

13 Likes
2

I was looking at benchmarks and thought wow either that is fast, but don’t remember Whisper.cpp being that slow.
So had to give whisper.cpp a refresh

system_info: n_threads = 4 / 8 | AVX = 0 | AVX2 = 0 | AVX512 = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 0 | VSX = 0 |

main: processing 'samples/gb1.wav' (3179927 samples, 198.7 sec), 4 threads, 1 processors, lang = en, task = transcribe, timestamps = 1 ...


[00:00:00.000 --> 00:00:09.000]   My fellow Americans, this day has brought terrible news and great sadness to our country.
[00:00:09.000 --> 00:00:18.000]   At 9 o'clock this morning, mission control in Houston lost contact with our space shuttle Columbia.
[00:00:18.000 --> 00:00:24.000]   A short time later, debris was seen falling from the skies above Texas.
[00:00:24.000 --> 00:00:29.000]   The Columbia's lost. There are no survivors.
[00:00:29.000 --> 00:00:37.000]   On board was a crew of seven, Colonel Rick Husband, Lieutenant Colonel Michael Anderson,
[00:00:37.000 --> 00:00:46.000]   Commander Laurel Clark, Captain David Brown, Commander William McCool, Dr. Kultna Shavla,
[00:00:46.000 --> 00:00:52.000]   and Ilan Ramon, a colonel in the Israeli Air Force.
[00:00:52.000 --> 00:00:58.000]   These men and women assumed great risk in the service to all humanity.
[00:00:58.000 --> 00:01:02.000]   in an age when spaceflight has come to seem almost routine.
[00:01:02.000 --> 00:01:06.000]   It is easy to overlook the dangers of travel by rocket
[00:01:06.000 --> 00:01:11.000]   and the difficulties of navigating the fierce outer atmosphere of the Earth.
[00:01:11.000 --> 00:01:17.000]   These astronauts knew the dangers, and they faced them willingly,
[00:01:17.000 --> 00:01:21.000]   knowing they had a high and noble purpose in life.
[00:01:21.000 --> 00:01:26.000]   Because of their courage and daring and idealism,
[00:01:26.000 --> 00:01:39.000]   we will miss them all the more. All Americans today are thinking as well of the families of these men and women who have been given this sudden shock and grief.
[00:01:39.000 --> 00:01:51.000]   You're not alone. Our entire nation grieves with you. And those you love will always have the respect and gratitude of this country.
[00:01:51.000 --> 00:01:55.720]   The cause in which they died will continue.
[00:01:55.720 --> 00:02:04.120]   Mankind is led into the darkness beyond our world by the inspiration of discovery and
[00:02:04.120 --> 00:02:07.000]   the longing to understand.
[00:02:07.000 --> 00:02:11.160]   Our journey into space will go on.
[00:02:11.160 --> 00:02:16.480]   In the skies today, we saw destruction and tragedy.
[00:02:16.480 --> 00:02:22.040]   farther than we can see, there is comfort and hope.
[00:02:22.040 --> 00:02:29.280]   In the words of the prophet Isaiah, "Lift your eyes and look to the heavens."
[00:02:29.280 --> 00:02:31.640]   Who created all these?
[00:02:31.640 --> 00:02:39.260]   He who brings out the story hosts one by one and calls them each by name.
[00:02:39.260 --> 00:02:46.400]   Because of His great power and mighty strength, not one of them is missing.
[00:02:46.400 --> 00:02:53.580]   The same Creator who names the stars also knows the names of the seven souls we mourn
[00:02:53.580 --> 00:02:55.580]   today.
[00:02:55.580 --> 00:03:03.140]   The crew of the shuttle Columbia did not return safely to Earth, yet we can pray that all
[00:03:03.140 --> 00:03:05.820]   are safely home.
[00:03:05.820 --> 00:03:12.640]   May God bless the grieving families and may God continue to bless America.
[00:03:12.640 --> 00:03:22.640]   [BLANK_AUDIO]


whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:     load time =   655.10 ms
whisper_print_timings:      mel time =  1611.87 ms
whisper_print_timings:   sample time =   436.05 ms /   520 runs (    0.84 ms per run)
whisper_print_timings:   encode time = 82173.55 ms /     8 runs (10271.69 ms per run)
whisper_print_timings:   decode time = 34584.14 ms /   520 runs (   66.51 ms per run)
whisper_print_timings:    total time = 119509.48 ms

real    1m59.635s
user    7m49.414s
sys     0m2.604s

‘faster_whisper’

orangepi@orangepi5:~/faster-whisper$ time OMP_NUM_THREADS=4 python3 my_script.py
Detected language 'en' with probability 0.989548
[0.00s -> 9.00s]  My fellow Americans, this day has brought terrible news and great sadness to our country.
[9.00s -> 18.00s]  At 9 o'clock this morning, mission control in Houston lost contact with our space shuttle Columbia.
[18.00s -> 24.00s]  A short time later, debris was seen falling from the skies above Texas.
[24.00s -> 29.00s]  The Columbia's lost. There are no survivors.
[29.00s -> 37.00s]  On board was a crew of seven, Colonel Rick Husband, Lieutenant Colonel Michael Anderson,
[37.00s -> 44.00s]  Commander Laurel Clark, Captain David Brown, Commander William McCool,
[44.00s -> 52.00s]  Dr. Kultna Shavla, and Ilan Ramon, a colonel in the Israeli Air Force.
[52.00s -> 58.00s]  These men and women assumed great risk in the service to all humanity.
[58.00s -> 63.00s]  In an age when space flight has come to seem almost routine,
[63.00s -> 67.00s]  it is easy to overlook the dangers of travel by rocket
[67.00s -> 72.00s]  and the difficulties of navigating the fierce outer atmosphere of the Earth.
[72.00s -> 78.00s]  These astronauts knew the dangers, and they faced them willingly,
[78.00s -> 83.00s]  knowing they had a high and noble purpose in life.
[83.00s -> 90.00s]  Because of their courage and daring and idealism, we will miss them all the more.
[90.00s -> 96.00s]  All Americans today are thinking as well of the families of these men and women
[96.00s -> 100.00s]  who have been given this sudden shock and grief.
[100.00s -> 105.00s]  You're not alone. Our entire nation grieves with you,
[105.00s -> 112.00s]  and those you love will always have the respect and gratitude of this country.
[112.00s -> 116.00s]  The cause in which they died will continue.
[116.00s -> 121.00s]  Mankind is led into the darkness beyond our world
[121.00s -> 127.00s]  by the inspiration of discovery and the longing to understand.
[127.00s -> 131.00s]  Our journey into space will go on.
[131.00s -> 136.00s]  In the skies today, we saw destruction and tragedy.
[136.00s -> 142.00s]  As farther than we can see, there is comfort and hope.
[142.00s -> 145.00s]  In the words of the prophet Isaiah,
[145.00s -> 151.00s]  lift your eyes and look to the heavens who created all these,
[151.00s -> 155.00s]  he who brings out the story hosts one by one
[155.00s -> 159.00s]  and calls them each by name.
[159.00s -> 163.00s]  Because of his great power and mighty strength,
[163.00s -> 166.00s]  not one of them is missing.
[166.00s -> 169.00s]  The same Creator who names the stars
[169.00s -> 175.00s]  also knows the names of the seven souls we mourn today.
[175.00s -> 180.00s]  The crew of the shuttle Columbia did not return safely to Earth,
[180.00s -> 185.00s]  yet we can pray that all are safely home.
[185.00s -> 189.00s]  May God bless the grieving families
[189.00s -> 194.00s]  and may God continue to bless America.

real    1m22.424s
user    6m13.880s
sys     0m17.584s

Maybe its the chosen CPU Xeon(R) Gold 6226R as surprised but the rk3588s is faster than a Xeon Gold 6226R on 4 cores…
Faster_Whisper is faster than Whisper.cpp but not > x6 faster, as great the 8bit conversion seems to run great but confused to the benchmark figures posted on Github? Anyone else the same.

The tiny model runs approx same ratio as the small which I 1st choose as that was the supposed benchmarks.

whisper.cpp_tiny

real    0m18.860s
user    1m6.639s
sys     0m0.945s

faster_whisper_8bit_tiny

real    0m16.380s
user    1m10.504s
sys     0m2.735s

That is purely curiousity at the benchmarks posted on GitHub - guillaumekln/faster-whisper: Faster Whisper transcription with CTranslate2

Suprised you started with Whisper though as still on a Pi even the tiny model isn’t the fastest, but will have to see what benches you get with Tiny.

Do you have anything in the pipeline based on custom LM say with n-grams incorporating Assist entity construction?

3

I’m getting a similar relative speed-up on my Ryzen 5950X: faster-whisper is about 1.4x faster than whisper.cpp. So I think their claims are a bit overblown, but maybe they were comparing to a really old version of whisper.cpp?

4

Yes, the HassIL library that Assist uses for intent recognition can generate all possible sentences too (when you have the entity/area lists present). I plan to use the same trick I did in Rhasspy, where I generate the ngram counts directly from the sentence templates (instead of templates → sentences → ngram counts).

1 Like
5

Yeah I am only getting x1.151 but Whisper.cpp is optimised for the newer Mac Arm silicon that seems to suit the RK3588s haven’t tried a Pi yet.
Yeah that is what I thinking, but still wondering if the option should be open to have a simple small fast predicate ASR and rules to route to a secondary ASR.
As a LM of just the predicate / subjects of Assist entities should make a fast accurate ASR specific to Assist where a n-gram is generated on command.

That is the wierd thing about ASR as Whisper is a great long sentence conversational ASR but from playing it can be no better and even worse than specific LM ASR, that is conversely as bad for long sentence conversational ASR.
So how do you employ a middle layer (simple predicate ASR) to route the ASR most suited, or there is a fallback list and if one fails, on fail it will try a secondary…

Looks like you have been very busy, Assist looks great and likely would benefit much from a ASR model with a specific entity subset language model.
Would failover to Whisper work as an option as likely much faster than whisper also?

6

This seems ideal, yeah. The tricky part is knowing when exactly to fail over, as these systems will always return something from the training set. The “confidence” scores from systems like Kaldi are really just the probability of the guessed sentence relative to the ngram model, not relative to all possible spoken sentences.

I have some prototypes of a constrained ASR system (based on Kaldi) that can guess pretty accurately when something spoken is outside its tiny LM. With that, we should have the best of both worlds :slight_smile:

Another option I’ve considered is using something like string-edit distance to check the Whisper transcript against a small LM, and “repair” the transcript if it’s close enough.

For example, Whisper often hears “turn off the bad light” instead of “turn off the bed light”. That’s only 1 replace op, so it could be repaired.

7

Quick question… I skimmed the Assist documentation but couldn’t find an answer… is it possible to disable the default sentences?
Rhasspy regularly misunderstands me, often under difficult circumstances, like when I have the tap running, but still. So unless the speech recognition miraculously became flawless in Rhasspy 3, I certainly wouldn’t want Rhasspy to have the power to freely toggle any entity on or off. I’m fine with Rhasspy occasionally toggling the wrong light but I certainly wouldn’t want it to turn off the heating, unlock the car, falsely logging a diaper change or things like that.

8

That and the token length I guess as duration between command and conversational is distinct?

I also had a search for how you enable/disable entities and if the default is enabled/disabled or something along those lines and guess the documentation just takes time.
Is it post to github or here?

1 Like
9

We’re going to add the ability to decide exactly which entities/domains to expose to Assist. Later, we also plan to add confirmation for specific things, like locks and garage doors.

Yeah, that should work to our advantage since longer conversational sentences should require more “repairs” and exceed some threshold quickly.

1 Like
10

Whisper is fine for a technology preview, Assist as an inference based skill, so far looks really good to me.
Congrats in getting it all floated.

1 Like
11

Pretty interested to know what you put together for this :blush: Congrats on the dev preview. Looks good.

1 Like
12

The Websocket API is awesome!
I think it will simplify for most ppl using a satellite because you don’t have to worry about a lot on the satellite anymore.

1 Like
13

Yeah good to see websockets and could maybe extend into net pipeline to run on other instances and just link them up in a serial chain.
Compression might be a useful option Compression - websockets 10.4 documentation
You can check the received data type if str or binary, so you can easily seperate audio stream and protocol. Server - websockets 10.4 documentation

The client can “end” the audio stream by sending an empty binary message.

A start message could also be handy and a text protocol ‘start/stop…’, the control flow of text and binary frames helps differentiation…
Never been a fan of the websockets documentation though, so great just to see something that works :slight_smile: good job.

1 Like
14

@synesthesiam

Awesome news to see the next gen of Rhasspy coming together. Looks like you’ve been hard at work both here and at Nabu Casa (I’ve been watching the intents project). Thank you for everything you’ve done for this community! Having a truly local voice assistant is the piece I needed to step into the voice assistant world.

As a semi-new member of the community and potential skill developer, selfishly, I’m wondering if you could share you plans for intent handling in the future? I see Rhasspy 3 works with HA right now, but you seem to say that isn’t the long term plan. Do you plan to go back to a MQTT type connectivity, purely websockets, etc.? Very interested. I have recently started development on some skills for Grocy and Jellyfin for Rhasspy 2.5 using the Rhasspy Hermes App project as my basis, and released a polished time skill and timer skill, but now it seems you may be moving away from that model? Am I seeing that right? If so, I’m all for learning a new path just interested in what that path may be so I can get started asap.

Also, as someone who has been trying to develop some useful skills on 2.5 a suggestion for the Rhasspy API going forward. If you think of the use case for these “devices” one of the very early things someone is going to want to do with them are some of the basic O/S level things, like connect a bluetooth headset, turn the volume down, switch to a different audio output (maybe even turn a screen on and off). If you can build the ability to manipulate those things on all devices in the Rhasspy infrastructure via the API then intents can be built to take advantage of that. I am also working on something like that for 2.5, I call it a satellite skill, designed to be run on the satellite directly, still Rhasspy Hermes App based, but registers dynamically named skills on the MQTT bus (name is based on intent action + satellite name so each satellite gets unique intents to control themselves). Right now I have basic volume control working, volume up/down/set/mute. Bluetooth intents were next. Again looking to how to “migrate” this to Rhasspy 3.

Again thank you for everything! Please don’t take any of this as complaining. Just hoping to catch the train as it’s leaving the station and not be too late!

1 Like
15

Just hoping to catch the train as it’s leaving the station and not be too late!

As yeah its a great time to discuss before things become set in stone, as Rhasspy 3.0 is probably a bad name as it tends to indicate Rhasspy 2.5 has an end.
I am reading @JoeSherman and thinking the reason for the huge divergence in Rhasspy 3.0 is partially due to the MQTT protocol 3.0 is trying to make a clean start on.

If your mindset and solution fits the Rhasspy 2.5 model then develope on Rhasspy 2.5 as Rhasspy 2.5 has a more descriptive name as Raspberry/Hass/Pi has strong connotations, whilst Rhasspy3.0 is already employing tech ‘Whisper’ that stretches any Pi hardware, whilst due to the nature of many current SotA models have beam searches and use context and are race-till-idle suited than streaming.
Also what Hass have done with Assist Assist - Talking to Home Assistant - Home Assistant is absolutely amazing stuff and Rhasspy is no longer as the ill placed. Hass has moved to a inference based skill that the bigger user & dev community of Hass will supply to maximise support and usage that can now utilise any Voice system that can supply inference.

Thats a win/win situation as Hass now gets a great inference based skill and RPy gets this really great skill and that is an important difference between the 2.5 & 3.0 model, but I am not kidding about the name though but its 3.0 that maybe should change and both continue with clear and distinct directions.

For Hass the Assist module has just opened up so many avenues as a Dev on any platform only needs to feed intent and there is a clear partition between voice system and a skill server.

Again its a different mindset but instantly I am questioning why embed audio functionality when great opensource wireless audio projects exist, with bigger user bases such as Squeezelite or Snapcast and a whole load of RTP libs and audio servers, will do this.
An inference based skill just like Hass will get more users, likely become more strong because of the obvious introperability of inference based skill adds, than embedded functionality.

Hass does have an audio system, that likely in 3.0 can do this already if you employed the Hass audio system. I am not a Hass user and a read some articles quite a while ago, but dev is ongoing and the rest I have forgot.
In 3.0 I could tell you how to employ this with wireless audio quite easily as part of the TTS module should be an audio router that with a ready made wireless audio systems be it Hass, Squeezelite or Snapcast installing clients is not a huge endevour that why repeat the Dev we will never achieve of what they already offer in wireless audio?

I am not saying that idea is bad it just a bad fit, but reading through the repo things like the The Wyoming Protocol is a huge paradigm shift from 2.5, so much so that are these now distinct companion projects?

2.5 might get more Pi based focus, whilst 3.0 is more of a Question of how to intergrate SotA models, applications that has some freakenly insane ML going on.
There seems to of been a lot of work and a big shift in direction, that its understanding 3.0 and ignoring 2.5 until things are clearer, or continue with 2.5.

16

Hi @synesthesiam
Thanks for all your hard work with Rhasspy and Home Assistant!
I tested out the HA addon and it works very well, but I noticed one thing that may be a bug.
It seems when the TTS gets a large chunk of text, you can’t hear anything coming back from larynx, though the log shows is sent:
DEBUG:rhasspy3.program:client_unix_socket.py [‘var/run/faster-whisper.socket’]
DEBUG:rhasspy3.program:vad_adapter_raw.py [‘–rate’, ‘16000’, ‘–width’, ‘2’, ‘–channels’, ‘1’, ‘–samples-per-chunk’, ‘512’, ‘script/speech_prob “share/silero_vad.onnx”’]
DEBUG:rhasspy3_http_api.pipeline:stream-to-stream: voice started
DEBUG:rhasspy3_http_api.pipeline:stream-to-stream: voice stopped
INFO:faster_whisper_server: What’s the weather like?
DEBUG:rhasspy3_http_api.pipeline:stream-to-stream: asr=Transcript(text=" What’s the weather like?“)
DEBUG:rhasspy3.program:handle_adapter_text.py [‘bin/converse.py --language “” “http://supervisor/core/api/conversation/process” “/app/config/data/handle/home_assistant/token”’]
DEBUG:rhasspy3.handle:handle: input=Transcript(text=” What’s the weather like?")
DEBUG:rhasspy3.handle:handle: Handled(text=‘Currently the weather is sunny, with a temperature of 43 degrees. Under present weather conditions the temperature feels like 36 degrees. In the next few hours the weather will be more of the same, with a temperature of 43 degrees.’)
DEBUG:rhasspy3_http_api.pipeline:stream-to-stream: handle=Handled(text=‘Currently the weather is sunny, with a temperature of 43 degrees. Under present weather conditions the temperature feels like 36 degrees. In the next few hours the weather will be more of the same, with a temperature of 43 degrees.’)
DEBUG:rhasspy3_http_api.pipeline:stream-to-stream: sending tts
DEBUG:rhasspy3.program:client_unix_socket.py [‘var/run/larynx2.socket’]
Real-time factor: 0.107471 (infer=1.27417 sec, audio=11.856 sec)
DEBUG:rhasspy3_http_api.pipeline:stream-to-stream: tts done

Also noticed after the request my dev console gets flooded with this:
image

Figured I’d call it out :slight_smile: When I tested other commands that had shorter text they all seemed to work fine.

EDIT: Opened issue on github as I figured that was the more proper channel for this :smiley:

1 Like
17

You’re welcome! This is great to hear :slight_smile:

My hope is that the type of connectivity won’t matter quite so much going forward. Rhasspy 3 pipelines are just made up of programs, so it’s not a problem if one or more of them connect out to MQTT. I could see a small bridge server for Hermes being necessary to make it work smoothly, but nothing requiring a total rewrite of the skills.

No worries :wink: The plan is not to deprecate MQTT/Hermes, but to see it as just one way among many to connect things to a voice assistant.

My hope was to boil skills, etc. down to their bare essentials in Rhasspy v3. For example, the handle domain can take programs that take text in (speech to text transcript) and push text out (text to speech response). So this is a valid “skill”:

cat

which just repeats back whatever you say! You need an adapter, etc. of course for Rhasspy 3, but the complete YAML isn’t too much worse:

  handle:
    repeat:
      command: |
        cat
      shell: true
      adapter: |
        handle_adapter_text.py

I considered renaming, but to me Rhasspy v3 is delivering more on the promise of Rhasspy being a toolkit for building your own voice assistant.

For example, you can build a satellite now that consists entirely of two GStreamer processes: one streaming mic audio to a base station, and one playing audio from a base station. You can add OPUS or whatever compression, and have an always-on satellite with the wake word running on the base station. Or you can play back audio to a Bluetooth speaker, etc. Just change the snd program :nerd_face:

18

I still think there are 2 camps as there are some who just want a relatively simple web-based setup to suit various hardware aka various Raspberry Pi.
Then there are those that may be looking more multi-room, home-control, distributed audio employing cutting edge SotA ML models that could scale into money is no object…
I was just wondering if simplicity & complexity where at odds with each other and distinct paths.

I think the Assist inference based skill is absolutely amazing and wish more projects supplied an inference interface as it is super convenient for any voice assistant and so much better for a skill to be supported by the project its a skill for.
Its very likely my choice will be Hass due to the Assist module now, but my focus on modern distributed audio means an embedded audio system isn’t of much interest as I intend to have a singular base station that will route audio to a audio server and don’t really have such a thing as a ‘Satelite’

I like the websockets interface and I like the emphasis on being a toolkit for building your own voice assistant especially you can wrap a module in the Wyhoming protocol so you can almost drop in a stream to stfout.
Still curious to why not something Gstreamer to try and minimise high level python with DSP.

I never did check that article but the lib example is much better.
Would it not of been great to have Gstreamer ASR & TTS modules?

In fact I didn’t know Kaldi had a gstreamer plugin until I wrote that.
Kaldi: Online Recognizers whilst I was googling there are some others.
Gst-nvdsasr — DeepStream 6.2 Release documentation but Nvidia…
Using PocketSphinx with GStreamer and Python – CMUSphinx Open Source Speech Recognition

19

This is what I want to do - preferably running on a Pi or VM hosted in the cloud.
And I think it makes sense to not focus on the home-automation config that HASS already has covered.

:partying_face:

1 Like
20

There is definitely an overhead, though Rhasspy 3’s core could technically be rewritten in something like Rust for less overhead. But the majority of the time spent on the Python side is in async await around subprocesses, so the only real savings would probably be in fewer memory allocations.

The GStreamer pipeline paradigm seems like it would be a good fit for Rhasspy too, since it can describe an actual graph and not just a simple feed-forward pipeline.

1 Like
21

Yeah I hadn’t really given Gstreamer python much thought until then but I had a look at GitHub - GStreamer/gst-python: GStreamer Python binding overrides (complementing the bindings provided by python-gi). This module has been merged into the main GStreamer repo for further development. and GST sinks/sources are not that complex to make.
The websocket as a GS sink or source as still trying to work out the terminology specific to GST (I would call it a source but maybe its a sink as will be a pipeline…) would be really cool and likely make many things easier.
GS is great as where ever possible use an existing module than writing your own, where likely your own module whatever language is in a GS pipeline.

Does that Kaldi GST build still work?

22

two things about R3 with porcupine I’m testing in a Ubuntu VM:

  1. it seems the base config is not being overridden by the user config.
    Changing the wake file to grasshopper_linux.ppn only took effect when I put it in the base config, but didn’t when put in the user config file. Anyone else see this?
  2. I created a custom wake word ( linux *.ppn ) file with the Picovoice console and can’t get it to work with R3.
    Is there something I need to do for compatability like an access key or something?
    Error I got is shown below.

What can I use for a custom wake word if/until I can get porcupine working?

$ script/run bin/wake_detect.py --debug

DEBUG:rhasspy3.core:Loading config from /home/sass/working/rhasspy3/rhasspy3/configuration.yaml
DEBUG:rhasspy3.core:Skipping /home/sass/working/rhasspy3/config/configuration.yaml
DEBUG:wake_detect:mic program: PipelineProgramConfig(name=‘arecord’, template_args=None, after=None)
DEBUG:wake_detect:wake program: PipelineProgramConfig(name=‘porcupine1’, template_args={‘model’: ‘alice_en_linux_v2_1_0.ppn’}, after=None)
DEBUG:rhasspy3.program:mic_adapter_raw.py [‘–samples-per-chunk’, ‘1024’, ‘–rate’, ‘16000’, ‘–width’, ‘2’, ‘–channels’, ‘1’, ‘arecord -q -D “default” -r 16000 -c 1 -f S16_LE -t raw -’]
DEBUG:wake_detect:Detecting wake word
DEBUG:rhasspy3.program:.venv/bin/python3 [‘bin/porcupine_stream.py’, ‘–model’, ‘alice_en_linux_v2_1_0.ppn’]
Traceback (most recent call last):
File “/home/sass/working/rhasspy3/config/programs/wake/porcupine1/bin/porcupine_stream.py”, line 110, in
main()
File “/home/sass/working/rhasspy3/config/programs/wake/porcupine1/bin/porcupine_stream.py”, line 61, in main
porcupine = pvporcupine.create(
File “/home/sass/working/rhasspy3/config/programs/wake/porcupine1/.venv/lib/python3.10/site-packages/pvporcupine/init.py”, line 64, in create
Traceback (most recent call last):
File “/home/sass/working/rhasspy3/bin/wake_detect.py”, line 80, in
asyncio.run(main())
File “/usr/lib/python3.10/asyncio/runners.py”, line 44, in run
return loop.run_until_complete(main)
File “/usr/lib/python3.10/asyncio/base_events.py”, line 646, in run_until_complete
return future.result()
File “/home/sass/working/rhasspy3/bin/wake_detect.py”, line 69, in main
detection = await detect(rhasspy, wake_program, mic_proc.stdout)
File “/home/sass/working/rhasspy3/rhasspy3/wake.py”, line 109, in detect
wake_event = wake_task.result()
File “/home/sass/working/rhasspy3/rhasspy3/event.py”, line 48, in async_read_event
event_dict = json.loads(json_line)
File “/usr/lib/python3.10/json/init.py”, line 346, in loads
return _default_decoder.decode(s)
File “/usr/lib/python3.10/json/decoder.py”, line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File “/usr/lib/python3.10/json/decoder.py”, line 355, in raw_decode
raise JSONDecodeError(“Expecting value”, s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

23

Regarding Procupine, I think your issue is that presently Rhasspy3 is using Porcupine1, which I don’t think supports the custom wake word generated from the present Picovoice Console since those now require the API key to use. That functionality is in Porcupine2. I know some users here (myself included) managed to patch in Porcupine2 into Rhasspy2, but doesn’t appear to be in Rhasspy3. Maybe a good enhancement request? :slight_smile:

Regarding your first point, I didn’t play with wake word yet, was just testing using the satellite in the browser, but my overrides did appear to work (I replaced the larynx2 TTS with Mimic3 as there’s a voice on there I like lol)

1 Like
24

Thanks. So what are my options for a custom wake word then? something else besides Picovoice that doesn’t require a fking API key?

25

Think the docs mention other wake word engines that support this besides picovoice.
image

I believe precise allows you to train your own wake model. Haven’t played around with it personally but I believe that was the same engine used for Mycroft. I believe you could also try installing picovoice2 and integrating it with Rhasspy3 as well, since the dev stack and modular nature should allow for that :slight_smile:

1 Like
26

wow, hard to find how to do this stuff - make custom WW w/mycroft precise.

btw, also saw the openWakeWord project, that’d require some code changes too.

27

This is an OVOS plugin for openWakeWord, an open-source wakeword or phrase detection system. It has competitive performance compared to Mycroft Precise or Picovoice Porcupine, can be trained on 100% synthetic data, and can run on a single Raspberry Pi 3 core.

28

I think the project page has a link to documentation on how to do it, with an included tool from what I remember: Training your own wake word · MycroftAI/mycroft-precise Wiki · GitHub

2 Likes
29

Thanks. Well it’s a build PITA. Requires ‘tensorflow>=1.13,<1.14’
looks like maybe I got it installed.

30

well the build tensorflow, for Mycroft Precise, using python 3.10 went mostly ok until near the end but never completed. I have to roll my python back to 3.7 and try again.

31

This is on my TODO list to fix, even though I don’t work for Mycroft anymore. Precise is such a simple model that it would be a tiny amount of PyTorch code these days.

Also, I have snowboy-seasalt as a Docker image if you want to train your own snowboy wake word.

Lastly, I plan to add snowman to the Rhasspy 3 wake word engine list.

1 Like
32

After a couple of years using my custom made vocal assistant I’ve come to the conclusion that there is a missing component in the system: the DSP that remove unwanted noise from the user input (be it kitchen noises or other people voices aka cocktail party effect).

Without this component any vocal assistant pipeline will be erratic at best.

I firmly believe that it also have to be in the todo list for any open source vocal assistant to work good enough for wide user adoption.

There have been some impressive advancements in this area in the last 2 years with Google Voice Filter Lite and more recently :

Using a generic KWS that feeds a voice recognition system based on the catched keyword audio (or a personalized KWS tailored for specific voices) that then feeds a centralized voice separation model before the ASR component will much more effectively improve ASR and NLU confidence than any other noise reduction, beam forming or AEC system.

The best solution should be to not only remove/attenuate noises but to only keep the utterance voice. Recent ASR systems are today pretty tolerant to some amount of noise but often fall short with overlapping voices.

It also voids the need for multiple mics which is a huge plus regarding hardware complexity (as demonstrated by recent Google Nest HW changes)

My 2 cents :blush:

1 Like
33

Snowboy is an iconic piece of KWS history as the 1st to employ a DNN system, but sort of similar to eSpeak in use as really pretty terrible in use now, as the pace of change of technology of early 1st’s has them both now left far behind.
I guess because you can?

@fastjack Yeah I am the same as often 3rd party noise is the dominant noise and even with beamforming it just gets swamped by the likes of TV or other voices.
The only code I have found was GitHub - BUTSpeechFIT/speakerbeam, but just been looking again.
GitHub - etzinis/heterogeneous_separation: Code and data recipes for the paper: Heterogeneous Target Speech Separation seems new and one I must of missed GitHub - Edresson/VoiceSplit: VoiceSplit: Targeted Voice Separation by Speaker-Conditioned Spectrogram and just a general BSS ML repo GitHub - fakufaku/torchiva: Blind source separation with independent vector analysis family of algorithm in torch

But for a while I have thought any BSS with a personalised VAD or KWS on each stream it splits into can detect a target be it personalised VAD or KW as that is the drawback with BSS algs as generally they will split into nSignals dictated by nMics that can find distinct sources from the TDOA they detect with no concept of content.

Seems that is what Espressif are doing as its a simple BSS spliting into 2/3 streams where they simply put a KWS on each stream to select the target.
Voicefilter-lite is a ML based BSS that steers a target into a single channel and can use the target to further filter the voice required, all in a single model.
For humans to interject requires 2 voices (noise sources) and when you get 3 or more it quickly becomes a cacophony where Google said great as with just 2x mics and a clever lite weight model they can get much better results in all scenarios that are usable anyway.
Esspressif have a 3mic version I guess because they have no voice-filter-lite and like beamforming more mics equals more resolution and seperation, but also another channel to scan for if its the one you require.

34

Thanks I will try it out. If it works good enough for now, will use it, until other options become available or easier to build.

35

So, my use case would be (in pseudo pipeline yaml): no audio on server… no heavy processing stt/tts on client.

client/satellite:
  mic:
  vad:
  remote: -> send vad wav to server
  sound: <- audio from server

server/base:
  asr-stt:
  handle:
  intent:
  tts: -> audio to client

It doesn’t seem like your current pipeline supports this? I could probably create a dummy wake word that always returns true… but it seem asr is hard coded with mic processing? Am I wrong about this?

-ryan

36

Got the snowboy custom wake word working, thanks!
I trained it with 25 utterances of various TTS voices from different countries/accents.
It seems to work pretty well in my Ubuntu VM. @synesthesiam Will this and Rhasspy3 work on a Pi4 also?

1 Like
37

@synesthesiam have you given any thought to including speaker-verification/voice-authentication (ie, biometric) into Rhasspy3? Does the new modular nature of V3 it make this easier to integrate this feature into a custom assistant? What open source projects have some speaker-verification/authentication functionality atm, DeepSpeech? Coqui?

38

Also which of your to-do are you working on or will be soon if any? Im particularly interested in the custom STT grammars and intent systems.

  • A user friendly web UI
  • An automated method for installing programs/services and downloading models
  • Support for custom speech to text grammars
  • Intent systems besides Home Assistant
  • The ability to accumulate context within a pipeline
39

Your product is incredible, are there any estimated dates for finished product by docker? And if I use rasspy 2.4 will it be hard to migrate? if i use remote http.

40

These things are all on my TODO list :+1:

Accumulating context within a pipeline is going to be needed for speaker identification/verification. I don’t have a specific program in mind for this yet, though. Some projects I’m looking at are personalVAD and Personalized PercepNet.

I don’t think it will, unfortunately. Seasalt only contained code for x86_64 systems, so I don’t think it will create wake words for arm64 systems. It might be possible to extend Seasalt, but I’m not sure how they created it in the first place.

Snowman, on the other hand, should work just fine on a Pi 4 :slight_smile:

Thanks! I plan to keep the same sentences format, and I will add some backwards compatible endpoints to the HTTP API. So using 2.4 to start shouldn’t be a big problem for migration. But I don’t have any estimated dates yet, sorry :confused:

41

@synesthesiam thanks for the info; that’s too bad about the Seasalt custom wake-word.
I really need it to work on a Pi in addition to x86 VM.
I think you should give some info on how other people can help you with the to-do’s…
ie, what needs to be done - research items, testing configurations, python code that needs to be written, etc. If you break down the to-do tasks maybe other people can help out?

what do you mean “Accumulating context” can you give some examples?

1 Like
43

For me accumulating context in a pipeline of passing via stdin and stdout, just creates complexity and unnecessary overhead in the audio stream for no apparent reason.
An audio stream is an audio stream and a accompaning YAML is a accompaning YAML that doesn’t need to be passed through each stage in a pipeline its just associated with that stream and also audio files as the pipeline is assuming a realtime stream but it could equally be a queued file that is simple same name different extension.
Websockets again is great for this as it has 2 packet types text & binary, so its super easy to seperate the 2 and send varying length text and binary without having to provide a strict protocol and its super lite as you are not applying a unnesscary protocol.
Linux has well defined streaming protocols and the python module for gstreamer looks absolutely perfect as does websockets as the network layer is extremely important to be able to link up instances.
A linux speech recognition system is just a very simple serial processing chain of streams that you should be able to route, queue and split so you can easily apply different processing chains to different hardware and make scalable by having parallel instances.

There is zero advantage to embedding context in a totally proprietory protocol when everything can already be accomplished with standard Linux interfaces, that are hugely supported and more performant.

1 Like
44

I mean something like having a wake word system that can also identify the speaker, and a speech to text system that has speaker-dependent models. The speech to text portion of the pipeline needs to be able to get the speaker id from the earlier wake word stage.

I agree, I always feel perpetually behind :stuck_out_tongue:

The reason is so we can just directly call the original programs instead of having to wrap everything in a plugin for a specific framework like GStreamer.

1 Like
45

Thats my whole point as you don’t call the original program you just route the audio to it and you don’t have to use Gstreamer to do that, you can if you want.
Gstreamer was mentioned as for some reason rhasspy3/wyoming.md at master · rhasspy/rhasspy3 · GitHub seems to be just a completely proprietory Gstreamer like protocol that once again creates complexity by embedding control events and audio stream as one.
That step makes the ease of passing linux audio as files or streams into one that needs programatic conversion into this gstreamer like with embedded event info Python protocol.
Your embeding audio into a purely proprietory protocol and unlike Gstreamer, Pulse, Alsa, Snapcast that all have ready interfaces for each other in highly optimised ready and tested code, you are providing a completely proprietory audio stream control, that has no need!?
Also to top it off you don’t call the original program you just need a simple serial processing chain of streams/files that you should be able to route to the app as with voice applications they should already be loaded and waiting for input to minimise latency.
I don’t have to wrap anything in a plugin as Gstreamer is a standard linux framework with a huge array of ready made plugins.
The oppiste is true where you have created this Wyoming protocol that exists nowhere and to use it everything needs to adopt it.

That is really easy to accomplish where speech recognition programs sit waiting for a files or streams of standard linux frameworks that is inclusive of a huge range of ready tools that can provide this … ?

46

I think its time to give Rhasspy 3 a try.

  1. What platform can the current Rhasspy 3 server run on ?
  • Your tutorial starts with CLI commands, so I guess it is not intended for HAOS ?
  • Will it run on a RasPi, or does it need an X86 and linux ? I think I noticed somewhere that one package isn’t available for ARM processor.

In other words, should I use my old RasPi 4 + HAOS system (from before I upgraded to an old PC) ? Or create a new VM on my X86 desktop ?

  1. Similarly, what platform can Rhasspy 3 satellite run on ?

  2. My objective is to produce a tutorial and/or documentation for non-developers.

47

I use it on a x64 linux (fedora server). It should also work on arm I think, many submodules are the same as rhasspy 2.

1 Like
48

But the Wyoming protocol and it’s requirements/dependencies is simple to learn and implement.
And that’s the stated goal of Rhasspy V3 - simple for apps/devs to integrate/interoperate.

“For v3, a project goal was to minimize the barrier for programs to talk to Rhasspy.”

1 Like
49

That is the strange thing as it is a barrier in itself because its a proprietary protocol requiring programs to talk ‘Rhasspy’.
Things seem strangely in reverse for both input and out programs as its not a matter of the Linux kernel adopting Rhasspy protocol as audio is embedded as ALSA ( Advanced Linux Sound Architecture) and various servers are built ontop of that.
They are standard high performnace Linux libs for passing audio and now all programs must be converting to Wyoming for input to Rhasspy.
Why I am scratching my head is why does this as yeah have a simple protocol but don’t embed it into standard audio streams and make them non standard whilst its so easy to use kernel interfaces (ALSA) and the existing audio servers (Pulse, Pipewire, Gstreamer) and just pass the json as an external config file (YAML).

So if you talk about examples that currently work with Rhasspy such as Voice-EN AEC, Speex-AGC, Deepfilter.Net and infact any VAD or KWS that used standard audio interfaces will now have to be converted to embed Rhasspy metadata and render the standard audio stream proprietory.

So if we take for example rhasspy3/wyoming.md at master · rhasspy/rhasspy3 · GitHub and the event types.

Events Types

  • mic
    • Audio input
    • Outputs fixed-sized chunks of PCM audio from a microphone, socket, etc.
    • Audio chunks may contain timestamps

Even the very basic simple plug in a mic to a Linux machine now doesn’t work with Rhasspy as its no longer a standard Linux kernel audio stream its this ‘chunked’ protocol !?

  • wake
    • Wake word detection
    • Inputs fixed-sized chunks of PCM audio
    • Outputs name of detected model, timestamp of audio chunk

So now every wakeword program needs to strip the Wyoming protocol and return back to a standard linux audio stream to process a Wake event and then reassemble the event into a standard audio stream to make it proprietory once more.
Also why it outputs name of detected model, timestamp of audio chunk as what use or function that has upstream I have absolutely no idea.

I could go through each step where its creating work on the same audio stream and have no need or value for the meta data its injecting that standard standard programs do not need, in fact doing so excludes them.
I am not going to bother but in terms of a Dev talk ignoring standard Linux kernel methods for audio and creating a proprietory protocol that only Rhasspy needs seems the opposite of

Event Streams

Standard input/output are byte streams, but they can be easily adapted to event streams that can also carry binary data. This lets us send, for example, chunks of audio to a speech to text program as well as an event to say the stream is finished. All without a broker or a socket!

Each event in the Wyoming protocol is:

  1. A single line of JSON with an object:
  • MUST have a type field with an event type name
  • MAY have a data field with an object that contains event-specific data
  • MAY have a payload_length field with a number > 0
  1. If payload_length is given, exactly that may bytes follows

Example:

{ “type”: “audio-chunk”, “data”: { “rate”: 16000, “width”, “channels”: 1 }, “payload_length”: 2048 } <2048 bytes>

I am reading the above and my head is exploding as the very things it extols as virtues are exclusions of standard linux protocols with a replacement that is more complex, proprietory and as far as I can tell not needed for a simple Voice system to work.

Say we take All without a broker or a socket! and do a wiki on what a broker is.

Wyoming is a broker and its needed at every stage of the pipeline to convert back to normal Linux audio streams that likely the original program now wrapped in the Wyoming stdin/stdout broker pipeline uses anyway.

This means it has to be implemented so that you have to be a dev of the apps whilst many apps are ready an complete and work on standard audio streams, what is being said seems to be paradoxical.

Wyoming seems to be very much a continuation of Hermes audio like control minus the MQTT which I never got anyway as its seems to focus and a load of unnesscary whilst ignoring standard linux process and what is needed for a modern multi room voice server system.
If you look at the new Hass Assist it needs the ASR inference text and absoluely none of what Wyoming/Rhasspy are creating a non standard protocol and workload for as all it needs is the inference text and what is worse in terms of modern multi-rrom systems the protocol lacks the zonal/channel info for Assist to create a simple 2nd level interaction message so its TTS text message can return as audio to the source input.

I keep stating that because it is and it really is simple but for some reason Rhasspy enforces the input/output audio chain to know Rhasspy than Rhasspy work with standard linux audio streams and so becomes complex.
I think I am going to bail on Rhasspy3 as for me it is purely a continuation of what I percieved wrong in 2.5 and likely still continue to play Mic hardware, KWS and generally keep an eye on voice technology.
As an example if you are interested I will sketch out how things should operate so that its not a matter of minimising barriers for programs to talk to a voice server, there simply are no barriers as it uses standard linux streams and protocols.

Example interoperable voice system building blocks
Rhasspy seems to make the assumption that Mics and KWS should be able to talk to Rhasspy so enforcing a Rhasspy protocol because Rhasspy is central.
Rhasspy has always lacked a KWS/Mic server as KWS/mics are simple devices that do a specific job and have no need to know any voice server protocol so that a dev can create a KWS/mic device that is interoperable not just with Rhasspy but various voice applications.

The websocket api GitHub - rhasspy/rhasspy3: An open source voice assistant toolkit for many human languages was a good start and where we do need a simple protcol of “start/stop” and simple quality metric its lacking and strangely embeds a nulll in the binary to singnify an end.
But its not part of a KWS/Mic server that aggregates input to a single stream based on a single zone by its channel and if multiple zones are concurrent it queues one as a file to completed after the 1st is completed or route to another instance as ASR is generally a serial process or can route to another instance.
The KWS/Mic server is missing and Rhasspy forces each mic/kws/process to embed meta-data into a broker protocol because it processes them after reciept than on input and also assumes a single stream where multi-room connections will be processed later.
If you process connections and streams local and remote as a KWS server then each device doesn’t need to know Rhasspy or have any embedded protcol as the KWS server can embed it, but its not even needed if audio is ready to stream, queued and routed.

There is a whole section at that start of the audio processing system that is absent in Rhasspy and it creates a whole load of complexity to do this after the fact.

Any Skill such as Hass Assist needs only to send a text message to TTS but is missing the important KWS server zone and channel to return audio to.
Also skill meta-data that initiates a KWS/Mic stream is also absent so that the system can not simply differentiate between OP command streams and secondary response streams, but that is another example of how easy and simple control should be.

Its that simple but wow, so I am going back to playing with Mics, KWS and voice tech news and refrain from giving an honest opinion andf getting more than the tempory forum ban I got last time for purely expressing opinion.

Bemused :slight_smile: is all I will say but on this topic and Hass Assist has proven how great inference based skills are, but a very simple kws-server protocol seems to be missing and the complex one proposed is likely not needed… :roll_eyes:

1 Like
50

I say it right here in the adapters section:

Using events over standard input/output unfortunately means we cannot talk to most programs directly. Fortunately, small adapters can be written and shared for programs with similar command-line interfaces. The adapter speaks events to Rhasspy, but calls the underlying program according to a common convention like “text in, WAV out”.

Most programs do not need to know anything about the Wyoming protocol, because they already follow a convention covered by an adapter.

You received a temporary ban for being unnecessarily negative and non-constructive with your feedback, not for expressing an opinion.

At this point, though, it honestly does feel like no matter what I put out, you will not like. Which is fine, but it would be much more constructive to criticize something you have actually used and understand.

3 Likes
51

What adaptors? I have to write an adaptor before I can use any program.
Apols Michael if I can not like, but I am looking at an input audio chain of in and out of a protocol via Python that I have to write adaptors and take the hit for load of use and dev time and its not about like its I have absolutely no idea why.
Seriously I am sat here thinking linux has a audio system written in C by much better than me that I can not figure out at all why you would have such mechaninsms.
So no prob I will not use it and maybe concentrate on a simple KWS server that feeds an ASR that feeds Hass Assist and hopefully other inference based skills that might arrive.
Just look at all the documentation length, adaptors and protocols just to get an audio stream to an ASR??!
If that is deemed unnecessarily negative and non-constructive, well to be honest it is just opinion and feedback on your Dev as your code is always great and tidy, but I honestly have no idea why its always seem so bloated especially when we are targetting embedded.

I am just going to stick to hardware mics and look at the websockets interface to create a simple KWS server for audio in and for audio out likely squeezelite or snapcast as also multi-room audio and guess I just don’t need Rhasspy as struggling to work out why I need Wyoming, adapters and the rest just to accomplish that?

52

Fine, maybe come back here when all you talk about would be ready for testing, that is in a github repository with a full documentation to deploy and test it. I already miss it :slight_smile:

1 Like
53

I should really have a go at a esp32-s3 kws due to the Raspberry stock situation, but procratinating over KWS model and taking the plunge with the Espressif IDF.
I was sort of hoping I would be able to use existing but not looking that way.
The multi-room is a no-brainer really as like the ready made multi-room audio (squeezelite.snapcast…) its just a zonal client/server infrastructure that is needed and associate the zones of input/output.
You need a bit of a ‘debounce’ with multiple mics in a zone to cope with latency (wait 100ms for all to reply and pick best stream).
Things are changing so fast with generative models…

So with everything changing so fast, lack of stock, procrastination and likely little if anything I could use I think I am losing interest.
I have been meaning to retsart the KWS work that I ditched, but can not seem to get the enthusiasm, so maybe not :slight_smile:

As for documentation the client/server simplifies operation massively, there are x2 servers, where both just route and queue.

1… KWS Server (Skill Server is a client)
2… Skill Server (KWS Server is a client)

There is no concept of a satelite as non is needed purely they connect and register with a server with a static UID as you setup the system.
When an ‘ear’ connects to the KWS Server its UID signifies the zone its been attributed to a YAML which is queued and routed with that stream session be it actual stream or file.

The Skill Server contains the ASR on on inference text routes this to the best appropriate inference based skill on the predicate of the inference, inference added to Yaml.
If a response is required a Skill will append the Yaml with TTS text and return to Skill Server.
TTS is a skill but input zones/channels are mapped to output and why its a KWS Server client as it forwards the YAML to start a response recording on the the initial Ear of that zone.
The next inference knows this is the response so YAML contains all data to route back.

Client/Server TCP is massively more simple as delivery is guaranteed and the whole inference structure doesn’t need the Timestamps and protocols that a UDP MQTT broadcast network needs that delivery can not be guaranteed or order.

I am a Snapcast fanboy but likely you would modprobe ALSA loopbacks and assign one to each room and just play into it and Snapcast does the rest.

That is it and covers all as a voice server processes voice and skill servers process skills, we just need skills and KWS (Ears) and decide if you are going to have a fast specific domain predicate ASR and forward to task based secondary ASR.
IE Whisper (Conversational ASR) whilst a domain LM Wav2Vec2 could quickly provide for command skills such as Assist or local media libaries of known entities.

What we lack is decent cost effective KWS ‘ears’ with good audio processing and good inference based skills.
Hass Assist is great as its the 1st inference based skill I have seen but for some reason it has an older tech strict word stemmer that needs declared formatted YAML, than say more modern NLP or even LLM (Large Language Model).

Rhasspy is back to front for me as skills and devices do not need to know a Voice Server or convert standard audio streams or files into proprietory protocols that are historically needed because of MQTT.
A Voice server is the really simple part and likely we will start seeing LLM specific domain skills and a ESP32 Guru or Rust/C guru might provide a KWS and far too much fort one person to do all, but I am absolutely certain the way foward is a simple client/server infrastructure and not a broadcast network of embedded strict protocols.

Its the skills that are the complex type as in training an LLM but they are jaw dropping amazing and running locally and now.

As in Introducing LLaMA voice chat! by
Georgi Gerganov

Thanks to Ge Orgi :slight_smile:

54

Rhasspy itself will run anywhere Python can, but most of the voice programs are built for x86_64 and arm64 systems. So, Raspberry Pi 3/4 with 64-bit OS.

There is an HA add-on that includes Whisper and Piper (formally Larynx 2). But you can’t change anything yet.

32-bit ARM support is getting harder and harder, but everything will work on 64-bit ARM.

The satellite code should work on 32-bit ARM if you use porcupine (as well as 64-bit ARM and x86 of course). It’s also possible to run a satellite that’s just gstreamer or a simple websocket client, since wake word detection can happen on the server too.

I appreciate it, though this might be a bit premature. The biggest missing pieces for non-developers are being able to (1) install services via the web GUI, (2) download models as needed, and (3) control which services are automatically started with the HTTP server.

55

Maybe it’s too early but I want to be prepared so I have a question regarding recommended hardware platform for Voice Assistant (VA) in Home Assistant.
If I plan to use VA will I need to upgrade my current HassOS host “Odroid C4” (quad-core Cortex-A55) to some more powerful ARM (Odroid N2, M1?) or even more powerfull (x86 Proxmox) host?
Or the Voice processing can be dedicated to a secondary system (Rpi3) and I can leave my current Hassio system as is?

56

I may have missed but I did not find anything about how this version will differ from 2.7? what are the innovations or major changes.

57

Will openwakeword be also an option along others for wake-word recognition?
I tried it recently and the results are just unbelievably precise compare to other embedded options.

Unfortunately I wasn’t able to make an image with rhasspy 2.5, since it’s based on debian 10 and had a 3.7 python, which in combination with RPi4’s aarch64 architecture wasn’t compatible with that library, so I had to make a workaround and basically run it outside the docker. But it would be such a good option.

I’d be happy to contribute though, if that’s a way to go. Especially after I looked at the structure of the rhasspy3, which is much more clear compare to 2.5, thanks a lot for that :clap:

2 Likes
58

what do you think about GitHub - pyannote/pyannote-audio: Neural building blocks for speaker diarization: speech activity detection, speaker change detection, overlapped speech detection, speaker embedding, maybe add it too? and GitHub - kssteven418/Squeezeformer: [NeurIPS'22] Squeezeformer: An Efficient Transformer for Automatic Speech Recognition

59

Hi
I’m currently a Mycroft (Picroft) user who’s reviewing his approach to voice assistants :slight_smile:

I’ve successfully run through the tutorial (nice - ty, a couple of minor bug reports pending…) and have a few comments/observations (mainly ideas I’ve been turning over for a while)

Firstly I personally think the ‘event to sound’ stuff should be almost broken out - it’s nice to have it all in the demo but conceptually I think the focus of a speech assistant is the input side so I’d tentatively agree with another post that a snapclient based/like approach to delivering TTS would be reasonably sane/light.

So now, thinking about the whole ‘sound to event’ architecture, my main thought is to question why it is so serial (if I’ve understood it correctly)?

My concern is that this has drawbacks in both user-experience latency as each stage of processing seems to need to finish before the next one starts; and it also means that the control of recording is driven by non-dynamic configuration.

May I suggest that adding a buffer-controller (BC) based approach could be interesting.

Something like this:

mic recording is constantly put into per-mic smallish (5s) ring-buffers.

The buffer controller (BC) provides (websocket & unix socket) access to the buffer using Wyoming.

Wake detection is connected to the BC socket and listens.

When wake is detected the BC is told to record properly and now just grows the buffer; the timestamp at the end-of-wake-word is available (speech-start). WD would emit an event with the speech-start timestamp (back to the BC for sending to other connections).

Anyway the buffer now grows and sends this event and timestamp to VAD which requests and reads until it eventually detects silence and tells the controller to stop recording and sends a speech-end event. (So now does the BC needs to be the event broadcaster to send that to ASR or does is this just how it gets an EOF pointer?)

At the same time the BC sends speech-start to any ASR(s) which eventually dispatch via intent recognition (Wyoming again) to an intent. The intent can now send a ‘send-more’ event to ASR which sends a start-recording to the BC which again activates VAD and provides an audio stream to ASR and a text stream on to the intent

Possibilities:

  • More responsive ASR as it doesn’t need to wait for VAD to trigger before processing
  • Multiple ASR could listen to the BC
  • Audio available for other uses (eg debug capture or false-positive wake word training by a “never mind” intent that ask for the wake-word raw data)
  • Almost continuous natural speech since the wakeword algorithm should be able to provide an entry point into the buffer for ASR.
  • non-wakeword triggered recording: an intent says “any more”. eg my shopping skill responds to “We need some [ and …]”… Then (via TTS) says “any more?” and starts listening again rather than me having to re-utter “ We need some ” - this is much more natural.

A lot of the pipelining work still makes sense btw. If I’m getting it right the buffer-controller would provide Wyoming interfaces over websockets, unix sockets or (via a socket shim) stdin

Anyway… just thought there may be some ideas in here that would spark more discussion

60

Feeling a littel dump right now but how can i actually install rhasspy3 addon for homeassistant?
Here GitHub - rhasspy/rhasspy3: An open source voice assistant toolkit for many human languages it states * Install the Rhasspy 3 add-on and links to GitHub - rhasspy/hassio-addons: Add-ons for Home Assistant's Hass.IO but i have already added this to home assistant addon repositories and only see Rhasspy 2.5

Edit:
i had to remove the addon, remove the repository and add it back

62

Hi there,
Not sure to be in the right topic (forgive me if not :roll_eyes: !).
Just find an hugging face repo on an “improve” Whisper :face_with_raised_eyebrow:.

As it’s a bit tech-savvy for me, is it a nice way forward or not applicable or not enough open-source ?
Thanks for sharing though :slight_smile:

63

@synesthesiam do you have a quick update for the Community about rhasspy3 ? Sound very interesting… unfortunately i‘m not a developer :wink:

1 Like
64

Indeed @synesthesiam, didn’t get around to trying to get Rhasspy3 setup over the summer, but coming back around to it was going to start poking into it as another project again. However the code doesn’t seem to be updated (at least in master) at all really for the last 9-10 months. Any progress forward? Where can I find the current development branch to try and setup?

65

@synesthesiam I agree that now would be a good time for an update on Rhasspy 3. I guess that the issue for Rhasspy 3 is that you have been focussed (quite rightly) on your paid employment, which is integrating Rhasspy 3 with HA. I just watched HA Voice Assist chapter 6 video, and it is looking better all the time for those of us using Rhasspy with Home Assistant. Thank you Mike for all your effort.

My understanding is that Rhasspy 3 code is actually working well with Home Assistant. Those of us using Rhasspy with HA should be migrating to HA Voice Assist, and looking for support in the HA Community forum and Discord ?

Rhasspy 3 (and this forum) therefore will become focussed more on those using Rhasspy as a toolkit with other applications ?

I guess that the Rhasspy 3 code is already doing most of what is wanted - but the documentation may be lagging behind ? And given that the toolkit can be used in very diverse ways, the documentation (including examples) will have to be more more thorough … which takes time.

As Drizzt321 mentioned, the rhasspy/rhasspy3 github hasn’t been updated for some time, but the wyoming, wyoming-satellite, wyoming-piper, wyoming-openwakeword, and piper repositories have all been updated recently.
Is wyoming-satellite the current version of Rhasspy 3; or is is specific to Home Assistant ?
If wyoming-satellite is HA specific, do the wyoming modules reference the rhasspy/rhasspy3 modules (and thus is up-to-date) ?

66

Hi there!

As I’m about to migrate my homeautomation system to a newer hardware platform (and recent bullseye), I experienced quite a lot of things to cope around with the old (2.5.11) version; finally “stable” Rhasspy is up and running now (together with mimic3 as tts server). Don’t want to go into details, but as I’m also maintaining the code base for Rhasspy’s integration into FHEM, the question raise up if it’s worth to give the preview version a try.

But Rhasspy3 atm seems to just support HA? And there’s not much to happen the recent time, or did I miss sth.? Basically: Is Rhasspy3 still alive and worth porting the FHEM “adapter” to it? (And if: How to integrate such an external intent handler to the Rhasspy3 ecosystem?)

1 Like
67

Rhasspy 3 is yet another dead open source project.
You can look to other solutions.

68

Any suggestions wrt. to “other solutions”?

Didn’t do much research on that yet, as latest Rhasspy 2 still can be installed (at least in bookworm and bullseye) and is working pretty well with my automation system (FHEM)… Additionally @synesthesiam still is active here in the community, so there remains a little hope for future development on that (or at least update to the installation packets to overcome some known installation and configuration problems for 2.xx version).

69

Hi everyone :wave:
Sorry as always for the delay in updates, etc. I always have way too many things going on :smile:

Rhasspy 3 is not dead, but I need to re-think about its place in the current open source voice ecosystem. My focus has obviously been on Home Assistant, but I purposefully designed the Wyoming protocol for distributed voice processing so no one is locked in to one ecosystem. I’ve been working on adding HTTP APIs to the various Wyoming services too (similar to Rhasspy 2) for additional compatibility.

Rhasspy 2 was an “all-in-one” solution which installed, configured, and trained voice services through a shared web UI. This paradigm started to break down towards the end as I began adding more diverse services with unique training requirements or a complete lack of the ability to train without an expensive GPU.

For Rhasspy 3, I’ve been rethinking the “all-in-one” idea and am considering having it be up to each voice service to do configuration and training. What I really want is what Home Assistant OS does with add-ons: you can do whatever you want in a Docker container, and there’s a common method for discovering, installing, configuring, updating, and starting/stopping each add-on. Something like this could be done with Docker compose, I’m sure, but it sounds very difficult to create and maintain.

In the mean time, my plan is to have a Rhasspy 3 server host multiple pipelines like Home Assistant does today. Each pipeline is made up of Wyoming/HTTP services, and there will be a web UI to create, configure, and test pipelines. For now, I plan to leave the installation and configuration of each voice service up to the user, though the tutorials will show how to set things up with Docker fairly easily.

What does everyone think?

1 Like
70

Dear @Synesthesiam,
I’ve never had a chance to thank you personally for Rhasspy; your work has brought me so much enjoyment (and utility within my implementation) so this seemed to be an opportune time to do so as well as weigh in on Rhasspy 3. I’ve loved voice technology for a long time, the first tech I used was IBM VoiceType (discrete speech!) but Rhasspy/HA combination really lit the fire for me. Learning about Docker, JSON, YAML, & basic Python to make things happen in Rhasspy have been incredibly fun, and have probably consumed more hours of my time than they should :). Regarding Rhasspy 3, I do like the idea of a “componentized” successor for Rhasspy 2. Allowing folks to “mix & match” whichever services they want need would be helpful, and potentially easier to maintain (e.g. provide all components use the “lingua franca” of the Wyoming protocol, each could be updated separately). I do wonder if it would be necessary to have all of them bundled in a single Docker container; I’m perfectly okay with spinning up multiple containers for different components which I could update separately or turn up/down depending on need rather than having a monolithic Docker where the App pulls in the various functionality.
I do have a request (for the future…), if possible, can you have a component/module that supports MQTT? I’ve build a lot of my functionality using MQTT (I really like the simple elegance of the protocol) and I think it would be useful for some of the “light weight” transaction (e.g. a IOT sending a MQTT message).
Again, thanks for everything!
Jeff

1 Like
71

@synesthesiam Thanks for the update … great information on your future direction,

I am puzzled by this. By spinning off HA Voice Assist you have set Rhasspy 3 up as the general toolbox (for use with non-Home Assistant systems) that i thought you had always envisaged for Rhasspy.

I think people want to know the current rhasspy3 status.
Is rhasspy 3 usable now without HA ? If so, which github repository should people use ? The rhasspy/rhasspy3 github hasn’t been updated for some time, but the wyoming, wyoming-satellite, wyoming-piper, wyoming-openwakeword, and piper repositories have all been updated recently.

I understand that you’re up to your eyeballs in the detailed code; and that user-level documentation is a time consuming chore … but could you please spend some time on fleshing out the current documentation. Hopefully a couple of rhasspy users with non-HA projects will document their experience with Rhasspy3 and post here.

72

Okay, you have been looking for a different way to confirm that Rhasspy3 is a project that in its current state makes no sense, and is in fact dead. But that’s okay. No need to waste time on useless projects (time is money).

Yesterday I read some of your responses to some users on Reddit, where you recommended “extended_openai_conversation.” Today, since you work for Home Assistant let me offer you some ideas for the future. The first: create a “micro AI” that allows novice users to make a request for the creation of scripts or automations, in human language. In short, a micro AI that is able to make Home Assistant “user friendly,” even using voice. The user says, “I want the light in the garden to turn on at eight o’clock at night and turn off at seven in the morning,” and the AI, after getting confirmation creates the automation.

Second point: is it possible that it didn’t occur to any of the developers to turn the Home Assistant companion app into a media player? This just boggles my mind, I really don’t understand.

73

First: I’m really glad to hear, you still having (some) focus on your great project Rhasspy!!! Also big thanks for what’s been achieved by now from my side!

From the technical prospective, I’m not able to give any solid feedback to your ideas, and I don’t have (and don’t neither need or want to get) any experience with HomeAssistant. So I’m also puzzled in what’s the essence of the “multiple pipelines” idea and so on.

Based on that background, I try to add some user focused thoughts here:

  • Doing the Rhasspy integration (at least) in FHEM as automation system already is a big challenge for quiet a lot of users. I’m just talking about things like get names, categories, colour settings etc. (all kind of “labels”) bridged from the home automation system to the STT and intent recognition. So most of them really appreciated Rhasspy beeing a “one stop” solution. Breaking that up into more pieces might raise complexity to an even higher level, so I really doubt this to be “attractive” for a (imo) really huge part of the people interested in home automation!

  • Beside the fact, I always disliked the idea to use MQTT as transport level for audio, MQTT (or the hermes protocol?) was a relatively “easy to understand” and transparent transport layer. So wrt. to user support, advising to have a look to mosquitto_sub’s output was always a helpful hint for everybody. Imo, independent from the used automation system, there should be any comparable (debugging) option offered.

  • Why (for the text based part) a new protocol at all? After having a look into the (rudimentary) documentation, Wyoming seem to be a complete new playground, which was really disapointing leaving me wondering, if I’d have to reinvent the wheel for the FHEM side as well. That’s not a very attractive perspective, to be honest.

  • Don’t know exactly why, but personally, I always preferred direct installations, and avoid using docker whenever possible. Might be irrational, but that’s how it is, and the recent try to install Rhasspy 2 dockerized ended up in a mess, so I finally once more went back to the “classic” way and used the (patched) deb. Most likely, I’m not the only one with that kind of experiences and feelings…

  • By now, we should avoid people getting frustrated by Rhasspy. Imo, first step should be to keep the “stable” tree (Rhasspy 2) usable for those people using it already. So please first provide (installable) deb versions for at least recent Debian distro’s and/or docker images with working mosquitto settings (authorisation disabled for internal server).

Most likely, those fears are just irrational and my personal problems where just “home-made”, so once more: first of all I’m still glad Rhasspy still is alive!

Looking forward for the things to come!

2 Likes
74

You’re welcome! I’m very happy to hear that Rhasspy has worked out for you :slightly_smiling_face:

As I play around more with ESPHome, I’m really liking its architecture (which itself is modeled on early Home Assistant). Components are separated into “domains”, and a “platform” groups components based on something like a protocol. So there may be mic domain that has a udp platform and thus a UDP microphone component.

No, I would not say it’s usable at this time. The rhasspy3 repo represents an early vision of what became Wyoming, and I think I can do better now with the perspective gained.

One of the use cases for Rhasspy 2 was people just needing a few tools from the toolbox. They got this by spinning up a Rhasspy server and only enabling some of the services. Then, the Rhasspy API provided access to these services over HTTP and Websocket (the services themselves used MQTT).

Nowadays, I expect people to be able to spin up Docker containers for each service they want like wyoming-piper, wyoming-openwakeword, etc. I am slowly adding HTTP APIs to these services, and will eventually add MQTT/Hermes support. So something like Rhasspy would not be needed if you just want services à la carte.

Where I do see value for Rhasspy 3 is:

  • Managing all of these services from a single configuration (likely using docker-compose ultimately)
  • Coordinating pipelines across services and exposing high-level APIs for running pipelines and talking to satellites
  • Experimenting with pipeline concepts that are too complex for Home Assistant (multiple levels of fallback, multi-language, etc.)

Nope, that’s not at all what I said :slightly_smiling_face:

We’ve discussed this idea, and in fact it has been done before (HA partnered with a research group that later shut down their servers). I do believe this is possible with an LLM, likely even a “small” LLM with the proper training set.

I don’t know if voice is the best place for this, though. I would prefer to file this under “natural language interaction” instead – for example, you type the automation you want in natural language, and HA presents a filled-out automation for you to confirm/edit.

Wouldn’t it make more sense to add support for an existing media player app to HA?

This is a good point, and I don’t have a great solution for it. Hassil is (in my mind) a spiritual successor to the Rhasspy template language + slot lists. I do think that adding just a bit more complexity to the lists would help out – specifically, parent/child relationships between items (like a device being in an area). Maybe that would be enough complexity for most home automation systems.

I will have HTTP/Websocket/MQTT options for the services as well, so there shouldn’t be any need to reinvent stuff.

I went with a new protocol because of this logic:

  1. MQTT doesn’t do peer-to-peer
  2. HTTP is pretty lightweight, but you have to base64 encode binary data (audio) and two-way communication is a pain
  3. Websockets do most of what I want, but (1) they are complicated enough to need a library and (2) you can’t attach metadata to binary messages

So I thought, here’s all I want: TCP with JSON headers and sometimes some binary data (audio). And thus Wyoming. Initially, the header just had the event type/data and the length of binary data (if any) in a single line of JSON. Then it turned out that Python has a hard-coded limit on the length of lines (WTF), so I moved the event data to a separate section.

Wyoming works over TCP, but it also works great over Unix domain sockets and even standard input/output! So unlike HTTP, Websockets, and MQTT, you can communicate with Wyoming services without even bothering to open a port.

I’m worried about the amount of effort this would actually entail. I can only imagine how many Python libraries in Rhasspy no longer have working versions for Python 3.11+. With my limited time, I’m always struggling to decide between time spent on creating new things (like Rhasspy 3) and time spent supporting older things :slightly_frowning_face:

As I live an breathe the open source world more and more, I have come to discover that I’m not very good at delegating. This is something I’m continually impressed with in the Home Assistant community, and I feel I’m lacking the necessary skills to make it work here :confused:

2 Likes
75

Okay, in this last answer you made sense of Rhasspy3.

I’ll try to tell you my use case: I use Rhsspy mobile on a tablet in the living room and on my smartphone, both connected via MQTT to rhasspy2 which only takes care of managing the intents and sending them to Home Assistant. On rhasspy I use my favorite TTS service, with its own defined voice, and, for example, with a home assistant automation I can send a wav file (an alert sound) before it tells me what time it is. Another example: every morning another automation sends a signature tune that precedes an almanac, with calendar events, weather, news, etc. Well, I would be very happy if I could use the Home Assistant companion app to manage this type of automation, especially on my smartphone, but to date, based on my knowledge, it’s not possible because the media player service is missing.

If the Micro AI project goes through, for training ask the community if they are willing to share automations. i would be proud. I think most will accept.

1 Like
76

Thanks a lot for your elaborate answer!

Imo the essential point for interaction between any home automation system and the rest of services making part of sth. we may call the “wyoming ecosystem” really is some kind of compability in the intent recognition results. The JSON blobs in the hermes protocol are not very easy to understand and analyze, but contain (in case the intent recognition system is configured appropriately) all stuff needed to execute (in most cases) the action wanted by the user.

The “strenght” of the FHEM solution imo is: it automatically filled all the slots needed for most use cases, once you put any “entity” under Rhasspy control. E.g. adding new lights is just a piece of cake: Activate Rhassyp service for them and add a (speakable, but not necessarily unique) name for each of them…
Training etc. is then initiated automatically, no need to edit any yaml file (this is what (HA orientated?) hassil at first sight looks like, but I may be wrong) or code any parent/child relationships on the intent recognition side. Obviously, this also has the disadvantage of beeing a little to open with intent recognition results. But this way round, it’s very easy to handle, especially for users on the learning path…

Wrt. to updates for 2.5: Imo, atm it’s just two tweaks needed to get that done. (libgfortran version in deb file, MQTT without user/password in docker, see Bullseye refuses to install). So the efforts to spend on that are relatively moderate… But I clearly understand your focussing on developing for wyoming!

77

@synesthesiam thanks for all your work.

From my point of view, I like the idea of using docker with docker compose. If the documentation is good, an example of each service with all docker-compose configuration is pretty easy. Just copy and paste and adjust the directories and the services you want to use. In addition, from maintenance perspective, it is very easy, you don’t need to worried about new library versions. It is a nice solution, with a very small overhead.

In my case, I finally give up to use Rhasppy3 and I’m using Home Assistant with the Wyoming docker addons. In case the you are using a Home Assistant docker installation, that part is not very well documented.

The only thing I miss is the program that we have in Rhasspy 2.5 to extract all HA entities for Vosk. In my case, faster whisper didn’t work so well with spanish. I also tried some other models but in general the quality is far from the english version.

Finally, let me know if you are interested in a new tutorial with HA and Wyoming addons all in docker.

78

Part of the problem is the assumption that the Sota WER levels it posts is the same for all language models.
Its not as the WER rockets, as Spanish is the best supported in the large model (OpenAI Graph), but drops to 4th in tiny, where all languages post pretty bad WER.
The 0.37 WER of the Tiny Spanish model is a mile away from the Sota scores, but so are all languages, not just Spanish and why Whisper, as we had and there are much better models and much smaller, for the general hardware level Rhasspy is aimed at.

Speechly did a good review of Whisper Language vs Model Size and for the models supplied, the news for a long time hasn’t been good.

Also Whisper has been trained on standard mic inputs and works with moderate RIR and noise but use DSP to remove and WER gets much worse unless you fine train Whisper.
So you end up having to finetrain absolutely huge monstrous sized models and so far its DiY as no training framework has been given.

As there are many languages that have very bad WER even in the large model and as this guy finds out language training a fully opensource ASR framework such as Speechbrain creates a model x10 smaller than Whisper Large models whilst being nearly x2 better with WER & CER.

I have not used Vosk but I am having a guess VOSK has a LM (Language Model) that are super small accompanying vector databases of the text words of the language you want to use (simplistic).
This is why I really like what they have done @ wenet as confining a ASR to the words you use for ASR where for control such as HA its a small subset can massively increase accuracy.
I will let Wenet explain why as they do so far more eloquently than me.
https://wenet.org.cn/wenet/lm.html
Also here to add context biasing to add weight to the n-grams and further increase accuracy.
https://wenet.org.cn/wenet/context.html
Basically you can make vastly smaller (faster) more accurate models by using old tech but being domain specific. In a domain specific area such as HA this makes total sense. Also this could well extend to multi-modal ASR based on predicate detection or on-the-fly LM creation/ domain LM loading for smart devices.

Containers from a Dev perspective make 110% sense because of the nature of Python and importing and adopting so much opensource by others its creates a Dependency nightmare.
Library handling with Python is a tad raw with many just putting absolute versions to be used, but also the libs of the distro release often change.
When you grab the plethora of modules to complete a working ASR system you can soon enter a Dependency Hell where no matter what you do when you change or update Libs that creates new lib dependency probs, that become cyclic.

This is why Docker and containers are so great as you can split modules ‘contain them’ into isolated contained modules connected by declared net or /dev devices such as file system. Have different Python libs, different releases and even Distro’s.
It allows you to put together a whole range of software that would never work in a bare metal single enviroment due to dependency hell as what one module will run on, another will not.

Docker is a great tool, but also an image with all the docker containers ready running would also be great for the zero dev/config install users.

Why Whisper is a good question as is why does Rhasspy hardcode ASR?
With the rapidly changing arena of voice recognition and huge array of language and functional domains it doesn’t make sense to hard code.

We need a pre prepped ASR container that has a transport mechanism to accept audio speech binary and also the metadata of where it came from so that it can be passed on.
ASR does not need anything system specific and with very little modification any ASR could be used.

ZonalMic->WirelessZonalMicServer/QueueRouter->ASR->SkillServer->TTS->WirelessZonalAudioServer.

The speech recognition process is a simple serial chain and all elements can be agnostic of any system and just need the destination to pass on binary audio data or text metadata or both.
It screams individual contained process (containers) via network, but also ALSA and devices such as file system can be declared via docker.
A container can play into a shared Alsa-loopback and another container can be listening on the other side recording the mic via low latency C optimised kernel code of the Advanced Linux Sound Architecture - Wikipedia
Also can also have file type objects or even TCP sockets can be used via netcat.

Should really be able to use any ASR and route and queue to multiple instances as a web dashboard is still a singular control and config interface and be even easier than it currently is.