2023 - Year of Voice

Announcements
1

Paulus from Home Assistant has just blogged about Rhasspy regarding the “Year of Voice” in 2023 :partying_face:

We’ve started with intent recognition, since even with perfect audio processing and speech recognition, you still need to do something with the text! We also wanted to jump start crowd sourcing sentence templates for the IoT domain in as many languages as possible (similar to @Jarvy’s Home Intent).

In parallel, I’m laying the groundwork for the next version of Rhasspy (v3). This has been difficult, since there are almost as many use cases for Rhasspy as there are users. In general, I see Rhasspy’s goals as:

  • Providing a simple, uniform API to local open source voice tools
  • Enabling users to create custom voice assistants with those tools

Architecture-wise, I also need to make Rhasspy more amenable to contributions. Adding a new service is pretty difficult right now, and I think that should be one of the easiest things to do.

Here are some “what if” thoughts I’ve been having about a future version of Rhasspy. I’m curious about everyone’s thoughts :slight_smile:

  • What if Rhasspy didn’t come with a web UI, just HTTP/Websocket/etc. APIs?
  • What if Rhasspy had no “plugins”, but only ever called external programs?
  • What if training in Rhasspy was separated into its own standalone application?
9 Likes
2

Such great news, congrats! And great timing with Pi2ZeroW’s becoming available again in 2023 :wink:

What if Rhasspy didn’t come with a web UI, just HTTP/Websocket/etc. APIs?

You mean, making it harder to use and setup? :smiley: Now splitting the UI off into its own project might make sense, but there needs to be something to do all the settings in one somewhat coherent UI.

What if training in Rhasspy was separated into its own standalone application?

I think that makes sense.

3 Likes
3

Configuration via config files would be ok for me, the web UI isn’t strictly necessary and sometimes even makes things more difficult.
However, the dropdown boxes showing all available options are really the killer feature of Rhasspy.
Take for example Mycroft, if you really want local STT you have to first find the correct documentaion and then start fiddling around, to the point where the claim of being “privacy-focused” is borderline fraudulent - if you really want privacy then prepare for a night of reading external documentation and fiddling with the command line.

Sure, isn’t it like that already? But installation, model download, etc. of those programs shouldn’t be less automatic than it is now. I’m willing to start a Docker container each for Porcupine, Kaldi, etc. but I’d prefer not to fiddle around with pip and Python dependency version conflicts and finding some models in the correct format.

Do you mean the “training” that happens after you change sentences.ini or training a whole new wakeword/language/voice?
Calling a command line tool after changing sentences.ini isn’t too hard. I’ll probably set up a watcher of some kind that runs it whenever sentences.ini changes. Make sure it doesn’t corrupt everything if it is run twice in parallel.
Training new languages/voices is already an external process, and one that we mere mortals aren’t supposed to do anyway, isn’t it?

1 Like
4

I agree, but I think (to start) this could also be a command-line “wizard” that asks you a few questions and then configures everything. It could even be aware of the hardware and suggest changes for better performance.

As @cwagner said, I may want to split the web UI off into its own project. Historically, it’s been one of the hardest parts for me to design and maintain since I’m not really a web developer. I’d much rather expose everything over a nice web API and have someone with more skills make it nice :slight_smile:

Most things in Rhasspy right now are wrapped with a Python library/service that talks over MQTT to the core. My idea is the “external command” options for different services would become the model for all services.

Installation and model downloading definitely still needs to automatic :+1:

Mostly the changes to sentences.ini, but I think it should eventually include wake word training too. One idea is to have a training server that Rhasspy base stations and satellites can download updated STT models, etc. from.

5

What if Rhasspy didn’t come with a web UI, just HTTP/Websocket/etc. APIs?

I think this really depends on who you want to use Rhasspy. By getting rid of the UI, it’s sort’ve makes Rhasspy more for the developer than the hobbyist. It also begs the question if you want it to be used for integrating with other software or something that can run more standalone.

What if Rhasspy had no “plugins”, but only ever called external programs?

This would really decrease the barrier to adding new plugins! Just call out to the host system with the right commandline options. Rhasspy just needs to keep track of how to call it and you’re good to go.

I think to aid in your frontend development, you could make a “config” file (or even a pydantic model) that define the options and types for a plugin and then just generate the fields in the associated dropdown. That’s how some of the UI in Home Intent is rendered. (You can see the Home Assistant options here that get auto-rendered into the full UI component - pic is slightly out of date). It was intended for people to be able to define their own plugin settings and just have the UI get built automatically.

I’ve done it a few times, and as long as you don’t need super integrated forms, it can work well, and really aids in speed of development. Also, I set it up to be overridable - so for example the HA script control is a custom UI component, while the rest of HA settings are auto generated - but it all lives on the same page.

What if training in Rhasspy was separated into its own standalone application?

This could prove to be really powerful. Folks might have a more powerful computer at the ready to do training on and then run HA or Rhasspy on a lower-powered Pi. If training was separate, you could offload it to more powerful machines (Rhasspy training as a service?) and share the models back. Right now Rhasspy model generation (with the “recommended” plugins) can run fairly quick on a Pi, but I’ve never really pushed it to see when it tips over. In theory, better and more computationally complex models could be used as long as the output runs well on a Pi (or whatever we’re using)

1 Like
6

Michael my quick look at Paulus’ blog post and your post here leaves me confused and disappointed.

I had assumed Rhasspy Junior was intended as a user-friendly interface integrated into HA; which uses Rhasspy 2.5 “under the hood”. I gave Rhasspy Junior a lot of thought a month ago, and came to the conclusion that Junior does not need to be a big deal – basically a HA integration which is web interface calling current Rhasspy APIs; with only 2 areas that require much development effort. But Junior doesn’t work for me, and seems to have been abandoned :frowning:

You have already spent 6 weeks on Rhasspy v3, so you obviously have your own ideas and plans which we can only guess at. Your comments and questions above sound as though you are throwing Rhasspy 2.5 out the window to make a new framework for system integrators. Certainly there are improvements which can be made to 2.5; but it seems to me already an excellent extensible modular platform which achieves your stated developer-oriented objectives. Is there anything actually WRONG with rhasspy 2.5 ?

From where I stand Home Assistant/Rhasspy only needs:
• user-friendly perspective (full Rhasspy 2.5 is under the hood if/when users want more control)
• HA Rhasspy integration to have a user interface
• HA Rhasspy integration to scan hardware and create default sentences.ini
• cheap satellites with decent hardware (ESP32 based ?) and easy to install.

I get that you’re a developer; I am (or was) too; focussed on the technicalities and making a technically superior system. I get that Paulus, Frenck etc at Nabu Casa also have the same mindset. I know Home Assistant is a project by developers for other developers, and has a tradition of being user-UNfriendly to new users without the skills and expertise … but I believe that implementing and using Rhasspy should be the easiest thing to do because that’s what more and more people will be trying to do.

What level of technical expertise is required to use (drive) a car ? Does one have to be a Mechanic ? A Mechanical and Chemical Engineer ?

Can a modular privacy-focussed local Home Automation system be useful to non-engineers ? YOU BET !
I get that it is a waste for developers to get bogged down writing user-friendly documentation and answering the same basic questions of the support forum, But there are others with appropriate skills who would love to be allowed to help.

As for your questions …

It doesn’t make any difference to me how it works behind the scenes. I want it to work and be easy to learn and use.

Sounds like a plan. My coding and web development skills are about 15 years rusty though. If you want my help.

1 Like
7

Just please take other languages really into consideration. I’ve been trying to set up mycroft with Hungarian before, and the sentence/template syntax was not just PITA but eventually I just gave up my translation efforts. Hungarian is a type of language which uses endings where the vowels depend on the word, so it causes problems both ways (commands and responses).

I’m really looking forward to this (as the wife-approval-factor for non-English speakers is heavily depending on such a feature), but until I see how the templating would work I’ll have to remain sceptical.

8

110% behind this direction and for me been a no-brainer for some time.

This will make things far more manageable and the simpler modules are just building blocks for what is essentially the serial chain of voice modules.
This should of always been decoupled from skill servers and all that is needed is a skill router that allows for the simple or the highly complex as you merely add more skill servers without need to maintain or understand the controls and methods of a skill but just pass inference.

A voice system is merely a set of applications / containers / instances that queue and pass to the next module in what is essentially a serial queue.
The less that is embedded into rhasppy means a bigger choice of implementation that is also more scalable.
The metadata needs for a voice system are extremely simple and that simplicity creates a building block system where complexity is choice.

It will be more manageable, offer more modules, be more scaleable and if it done right we could start to see plug & play linux inference based skill servers that can gather bigger herds because they are interoperable and not limited to a single system.

Its as simple as queue → routes that connect to the next stage that just advertises if busy or free.

What you have posted is Intents for Home Assistant and there is absolutely no need in a voice system as that should happen in a HA skill server that is routed and passed an inference?

9

  • What if Rhasspy didn’t come with a web UI, just HTTP/Websocket/etc. APIs?
    I’m a little confused, too. I think it would be very important to stay on an easy GUI. I can understand, that it’s not your point to focus on, but it’s an important entry point for all new users.
    It’s also useful if you just want to change a small thing, without remote connection. Or a computer without the right setting.

  • What if Rhasspy had no “plugins”, but only ever called external programs?
    I’m not sure if i have an opinion to that point.

  • What if training in Rhasspy was separated into its own standalone application?
    That would be nice, if we can train our models on more powerful processors with the ability of sharing.
    Would be also cool to train only sentence-files for a specific skill, but i think thats not the way it works. Or a Server to expand the ability of understanding.

2 Likes
10

There isn’t really enough info to go on but its such a radical change it doesn’t mean necessarily there will not be a webui even if how currently implemented massively insecure.

There is a problem with the current infrastructure of an all-in-one in what is not just the fastest evolving tech scene its one that is evolving at unprecedented speed.
Already much of what is contained in Rhasspy is obsolete where better open source SOTA boasting models exist freely that are aimed at various platforms from mobile to GPU.
Then we have hardware that in this scene is seeing almost as fast rapid evolution from Apple < 7watt idle RTX2080ti + ML perf to RK3588, NPU accelerators and problems with the Pi supply chain.

The current choice of all-in-one means it gives a few choices of certain modules and elsewhere specific modules and a protocol that is specifically rhasspy.
This means the current system is relatively locked in to a very narrow spectrum that also provide 100% support needs by a small (singular) dev team.

The OP that has questioned current infrastructure has been posed has been well overdue for sometime and 2023 and technology in general is adopting voice methods at a fast pace and the current all-in-one is just a huge constriction to choice, scalability and security.
The current voice scene is so fast moving that current modules are already relegated to a toy base.

Likely there will be a HaSkillServer but the current system and protocol is applied to all modules and is massively over complex as a voice system is not a control system and currently there are huge swaithes of control protocol on modules without need purely because they are part of an all-one.

The training of rhasspy currently only works due to low command volume and relatively unique phonetic collections as the ASR and NLU methods are quite old and even with fairly modest additions of ‘subject’ and ‘predicate’ accuracy will plummet.
It works on low volume predicates like ‘turn’ and subjects such as ‘light’ but a common skill such as a audio server with a modest library could flood an all-one-one with subjects and decimate how the current system garners accuracy.

If Rhasspy and Hass have any ambitions to be more than a toy system it needs a complete rethink in terms of voice control and like CISC vs RISC complexity can be built by reusing simple building block modules that scale.
A model doesn’t get better when you train it on a more powerful processor it is locked because a model is what it is and you can just train it faster and currently we are not really training a model just reorganising phonetic catchment.
The models we use are part of an all-one-one that has a hardware specific of raspberry pi and that is why we have the models we have which is also hugely restrictive.
It doesn’t get better with better hardware because we have specific models aimed at specific hardware.
Accuracy can be maintained or even increased by partitioning into predicate and subject domains whilst an all-in-one at any level will do the opposite and why the current infrastructure was and is deeply flawed.

But your worries are also misplaced because we never needed a front-end voice complexity that we currently have.
A very simple simple zonal, channel based system of KWS->KWS/Audio processor->ASR->Skill router->TTS is all we need and its a very simple serial chain.
The complexity under the hood to create a working voice system was never needed as it confused control with voice and partitioning this should give choice of hardware, model, scale and complexity and reuse of software from larger herds will reduce maintenance and increase support availability than pointlessly refactoring code to a smaller pool and embedding system specifics.

1 Like
11

@donburch Hopefully I can clear up some confusion :slight_smile:

I’m not saying Rhasspy will be dropping the web UI, just that it should be optional. Like Hermes/MQTT, having so much baked into Rhasspy’s core has made it difficult keep up with the pace of change in the voice space (as @rolyan_trauts mentioned).

Regarding Rhasspy 2.5 vs 3.0, I believe for many users that the internal workings are less important than their sentences, slots, and profile settings. I will do my best not to break things unnecessarily, but it may take me a while.

I agree! As we’ve talked about with Rhasspy Junior, I think it’s possible to layer a user-friendly interface on top of something that more advanced users also enjoy. My plan is (loosely):

  • Voice services as regular programs that can still be used independently of Rhasspy
  • Small HTTP/websocket servers that wrap the voice services for satellites
  • Rhasspy’s core, which configures and coordinates the voice services into voice loops (wake → spech to text → etc.)
  • Web UI and other protocols like Hermes on top of the core

One feature the new parser has is that you can embed template pieces into words. In English, for example, you can have turn on the light[s] for both “light” and “lights”.

This helps with matching, but responses are usually more difficult. I’d be very interested to hear about what sorts of information needs to be tracked for Hungarian (gender, case, etc.). Please PM me or reply here :slight_smile:

In the year I was at Mycroft, things changed so much! The best idea I’ve had is to lower the barrier to entry for adding a service to Rhasspy. Something as simple as: if your program takes a WAV file and returns text, you can be a speech to text service. No Python, no MQTT, just a program with arguments, standard in, and standard out.

But I do want there to be an “easy button” for users, which selects the programs based on some constraints (Pi 4 vs. GPU) and installs them.

And a logo that doesn’t look like it was drawn by a programmer would be nice :grin:

12

If you look at some of the models that BigAI are doing such as GPT3/ChatGPT or Whisper things are moving at unprecedented speed.
You could do something extremely simple by sharing a host folder that is the output of one container and the input of another and a simple Inotify folder watcher to run a command.
The reciprocal could happen to state that has been cleared as an ‘I am free’.
But basically inotify-simple · PyPI and whatever is the run command.
My preference would be Unix sockets as they can be both file and net based and the same inter-process queue-bridge could be used at each step in the chain. As a file socket would act the same as above but with web based you can have multiple instances to scale to needs.
The only conf would be a filename or host:port for the chain to connect to next.

There are SOTA models now that if you wish and have intent on buying the hardware Large models such as Whisper, Hi Fidelity TTS and GPT style NLU is a valid option as selecting much lesser models to run on PI.
So I don’t think especially with HA that you can provide specifics just the queue/bridging models to link them as if you provide for one you exclude another or have to provide all.

If you take Whisper the install is
pip install git+https://github.com/openai/whisper.git
It uses ffmpeg sudo apt install ffmpeg
It runs via whisper audio.flac --model medium

It really doesn’t need a web page to be setup… Its support is on its webpage and its herd is much larger than rhasspy with multiple how-to’s and alternative refactored code.

13

Michael, you have indicated that web development isn’t your thing … understood, and i agree that your effort is much better spent on the technicalities (what I think of as “the back end”). So I am seriously considering giving the Junior UI a go myself.

I am particularly suspicious of things like wi-fi which are sold as “it just works” like magic - because invariably they dont.

So separate, but definitely not optional - especially for new users. People need to check that their audio devices are working, setup friendly names for HA devices, and check/edit the values for the arguments in intents, and see error messages.

EDIT: body of post moved to a new topic: Home Assistant Rhasspy Integration GUI

Well, not by me then either :wink:

1 Like
14

2023 sounds amazing !

About web UI, if there is a good web api, UI will follow. There is a big community, someone (me ?) will develop a ui if there is a web api.

Manage it with command line would be a really great plus.

2 Likes
15

Another thought …

I appreciate that Rhasspy Satellite is a fairly recent concept which required a significant refactoring not so many versions ago … and so at the time it was considered an advanced option … but has it now proved itself as the best logical approach for Rhasspy going forward ?

Experience has shown that Rhasspy satellites use only audio input, wake word detection, MQTT and audio output modules. By packaging just these modules (yes, I strongly believe it should still be modular) the overhead is reduced.
Would this be a reasonable subset to implement on a cheap ESPHome platform ? Have you had discussions with Nabu Casa’s ESPhome team about audio options ?

How many users have only an all-on-one Rhasspy ? And in these cases would it be reasonable to run separate instances of Rhasspy Base and Rhasspy Satellite on the same machine ? To run Base it must have reasonable CPU, so would the extra overhead be significant ?

So… am I really suggesting splitting Rhasspy core into 3 or 4 separate but closely tied projects - Rhasspy Satellite, Rhasspy Base, Rhasspy GUI, and Rhasspy training ?

16

Rhasspy satelite is an absolute terrible bloat and a ridiculous idea akin to making a module for a Rhasspy Keyboard for input that uses a net based MQTT network to receive its key strokes.

We have always been missing a module which is a KWS server / audio processor that sits and queues KWS to an ASR and also contains further filters, VAD or AEC if that is how you wish to setup your initial audio stream.
Rhasspy talks to KWS server / audio processor and a KWS server contains modules that likely the only preferential constraint is that a single zone (room) contains the same model of KWS so that argmax is comparative but even that is not essential.
KWS are just ears that are extremely simple input devices that are set up as channels in a zone for input audio that simply mirror the same system of many of the current wireless audio systems available.
Its a very simple premise but audio in on a zone provides audio out on that zone…
It has a minimal number of commands which is not much from start and stop and it doesn’t even have a pixel ring as a pixel ring is a standalone Ha device where a zone may only have a single shared pixel ring whilst KWS might even be hidden, whilst a pixel ring could be prominent and central.

There is no such thing as a Rhasspy satellite as all was ever needed was wireless audio and wireless KWS in a simple zonal system.
We do need a Rhasspy KWS server just as RaspiAudio SqueezeLite has LMS server to cordinate or Snapcast, Airplay or even Sonos (Not that I know much of that system).

Or MQTT Rhasspy keyboards it is…

A KWS will stream to a KWS server that may filter and apply Rhasppy metadata of the zone and channel of origin so that TTS output is a simple mapping to the same, where is purely a bridge so any KWS device can work with Rhasspy.
Streaming from that point is a strange one as all the latest and best ASR uses quite long CTC and uses a mixture on phonetics and sentence context to make highly accurate results as does say OpenAi’s Whisper and actually trying to stream to such models it causes a hike in load and lowers accuracy as often the context width is reduced and from playing the latency is not all that much different.

If you are going to copy consumer ewaste from the likes of Google & Amazon where each unit is this all-in-one then a streaming mode of older smaller models because that is all will fit and run then maybe streaming mode is a thing.

If you are going to have a modern multi zone Sota voice system you would have a single brain fed by distributed KWS and models would not be streaming to garner context but run far faster than realtime so latency of return is not noticeable but also so they don’t lag on multiple requests.
You only have to get to 2/3 zones and the investment cost of a central well powered single brain starts to become more cost effective as the only addition is KWS ears and Audio out cost can be discounted because it is already encompassed as that rooms wireless audio system. If you went for a PI4 with constrained models the 2nd only needs a Pi02W for audio in & audio whilst processing happens on the 1st and here is where argmax comes in as the kws-server could pick the best stream or a preferential default.

You can still put a centralised system KWS & Audio in a box and use it as an all-in-one but the all-in-one peer-2-peer type control network of Rhasspy satelite is an absolute thunderclart of unnecessary and complexity as why are we copying Google & Amazon when there are clearly better less ewaste infrastructures that can be easily accomplished where opensource can excel.
Its even a copy of a single enclosure but even Google & amazon worked out client server is the most efficient way and that should of been copied as a home server not a single box.

KWS are generic devices that just need a ‘driver module’ installed in the KWS server make a brand of one yourself by all means but the are just an auto broadcast on KW mic with start and stop commands and literally that is all that is needed.
Voice commands are highly sporadic and voice system that spend much time idle its absolutely text book centralised server and for some reason we have gone peer2-peer and lost all the advantages of cost and load that can provide via a single home server where the only clients needed are audio in and out and absolutely kick the ass of Google & Amazon.

17

From this comment, it honestly doesn’t seem like you really used Rhasspy satellites much. We all want things to be improved, but there’s no need to be so negative about something a lot of people here worked hard on. Especially when many of the complaints were addressed ages ago.

Most satellites used an internal MQTT broker with local KWS and VAD, and just did HTTP calls out to the base station for speech to text, etc. with their siteId (which could contain a “zone”). So no, “keystrokes” were not going over the network. And the satellite didn’t even have to be running anything related to Rhasspy, as long as it could HTTP POST some WAV data.

Again, there is a lot of room for improvement here. Streaming raw audio from satellites over MQTT/UDP is obviously not going to scale with many satellites. And setting up satellites in Rhasspy is unnecessarily complex since it was bolted on later, rather than part of the original design.

As @donburch said, a lot of Rhasspy users are probably using it in base station/satellite mode, so this needs to be at the forefront for designing v3. And I absolutely agree that “design” here should not entail Rhasspy-flavored versions of already existing standards!

This is what I’m thinking, though “Rhasspy Satellite” could just be a configuration in the base station relating an existing streaming audio service to a zone (as @rolyan_trauts has alluded to).

Yes, I’ve talked to Jesse (the ESPHome maintainer) about this some. Paulus has a contact over at Espressif, so I think the plan would be to get their audio framework involved. Espressif has a number of two mic boards based on the ESP32 that could form the basis of a fairly cheap satellite (that dev board is $20 on mouser). I don’t know what would be involved with getting ESPHome onto it, and if it would be possible to still do local KWS and AEC.

1 Like
18

You know very well from multiple comments and from the very start of dev on the ‘satellite’ I was totally opposed to the bloat and complete lack of need for it.
I can not help that fact people spent a lot of wasted time developing something without functional need and my objection has always been the unnecessary was developed and still is unnecessary whilst a crucial part of audio processing has always been missing.
The satellite mechanism is completely pointless and is just wasted load on what a satellite needs as its purely audio-in & out and its not my fault the dev continued whilst I was ignored.

Non of the complaints where ever addressed and there is a constant stream of confusion in the forum history on how to handle very simple multple KWS zonal systems.

Yes and it has never been fixed and I have repeatedly posted for a long time how simple the fix is and you just contradicted yourself in the next sentence, its not a fix its a badly fitted bandage.
There is no value or IP to what has been developed the ‘satellite’ dev veered off at a acute and complex direction to the detriment of the simple addition of a KWS server where a hugely important large load of audio processing could be shared that could allow even simple micro-controller to be satellites and I have been constantly bemused to why?
VAD can be central all you need is to be able to tell a KWS mic to start and stop and once more the fix is that simple. VAD should be able to reside on the satellite or central but currently it can not and the supposed fix forces so much unnecessary as a peer2peer client style architecture when a simple client-server would of sufficed.

Is it not time to get it right and fix it?

19

I will write it here in a relatively brief explanation but if you partition elements in basic lowest common denominator building blocks you can just collect those together to create any form and complex but there is choice of all.

If you embed function without need you will always be shackled providing for ill placed function and create a confusing and complex infrastructure and exclude certain choice.

There are only 2 types of interaction in a voice system Instructions and Responses.
A instruction is the OP and a response is prompted by a TTS question.
A instruction just needs the zone/channel and audio, whilst a response which turns on a mic has a skill server that got the original instruction and merely returns that zone/channel metadata but includes the skill server it is so the response audio can be returned.
A KWS server receives that and turns on the corresponding mic and the next response audio is shipped and returned to where needed because the skill server data is there.

Thats it that is also how simple the protocol could work because a voice system does not need to know about control.
It merely ships and routes what a voice server should do whilst skill servers do control.

There has always been 2 elements missing in the chain firstly a KWS Server and secondly a skill router.

The skill router is an intermediary fed by ASR that uses the attached metadata to do some very simple routing.
It forwards on predicate to the matching predicate skill server and if that skill server requires a response it returns to the skill server as there is only need for a single 1to1 simple low latency connection.
The Skill router sends TTS text to TTS and awaits a completion and then tells the KWS to turn the mic on.

Its the same for any type of voice interaction and everything is just a repetition of the above and its really simple and partitions the modules into basic function and those simple methods can be reused to create whatever needs and the complex but that is choice.

Keep Rhasspy as it as and restart anew V3 ( as new and seperate) with a simple, uniform API to local open source voice tools as what is needed is exceptionally simple and huge swathes of current has really no functional necessity apart from that it is.
Then if people want to use what exists because they developed it then they can, but don’t shackle once-more to what in the majority is functionally unnecessary for a voice system and even worse still not implement crucial elements such as audio processing.

20

No it is not. That is just your, as always, totally unnecessary negative opinion.
If you would have put as much positive energy in helping Rhasspy to become what your vision is, Rhasspy would now be very close to that.
Instead you have chosen to only put a huge amount of negative energy into complaining and whining again and again into what you think is all so terrible.

Why is that? Why do you only choose the negative path on this instead of putting that same effort into actually changing things to the way you would like it so see? I have asked that a couple of times, but still no answer.
I really do not understand this and when Rhasspy as a whole is so terribly you still keep posting your negative comments instead of just finding other systems you dó like.
Most of the time I skip your lengthy and incoherent posts, but that question always pops up when I scroll past them.

2 Likes
21

I picked up on rolyan’s comment a while back that he abandoned Rhasspy several years ago and has no experience with rhasspy satellite. Yet based on this total lack of actual experience he is stuck vehemently repeating allegations that only he seems to believe (like that Rhasspy is inextricably linked to Raspberry Pi), about software that is long since history.

Personally I don’t see much conceptual difference between cheap devices with mic and speaker spread around the house which listen for a keyword that are called “ears”, and the same device with same purpose called a “satellite”. Sure a Rhasspy satellite has the same user interface, but I don’t consider calling modules on a server to do all the cpu intensive processing as “bloat”. Similarly that rhasspy’s modular client-server architecture somehow does not allow KWS to be done on a separate shared server if one so wishes, or on the client so the audio doesn’t have to go through the LAN. He seems so fixated on using his own terminology that he can’t see that Rhasspy is conceptually pretty much what he is promoting. :frowning:

I freely admit that, while Rhasspy’s documentation does contain all the required information, it is not arranged in a way that makes base+satellite configuration clear. I guess there must have been quite a bit of confusion at the time of the transition. And the confusion continues, resulting in new users needing to ask for help on the forum; often having struggled to piece together the necessary pieces of information spread through the documentation. Please Michael don’t take this as an attack - I don’t like writing documentation either, and at the time you were adding satellite to existing documentation.

I suggest that rearranging the current documentation to make base+satellite the default configuration (and all-on-one as the advanced option) would help. And a comprehensive tutorial for new users … which I started and got to 30 pages before deciding I needed to re-think my approach. Now I’m not sure whether v3 will make it a waste of effort.

Bottom line, I really am puzzled that rolyan spends so much time on the Rhasspy forum, given his extreme prejudice against it. I suspect rolyan could have developed his own system with half the time and effort he has spent trolling Rhasspy.

rolyan I don’t understand why you could feel responsible for other people’s effort; more so because you are the only one that considers it a waste. If you really believe it to be a waste, why not just move on ?

1 Like
22

Beside the logo i dont like the name rhasspy either. In my language it sounds the same as raspi which leads to so much confusion i nearly never are able to use the word rhasspy. I need to use something like speech thing or such.

23

Light is “lámpa”, but in turn on the lights it’s “lámpát”, so the original word is changing as well, so it’s not just “lámpa[t]”. Turning on the lights IN the living room (“nappali”) is “nappaliban”, but in the bathroom (fürdő) it would be “fürdőben”, so the ending is different. Gender is not important for grammer. Verbs are sometimes attached to conjugation sometimes not, e.g. “turn off the lights” is “kapcsold ki a lámpát”, but “turn off lights” may be “lámpát kikapcsolni”, so where templates would work normally in English may not work with the same placeholders in Hungarian. I could circumvent some of that with variants like [entity](ban|ben), but that doesn’t cover everything. Oh, and we have “a” and “az” articles (this is for “the”, not for “a”/“an” which is simply “egy”), depending on if the word starts with a vowel or not. Yes, it can be put as “(a|az)” but then a single longer sentence ends up having 6-7 of such variants, makes it difficult to write/read/understand and still not grammatically perfect.

The translation web UI was really difficult to use with mycroft, it was really slow and made it difficult to see context, understand how single-word entries would be eventually used. (I was even considering doing it straight in git instead.) Ok, there is a word “start”, but in what context and term is it used? Makes a difference considering how endings are applied. Start is “indít”, but “start the timer” would “indítsd el az időzítőt” and “start a timer” would be “indíts egy időzítőt”.

I found other difficulties when doing translation for mycroft but can’t recall from the top of my head. I know there is plenty of supported stuff, like being able to customize numbers (e.g. we say “two” in two ways depending on context and that I think is handled), but I remember it all being so complicated to get my head around it. Perhaps this is not an issue if the core can cover most stuff and a selected few :slight_smile: can cover that. Other services (like chatgpt) that have language engines (or whatever) that knows grammar works better than having templates in this regard - I guess it’s no wonder that many language related services only support a selected core list of languages.

Not sure how to PM, though I was trying to find it, lol. :slight_smile:

24

I like this idea, but I would prefer it to be a networked option, so a plugin can be placed on multiple devices according to their resource needs.
The current MQTT solution would mean too much upload and download.
The audio port feature seems like a better solution, where there is a direct connection between the plugins.

25

Sorry Michael but I’m confuzzed again :frowning: Are you meaning that the satellites are part of the Base station ? And that the satellite’s Audio Output can be a streaming audio service ?
I was thinking that Rhasspy GUI should be a separate web server running in its own container, and uses API calls to Base and Satellite processes.

Looking at the recent posts on this forum, many are people struggling to configure their Base + Satellite rhasspy.
Moreover I realise that I often struggle to help because they only give the Base part of their configuration … as though they expect all the configuration to be in one place. I guess using the same UI can also add to confusion for new users.

I note that you have previously suggested auto-discovery of Satellite units, which I assume implies that they be controlled (or at least configured) centrally. I can see that it would be easy to provide a simplified HA Rhasspy Junior user interface using multiple tabs (as suggested here) for Base and each Satellite since only limited options would be provided for each unit.

In theory one web UI for Rhasspy 2.5 or v3 could also use device discovery and call Satellite API routines to provide remote configuration - though all the extra options will make the UI more confusing. I guess there’s not much point going down this path until we see what v3 brings ?

26

This is my first post and hopefully it’s in the right place.

With the changes coming to v3 is something like the ESP32-S3-BOX-LITE a good option for satellite hardware?

I have a dedicated server room with a TureNas Scale server that has tons of compute (AMD EPYC, as well as Tesla cards for acceleration) that can be used to run the main Rhasspy app on.

I’m not sure what kinda hardware is best for satellite nodes in each room though. My house currently has 10 Google home devices. Some are low end mini’s while others are dual speaker setups for music.

With hardware becoming more affordable I’m curious on what will end up being the best hardware for 2023.

27

As are we all :frowning:

Google and amazon have put lots of money into developing their own excellent hardware devices - which are locked to use their cloud service.

Raspberry Pi with reSpeaker 2-mic HAT seems to have been the most popular non-proprietary option - not because it is particularly wonderful, but because they were cheap, readily available and easy to program. Then came covid, and chip shortages :sob: There is also the fact that (despite having 2 mics in hardware) neither the driver or Rhasspy provides any of the Digital Signal Processing (DSP) to take advantage of both mics on the reSpeakers.

I know that some forum members are already using S3 devices as satellites. I understand that the ESP32-S3 will be particularly suitable with some AI capability built in; and up in message #17 above @synesthesiam suggested that ESP32-LyraT could be a go. It will of course be very dependant on getting the software. My fingers are still crossed.

As for Rhasspy v3, Michael hasn’t let anything slip, so far.

28

Hello all. I am new here but have extensive experience with raspberry pis and, separately, with voice interfaces. I am planning to use Home Assistant with voice this year and perhaps can contribute to the documentation by sharing my “journey”? I’d propose that I would start by writing a diary of what I am trying to do and what I discover, and then discuss my diary with someone who is taking responsibility for documentation. What I am trying to do is not an uncommon starting point, but I know that it does not fit comfortably with the wake-word/intent model made popular by the Echo.

29

The standard esp32 which the Lyra-T is really tight with the models you can use whilst that is what the esp32-s3 is about as the lx7 microcontroller is very similar apart from the additional vector instructions that boost ml close to x10 over a LyraT esp32.

Esspressif often make demo boards rather than dev boards where they throw the kitchen sink and everything else as a demo of what you could do than actually really have good use.
It has all the audio codecs, even mics on board and 2x 3watt speaker out, but its a demo as actually due to postioning and spec they are not that usefull and very toy like.

Even the esp32-box-lite is a demo and toy like where again they have thrown the kitchen sink of kws, audio processing, asr, tts & screen all on one esp32-s3 as a demo and gives you something to start with and hack away that likely a custom board or standard dev kits and modules would be more appropriate.
The esp32-box-lite does a huge amount not very well as a demo to demonstrate potential load so that if you made a specific KWS you have the 2x I2S ports of the S3 and those vector instructions to make something really low cost and modular that can run less models but better models.

Espressif have created some interesting audio processing libs and also ML where the demo products of Lyra-T & esp32-box-lite are a great dev resource but likely not end product and its only the S3 that is capable of running the more advanced libs, but as product you probably wouldn’t want what is the esp32-box-lite.

Also those demo boards are not that well priced, as if we where able to get Pi02W then likely they would not get a look in.
But if the community was up for it then a custom S3 KWS that is far more capable than the resource restrained one in the esp32-box-lite likely could hit the $10 mark if you get the qty’s as its ADC & mics that is all that is needed or I2S mics, but then there is the dev curve where the board design is likely the easier.

30

When you first mentioned ESP32-S3-box I was impressed at their demo - but noted that it is a demo, not a consumer product. But that is the nature of the beast. Neither Seeed or Espressif are in the business of selling direct to end users - they require system integrators to add value and marketing.

Espressif are at least showing that they can put the hardware combo we want onto a small board for a good price - we just need to add the software - communications, wake word detection, and as much DSP that is available and fits in memory - and a pretty box with speaker and power. Sounds simple, but way over my head technically :frowning:

And that was behind my asking @synesthesiam about discussions at nabu casa. Without cheap “ears” users cannot be expected to move from the cloud-based solutions to any local voice assistant. I was hoping that with the demand for low-cost “ears” and ESPHome as a base, maybe nabu casa might take the plunge to develop a S3 into a end-user product - as they already have with Blue and Yellow. I think its almost inevitable, but will take time.

31

esp32-s3-box is really impressive as they have all that running on what is a microcontroller! Its quite an accomplishment but in use its not going to be the next alexa.

I have been meaning to take the plunge as I don’t know how effective the audio processing chain is and would it provide much better results purely as KWS apportioning more resources to a model.
Esspresif do have

Espressif Audio Front-End AFE integrates AEC (Acoustic Echo Cancellation), VAD (Voice Activity Detection), BSS(Blind Source Separation) and NS (Noise Suppression).

Part of there GitHub - espressif/esp-skainet: Espressif intelligent voice assistant but I find the binary blobs a bit off putting as not so sure how much you can hack around tweak & config and keep meaning to but haven’t progressed much further than there.

There are https://www.aliexpress.us/item/2251832644497273.html I2S ADC stereo modules for a few $ that could be used with any esp32-s3 dev kit https://www.aliexpress.us/item/2255800678628772.html

So you don’t have to go full custom, one is more suited to the job than the other in terms of easy interface but as per usual forgot which one and would have scratch my head with datasheets once more.

32

I went through several learning curves getting my Home Assistant /Rhasspy Satellites + Base setup working, and tried to document as i went. At 30 pages / 1Mb of tutorial I started to think a different approach might be warranted. I am happy to share my document, especially to someone with fresh eyes - but email is probably better than posing here :wink:

I think the problem is mostly that (like all FOSS projects) documentation is written by developers for other developers, and so is rather technical. By stating that “Rhasspy is intended for savvy amateurs or advanced users that want to have a private voice interface to their chosen home automation software” they conveniently avoid mere “users”; despite non-technical people also wanting a private voice interface.

Honestly i think Michael’s Rhasspy documentation is better than Home Assistant’s; and i am hoping that (when he comes out of his cave developing Rhasspy v3) Michael will allow some of us users to get involved in developing a proper user-oriented website for Rhasspy project.

1 Like
33

Hi @Petr, welcome :wave:
I’d be curious to hear about what you’re intending to do.

Thanks! Not much to report on v3 right now. Most of my time is going towards the intent recognition for Home Assistant. Fortunately, that will be reusable for Rhasspy too.

As I work on v3, I’m doing my best to keep “mere users” in mind :slight_smile:
Automatic installation of the configured tools is a tough nut to crack. Rhasspy v2’s solution was to try and pack it all (pre-built) into the Docker image. I’d prefer not to do this as the image just gets bigger and more difficult to build over time. I’ve had some success building self-contained binary packages with guix, but it remains to be seen if the ML runtimes can be folded in.

I’m planning to order one of the non-lite versions to test. From what I’ve read, the lite doesn’t support acoustic echo cancellation (AEC), so it would have a harder time hearing you if it’s playing audio.
But the ESP32-S3 is definitely the current target for satellite devices. The big questions I have are:

  1. Can we use our own wake word system with Espressif’s audio framework? (you can’t train a custom wake word with their framework without paying them)
  2. How well does their audio processing work in practice?
  3. How much RAM does their framework take with speech to text turned off?

If the Pi02W can get back down to a reasonable price, I’d much rather go with that. We’d have to roll our own I2S in that though, wouldn’t we?

34

So would I as for $15 for 4 core perf it really has no competitor and its my fave raspberry product, so I have been tuning in for raspberry mention which sadly its had a lack of, it could even be 2024 before we see stocks.
I still have x2 on order with Farnel Available to manufacturer lead time of 373 days sort of backs up the sad state of affairs.

The company has been prioritizing its commercial customers, with the 100,000 units for enthusiasts containing “Zero W, 3A+ and the 2GB and 4GB variants of Raspberry Pi 4”.

Pi02W has had zero mention for a while there is the 3A+ which really is near the same but $10+ more. The orignal Zero has had a price hike to $10 and the ZeroW is $15 which likely means there isn’t a chance of seeing a Pi02W for $15 and for now they seem to of shelved it.

We have the 2 mic hats and plugable do a stereo ADC usb soundcard if you want to wire up a pair of mics.
There are fakes as with my luck I have one on my desk next to the one I purchased direct Plugable USB Audio Adapter – Plugable Technologies

I think one of the above I2S ADC also works as a standard slave just like the adafruit I2S mic driver but with only a single available I2S port on a PI the 2mic or USB prob have preference.
If anyone can state if the other respeaker still have the random channels due to TDM mode sync problems there is those whilst generally USB versions act like conference mics rather than a smart assitant targetted voice solution. I have the 4 & 6 mic hat on my desk and just despair at the driver status that seems to break on each kernel update.

As said the Pi02W is my fave product but as well as stock when it comes to ML raspberries top end Pi4 is landing short of a sweet spot of quite a varied range of models from all aspects of KWS, ASR, NLU, TTS and there long term partner Broadcom is trying desparately to acquire IP after the fallout of being the biggest backer of the Nvidia takeover and what seems more of a fallout with RS who used to manufacture under licence, who has switched to Rockchip.
Raspberry at the moment is a complete no-mans land of if and when, Pi02W has had zero mention and you have chipsets like the RK3588 kicking raspberries butt.
The OrangePi5 I recently got delivered for £86 for 4gb the CPU alone runs ML x4 Pi4 speed and that doesn’t even include the MaliG610 that with ArmNN has about 90% the ML perf as the CPU and we still haven’t mentioned its got a 3 core 2TOPs NPU which all in all if it was all utilised maybe a possible x20 ML boost over a Pi4 to demonstrate a gap that is quite huge irrespective of fan base.
Also slowly we are seeing some very capable 2nd user hardware come down in price where state of art home AI likely will be very much a thing.

I would prob be more likely to say there is better chance for the community to provide models that can run on a esp32-S3 via a model zoo for KWS with purchased in hardware such as a ZL38042LDF1 from microsemi as for me that solves the esspresif blobs as the models can run Opensource via Tensorflow4Micro.
If you where going to build something custom the total solution likely be around the price of the equivalent Pi Hat with ref designs and dev kits to clone such as ESP32-LyraTD-MSC Overview | Espressif Systems where opensource can be both fabbed and distributed via the likes of Seeed, as a S3 is pretty much a drop-in replacement for the lower ESP32.

Thats why the 2mic or USB soundcard maybe not optimal, but avail and I have working beamforming code, just no Pi’s available to purchase or at least the one I would prefer Pi02W.

The R in Rhasspy is currently both restrictive and not really available and have been honestly wondering if it is still viable as stuck for clear cut solutions.

35

I can half answer that as there wake word system is commercial but Tensorflow4Micro is avail on ESP32 it just has less layer types that it supports most notable are recurrent layers such as GRU or LSTM but a CNN or DSCNN model should run quite well on a ESP32-S3.
So you don’t use the Esspressif KWS as it seems extremely heavily quantised to fit anyway and custom is pay4 unless your prepared to run with ‘Hi ESP’ and run TF4Micro instead with there Audio Front End SR.

The ‘Lite’ version has a 2 channel ADC rather than 4 channel of the ‘Non-Lite’ and this is why AEC doesn’t work as a 3rd channel is used as a hardware loopback fed from the DAC as the AEC Ref Channel.

I still think there is a misnomer here in the term satelite as the community is being blindsighted as what is envisaged as commercial units as there is no such thing as magic AEC as the construction of smart speakers has some really clever structural methods to isolate microphones from speaker to give AEC a chance.
Have a look at the Google Nest Audio teardown.
I actually have a ESP32-S3-Box & ESP32-S3-Box-Lite and the AEC didn’t seem to be that great as I think its just a port of Speex-DSP that attenuates but doesn’t cancel and the plastic case supplied is nice but actually acts like a resonant box aka guitar like.

There is some simple lateral thought needed here and its just don’t stick your Mics and speakers in the same box unless you have the resources of big data and create more friendly reusable maker product via seperates such as active wireless speaker and wireless mic/kws.
By doing that you give yourself a huge advantage as the magnitude of the SNR of the resonance through a single case is absolutely huge and I keep desparately trying to explain that we shouldn’t tunnel vision on a ‘satelite’.

You can run both client & server audio system be it LMS, Snapcast or Airplay and clients and it will run quite happilly on a Pi3/4 with Rhasspy in a box with a amp or speaker or my personal favorite for ease is in free air screwed to the back of a 2nd user Bookshelf speaker(s).

There are so many advantages to employing a modern wireless audio system where the maker space can actually compete and hook up a home to give that vital reference signal and some really cool solutions that work…

The best people to talk to would be Phillipe & Sebastion from GitHub - sle118/squeezelite-esp32: ESP32 Music streaming based on Squeezelite, with support for multi-room sync, AirPlay, Bluetooth, Hardware buttons, display and more

36

So my understanding is that the usual way to do this is a microphone → VAD → WakeWord detection → Intent Identification → dialog management / action.

The challenge is when dealing with multiple rooms there are multiple microphones and, assuming there is just one voice server, does each microphone get its own VAD and WWD (and possibly Intent identification) or does this stuff go on the server. A “satellite” is a mic or 2 and some mix of the rest to go in a room.

I’d like to try out an “always listening” system (hence no cloud) with a large(ish) set of wakewords, and dynamic intents. I am also curious about how a speech recognition “dictionary” is trained.

37

There isn’t really a set way Petr as very dependent on hardware that VAD could be before or after Wakeword higher up in the system.
Also you could have microphones where beamforming and blind source seperation help with far-field reverberation and signal extraction, but is something often lacking.

Usually its more convienient to train and utilise a single wakeword that is “always listening” purely for that wakeword then transmit on recieve a command sentence to ASR even if its only “Stop”.
Some ASR have a phonetic lexicon as a “dictionary” whilst later models tend to do a beam search and try to use a sentence context like OpenAi’s whisper where occasionally it will get things totally wrong but the sentence will still be logical but overall tends to be more accurate with sentences.

There are a whole manner of ways you can do it but with phonetic lexicons, dictionary sparsity can lead to more accuracy and why I think Wakeword → Skill Router → Skill ASR could be more scalable.
Its more load and latency that way but the Skill Router is a predicate ASR looking for “Play”, “Show”, “Who”, “Turn”, “Set” that may pass both intent and wav to a secondary Skill Server to partition subjects that could be many controls “Light”, “Heater” where certain types of skill like a audio player could quickly rack up a huge subject dictionary so split them into skill servers, that could be a mixture of lexicon and context types suited and trained for a certain skill type.

wav2vec is a approx ‘phonetic’ type and can be fairly appalling without dictionary backup but compared to Whisper is extremely performant.

Is a good example where the addition of a language model greatly reduces spelling and errors that if was a specific skill ASR maybe even could be more accurate than Whisper and far faster or lighter.
It could even be possible to create a language model from a multivoice TTS that say the band names & track names of audio media skill that can be really hard to recognise with a general model you maybe could train a specific subject content model as you could just the controls you have.

You can compare against as both repo’s super easy installs especially whisper.cpp

38

As rolyan says, there is no set way. Rhasspy was intended as a toolkit and framework so we can mix and match and use the bits we want. They recognised 6 stages in processing a voice command, and provide multiple options at each stage.

A while back Rhasspy was refactored to support Satellite + Base operation - multiple satellites with microphones around the house + a more powerful central computer doing the heavier processing tasks - and leaving it up to the user to decide what stage is performed where.

The most popular configuration seems to be a Raspberry Pi Zero with microphone and speaker performing the Audio Recording and Wakeword detection (so all sound isn’t constantly being streamed over the LAN) and Audio Output for any response to the user. The Speech-To-Text , intent recognition, and Text-To-Speech all require more processing power and are more suitable for a more powerful computer (a RasPi 4 works fine, but a used PC is better).

But the framework is just as effective running all the stages on one computer, or any combination as @romkabouter explained recently here.

Unfortunately the documentation appears to have had Satellite + Base bolted on, making it rather hard to find the important information in the documentation :frowning:

39

@donburch I think you might find the single core 32bit Zero even a struggle for KWS as I was rubbing my hands with glee when the Pi02W came out @ $15 and there is still the more expensive Pi3A+ that does seem to have stock or if you can a 2nd user Pi3 of any denomination.

40

PS with it being year of the voice anyone else got any benchmarks / articles appraising last years best of?

I have a habit of going on hugging face and seeing what the most downloaded models are for as certain application type, from time to time.

41

So the key here is that satellite are in another room with microphone. I’d suggest that this is NOT mentioned in the “getting started” section of the manual, but is a separate use-case with a semarate entry. Of course it wants to be easy to do, so the “getting started” set-up should have clearly defined “modules” that can be taken off the voice server and put on a satellite machine. Similarly, multiple microphones for beam forming should be an extension style section of the manual. Finally, given the popularity of pis, I’d suggest the getting started part of the manual is all about a Pi3B. The mic-in-another-room(s) bit uses Pi Zero 2w - even if they are hard to get, because the migration of modules should be easy. And I’d suggest the beam forming uses the respeaker hat for a pi because they will have an interest in getting it working with flashing lights and so on.

42

Dunno about Pi Stock Petr even if they are faves but prob deserves a discusssion of its own so I created

43

Then…

Alas, while Rhasspy was re-factored to support the Base+Satellite model, I am of the opinion that the documentation was “bolted on”. I did find all the information in the documentation … somewhere … after at least 3 reads through the whole thing. The first mention of base+Satellite is tucked away down in the Tutorials section, and even there it isn’t fully explained (e.g. both approaches show Intent Handling as “disabled”).

image
image720×492 65 KB

Unfortunately I think at the time it would have been a major exercise to restructure the documentation to better reflect the base+satellite approach, while still satisfying the existing users. Michael also strikes me an excellent developer, and I doubt he considers documentation fun or even a core competency.

And now…

Well I made notes and tried building into a tutorial for my combination (RasPi ZeroW satellite + Base running HAOS with Rhasspy add-on), and at 30 pages started feeling that there is probably a better approach than one huge document … such as a website (allowing links to audio troubleshooting tips without bogging down the main flow). The other thing is that rhasspy is more a toolkit, with many ways it can be used … so really wanting multiple tutorials … and it would be great if they could be fitted into one framework.

It looks now as though 90% of users have adopted the base+satellite model, so easy to justify re-doing the official documentation, and with lots of new users maybe all that good technical information can be moved a bit to the back ?

Except that with v3 in the pipeline, is there much point ?

–
BTW, a few days back I tried sending you a private message offering to email my 30-page tutorial. Did you receive it ?

44

Sorry Don I am travelling at the moment and working off a phone and shared machines. I would love to see your 30 pager but can I leave it another 2 weeks when I will have a real machine and more time.

45

Of course !

There seems to be several new users asking for help setting up, so maybe I should buckle down and finish my tutorial now.

1 Like
46

Also, perhaps a crazier idea (maybe Rhasspy v4 or v5), would be using plugins compiled to WASM. Honestly, I don’t know much about how it actually works, but from reading/trying to stay on top of it - it seems like you could not rely on the OS to call out to cli and just execute them directly.

Weird benefits are the plugins could be easily downloadable and cross platform and maybe cross architecture. The biggest downside is that there are only a few languages that can currently compile to WASM (C and Rust, and i think Ruby), and I have no idea what interacting with WASM object from python looks like.

The tech might be a little to new atm - so it might be more of a future endeavour. It just seems to be the current hype machine around application packaging.

1 Like
47

Websockets would be great! It would be a lot easier to use for satellite communication.
Also external training would fix the issue that a rasspberry pi is sometimes too slow and times out.

48

Could be just a Socket but the ease of use and guarantee of order and delivery of TCP vs UDP is a big bonus.
Websockets has already been written and neatly differentiates text and binary packets which also makes it super easy to separate binary audio and text protocol messages.
Websockets is low latency and its 1to1 so it doesn’t broadcast across networked nodes just the destination.

Its included in SDK’s such as Arduino and ESP32 and scales all the way as if pretty much universally adopted, there are others but often microcontroller libs have less support such as gRPC.
Websockets really should of been a no-brainer.

2 Likes
49

I have been testing hardware and updating my simple beamformer code and really we have very few choices that are worthwhile.

Adafruit voice bonnet is good but for a ‘soundcard’ prob too expensive as also the respeaker 2mic is also OK but $10

The keyesstudio and other 2mic clones are often noisey and definately ewaste and not sure where it is as just got another, but be careful where you get your respeaker from as the 1st one I got was the same as the clones. This 2nd one is working fine.

I have GitHub - StuartIanNaylor/2ch_delay_sum: 2 channel delay sum beamformer /tmp/ds-out is the current TDOA
/tmp/ds-in if it exists sets the beam to the integer in the file, to clear just delete.
If anyone has even a touch of C/C++ finese then feel free to clone and tidy.

There is also Plugable USB Audio Adapter – Plugable Technologies with 2 channels or any el cheapo usb with a mic if you are not going to beamform.