I am trying to get my head round rhasspy-satellite and if its wrongly named or I need to think of another infrasture name.
I probably had preconcieved ideas on what satellite should be and maybe being too specific but to enable a mode where voice clients have a really simple layout and are basically mic/speaker wifi satellites to a singular more powerfull SoC.
I have been playing with snapcast for this and it looks like it should very much fit the bill, I need to do actual tests of multiple clients on a host but think you can actually hardcode IP and Port if needed.
If I take a simple 2 satellite system of left/right speaker/mic broadcasting is just a standard snapcast where the channel of the stream are set.
Each satelite can also contain a server and the thing to test is multiple streams/instances client side as not sure if this can be a singular server or instance, but there is always docker to the rescue.
Each stream can go to a loopback and on the other side would just look like another alsa source.
So essentially that is my satelite mode and it has some advantages as audio processing load is pushed to the left/right satelites or however many you may include in a room.
I am looking at the collection of libraries for satellites only and wondering why you would need so much if these are satellite only as it would seem to be practically a complete rhasspy install.
Even KWS is not needed on a satelite but an initialise KWS to an authoritive KWS brings some load advantages to the server if needed but everthing else should be server located and not satellite as its no advantage to satellite load.
By using snapcast you can vastly reduce latency as steams as they happen without the need for âvad silenceâ processing into wav chunks also the streams of multiple satellites are latency adjusted by the shared network time of the snapserver.
The multiple devices of the loopback can be aggregated via type multi then routed and summed as a quick start, but would be great to maybe add some finnesse to the audio routing of satellites, but as base its ready to go.
Looking at the modules included in rhasspy-satellite is not rhasspy-mesh more appropriate?
As it would seem from the modules included a distributed mesh of modules might be the idea?
Naming isnât really a problem apart from maybe it might confuse but for want of a better name I have a very specific and what would seem a logical model for a rhasspby satellite that actually requires nothing of the current rhasspy-satellite repo and curious to what rhasspy-satellite is supposed to be?
I would like to take a look at adding rhasspy custom rpcjson to snapcast to include channel vad for channel selection and return back state signals that could be used for LED indication, but after that due to diversification of use and what I can think of thats a complete satellite system.
That is the easiest way, and then control whatâs active via the settings.
You can also only run the docker services needed, but for non power users that is to complicated
Not in my opinion, mesh implies that satellites also connect to each other and relay messages. Which is not the case.
The hermes protocol was chosen, or better, evolved, because of Snips was shutting down
That is the easiest way, and then control whatâs active via the settings.
You can also only run the docker services needed, but for non power users that is to complicated
I sort of dissagree as a satelite only needs to pipe audio to and from a server and all you need to look at the ton of python libs and docker instances you have, that I have to say that is just not true.
Not in my opinion, mesh implies that satellites also connect to each other and relay messages. Which is not the case.
As name not so bothered but was trying to understand why so much of what is a complete rhaspby server was in the satellite package, I thought the rationale for that huge amount was so that a satellite could also be a distributed service of rhasspy as couldnât understand why so much was included in a satellite.
You do seem to have the makings of a mesh but usually a satellite is much more diminutive than what it orbits, but seems little difference in whats in the repo.
Doesnât have to have X satelites broadcast audio continously but can do so with a simple pipe.
Add vad or KWS it only broadcasts from mics that recieved good signal and provides huge advantages in a wide array microphone system.
But that isnât my point its why is there so much there in the Rhasspy-satelite that repo is enormous? Confused.com !?
Without hotword ? Just VAD ? no. In noising room, the satellite will continuously send via pipe or anything else.
For me, on satellite, you need audio recording (via pyAudio), hotword, (myChoice : TTS) and audio playing.
and tts itâs my choice but you can only user server one.
And you can do this 3-4 actions with rhasspy satellite. with good perf.
Yes rhasspy satellite is big, but itâs a project where a lot of modules need to speak together. And when you download rhasspy-satellite, may be you download module that youâll not use. But someone else will use. So it is in default rhasspy-satellite package.
But, as someone said before, if you want something light, you can use little docker images of strict necessary module.
I am afraid what you are saying is paradoxical as much of the docker service there rely on each other and from a guestimate of what would be required for a functional rhasspby satelite, that is quite a lotâŚ
My question is why is the satellite so big and no one can seem to give a reasonable reply apart from choice of it being so big as none of that is needed for a satelite.
You donât even need pyaudio as you can simple arecord into a pipe with a single cli command if you wanted to.
Seriously its paradoxical and non sensical as a paragraph that starts with the easiet way but too complicated for non power users.?!
That is the easiest way, and then control whatâs active via the settings.
You can also only run the docker services needed, but for non power users that is to complicated
I thought I would ask, I donât get it obviously, but you actually need very little in fact really none of it for a Rhasspy satellite and there is no paradox in that.
There is far more than just KWS in the repo and no you donât have to have KWS on a satellite if your network is cluttered or not by a few compressed audio streams is choice and also dependent firstly on VAD.
IMHO it would seem bloat, but I am not the one developing but the arguments for generally are not true.
That compressed streams of mono 16khz audio via flac or vorbis isnât a valid argument for the rationale of the design.
Not sure what the bitrate of S16_LE 16000hz mono is via flac or vorbis but its not that huge depending on compression ratio.
It might not be an option for you but the repo you are creating for Rhasspy-satellite is taking options away.
If you wish you can install KWS or all manner of things and that is choice but still much of what is there still isnât needed for this.
The thing I keep repeating is that you are not making things simple but doing the oppisite you are adding complexity to Rhasspy-satellite with what are paradoxical rationales.
I get that you guys want the whole Hermes framework, I donât know why but actually its not needed or makes things any simpler for use or employment.
If you are talking about Snapcast as an alternative for broadcasting audio frames between satellites and master, Iâd say that Iâm not a fan of basing such an essential piece of the Rhasspy ecosystem on someone elseâs software when we have already a perfectly working and simple solution using MQTT.
MQTT is now the âde factoâ standard for IoT devices and the Hermes protocol is pretty good and also completely open (it has some short comings though but they will probably be addressed in the future).
I really like the âpluggableâ nature of MQTT.
Also, Iâve used Snapcast extensively and although it is an incredible piece of software (perfect for synchronized audio playback), its group/client/stream management has given me headaches.
I agree with @ced_cox that broadcasting WIFI packets all the time is usually not wanted / required / advisable.
For an optimal setup, I think the satellite should only do audio IN, audio OUT and wakeword detection (pretty much like an Echo Dot or a Google Home). I agree that rhasspy-satellite is pretty bloated with stuffs that should probably not be embedded like NLU and dialogue management. This will surely evolve in the coming month as the official release of Rhasspy 2.5 approach.
The audio IN service emits audio frames to a MQTT broker by default. You can change the configuration of the audio IN service to broadcast all audio frames to a centralized MQTT broker if you wish:
Thatâs the beauty of Rhasspy 2.5 modular approach
That is exactly what I was saying and rhasspy-satellite seems to be already losing the beauty of that modularity due to much unnecessary bloat.
It doesnât have to be snapcast, could be pulseaudio but for me snapcast is this latency compensated, compressed stream audio system that seems to work beautifully and doesnât require programming input or support as that belongs to someone else.
I am bemused as Hermes-audio for satellite-rhasspy only rationale is because you want to program and support it maybe?
A Rhaspy-satellite doesnât need MQTT because its purely a voice capture and delivery mechanism for the server its the satellite for. The server needs âde factoâ standard MQTT but once I repeat this does not mean Rhasspy-satelite does and once more repeat why does this seems necessary?
From not saying it will be broadcasted all the time even if it is the approx 128Kb/s streams of mic audio are no problem with even low grade 56Mb/s wifi.
That security on the network is no different to the audio chunks transmitted in MQTT streams.
But losing the advantages and benefit of a wide array ad hoc latency adjusted microphone system seems to be a high price to pay.
I am looking at hardware now and need to get as much performance out of approx Pi3 level due to high audio processing needs. I can do this by offloading much, keeping things simple and because of the brilliant modular nature of Rhasspy can have an authoritve server do this for me.
Wav chunks by nature bring in and add latency that I can improve by having direct streams.
Its no trouble but sounding like I will not be using the Rhasspy-satellite part of the offering and will have to hack together something of my own, likely in conjunction with snapcast as in this instance I believe it to be superior to rhasspy-microphone-cli-hermes as there is need for instaneous stream and I donât need the packaging of MQTT for satellite to server audio.
The server will process that as that is what its there for, as is a satellite for Rhasspy as that is its only function to be a Rhasspy satellite.
At this point, MQTT is already here and it can be used a a satellite setup.
It is certainly not the only way to setup satellites
But for other setups, there is no software yet.
Since Rhasspy is open source, I think it would be a great idea to have different types of satellites.
MQTT is not already here you are just trying to enforce that its here in Rhasspy-satelite that isnât even here yet.
I donât understand the level of disengenuios comment on this topic is just rife as you post the documentation to provide contary evidence statements made 3 lines later.
PyAudio
Streams microphone data from a PyAudio device. This is the default audio input system, and should work with both ALSA and PulseAudio.
Set microphone.pyaudio.device to a PyAudio device number or leave blank for the default device. Streams 30ms chunks of 16-bit, 16 kHz mono audio by default (480 frames).
See rhasspy.audio_recorder.PyAudioRecorder for details.
According to the documentation the sofware that doesnât exist yet is actually the default.
The software exist and it require zero support or creation as its pure Alsa to set up the default playback device as a file or Fifo sink.
I am not going to use Rhasspy-satellite as I donât want to touch that repo as its massive huge bloat for the needs for a sattelite that has lightweight extremely effective off the shelf opensource solutions much built deep into linux already.
I have a hunch why there is such strong oppisition to using a strong piece of opensource software like snapcast, that is extremely lightweight and tackles the huge problem of audio latency very effectively.
From reading and looking at the snapcraft choice of GNU licence I know I will be using a different type of Rhasspy-satellite and it will not be from Rhasspy-sattelite as that repo is absolutely packed with unesscary bloat that has no need or rationale to be there.
That is an example of honesty without paradoxical statement and I am going to halt in this conversion as its been inflamed by some blatant untruths.
I have no idea why Hermes for satellite is so important to you, you havenât given a single clear rationale why apart its something you choose without justification.
Even in paragraphs under scrutiny what you say is a paradox?!
I am confused and bemused as always, have no wish for pointless argument but the replies have just exascerbated and made it more evident that the current choice is rather dubious.
I guess its all about sphere of interest and specific knowledge but there seems very little correlation to actual benefit.
So I will leave that one as is and apols if my opinion differs.
I really second the recommendation that you should take a look at voice2json.org the little sister of rhasspy if you feel that rhasspy itself is to bloated and has too much overhead for your usecase. Than you can still use the features you want but build the infrastructure around it to your liking.
I may do but no-one is saying rhasspy is bloated the point in question is rhasspy satellite which definitely is.
To be honest I will prob use rhasspy but the satellites will be very much my own implementation.
Thankfully it needs little to no coding and hence why I am wondering what is going on?
Having a fat rhasspy central sever and distributed light satellites has numerous advantages, rhasspy as a server can be as fat as it wishes and it is no bother but strangely in comparison to the proposed satellite rhasspy is extremely lean as there is very little difference between the 2.
@rolyan_trauts, I think you misunderstand me.
The software that is not there is: A satellite which uses gstreamer or http stream to push audio to the server.
Also, you do not have to touch any repo, but if you want a satellite not using MQTT there are options to build one.
This has not been done, because of the simple fact that Rhasspy has evolved from only a server. Then came the idea of remote microphone and we looked at Snips.
When Snips was being shut down, the idea was to give the people a replacement. Thus using the Hermes Protocol.
Currently there is no support for Snapcast as audio input/output, but I think its a good idea to see if it can be fit in.
There Hermes protocol is not important to me as such, but the communication between the different modules uses it so it is needed.
Rhasspy is build modular, so other ways for input/output of audio would be great.
Also other software for satellites would be welcomed much I think, because everyone has different needs.
Like all remote audio software and protocols generally you can always use as either a standard alsa or pulseaudio device sometimes both.
Where are you going to fit it in? As its already a standard alsa device
pcm.!default {
type plug
slave.pcm rate48000Hz
}
pcm.rate48000Hz {
type rate
slave {
pcm writeFile # Direct to the plugin which will write to a file
format S16_LE
rate 48000
}
}
pcm.writeFile {
type file
slave.pcm null
file "/tmp/snapfifo"
format "raw"
}
There is nothing to fit in as these present as standard linux audio devices.
Hermes protocol obviously is important to you as rhasspy-satelite has no need to speak to other modules because rhasspy-satelite should be what the name describes and be a satellite of a rhasspy-server not a clone of a rhaspy-server.
It doesnât need to fit in or need software as when on the server it uses standard alsa configuration or are you now going to claim alsa needs to fit in and this is why we need hermes-audio?
Call the repo rhasspy-clone or rhasspy-mesh but donât deceive and present as a lightweight satellite of rhasppy.
I really do not understand where you want to go with this.
There is no presentation of lightweight satellite, there is just one example on how you could setup satellites and a server.
Thatâs it, no more no less.
What is the point you want to make in this discussion, I do not get it.
If you can create a server and satellite another way then is currently available, that is perfectly fine.
I have create an MQTT audio streamer on ESP32, which was what I needed.
The hermes protocol is indeed important, because the choice was made to use that as a way of making the various modules communicate. It is not important to me as a person, but it was adapted from Snips when it was shut down.
Again, I am currently confused about the point you want to make.
Also its not just snapcast its any RTP audio in a real simple system for satelites where any audio after KWS auth, capture and playback is just a stream.
A 1 sec delay is nothing compared to some systems that wait for end silence before presenting complete sentance wavs to ASR and not sure where you get 1sec delay from but there will be delay from the KWS unless you have a permanent stream.
Someone or some project will catch on that high end commercial systems use distributed wide area microphone arrays to server based audio processing that with consumer based voiceAI has extremely strong paralells with wireless audio.
Low end can create low cost speaker/mics be it stereo, quad, 5.1 or above where the expense DSP is singular in a server and its a natural efficient distribution of cost.
Mic audio streams are extremely important as distributed wide array mics are vastly superior to any compact mic array by pure physical laws to how sound and capture operates.
Someone is going to catch on beyond the blinkered focus of singular capture points and badly bloated server systems.
Eventually it will be distributed room systems with a room processor that likely hooks up to a room media centre to also echo cancel any further sources.
If the Voice system is the media system then even that is not needed and quickly you start to outclass any singular point source.
Snapcast I mention purely because its amazingly simple and lite and gives latency adjusted streams.
Much of what we have with the current satelite system doesnât and needs a view of partitioning technology and use cost effectively and that will be the ultimate driver.
Also there seems to be mainia to create project specific versions of common already avalilable software just to tag with own name branding that greatly expands unnecessary support needs.
I agree with most of what you say, but I wonder how we can facilitate this into the right direction.
Communities play a big role in open source projects and re-wrapping and re-inventing things might be hard to avoid because it all lives by personal projects and the problems occurring in them.
So, what do you think is the best way forward? You have way more experience in sound than I have - I am just a software engineer -, and I already suggested to synethesiam to get you involved into the precise and mycroft workgroup - would you be interested or can we start some alternative threads projects to fast track some of the things you outlined (or at least check if there is a community to support that)?
To be honest I am no expert in sound and because I am not heavily involved I am not focussed or blinkered by the project.
I started my career at Advanced Music Systems but that was a long time ago.
I have been playing with equipment available and have noticed that the beamforming Aec device mic arrays are exceptionally good at picking up voice from a distributed noise field.
They are not so good when a singular predominant noise source exists.
In industry noisy backgrounds of many sources is common whilst in domestic situations the oppsite is true.
Its simple physics that a distributed wide array microphone system has the advantage by simple positioning rather than forcing physics through advanced DSP.
Its also a matter of cost and that a common use for Amazon/Google units is to play audio and quite cost effective they can be the audio source and distributed as a speaker system.
This also gives a distributed microphone system where generally a mic will be closer to the source than noise.
The Rhasspy satelite is new but its been completely bloated with all manner of unnessacary and also omits hugely important AEC or otherwise expects hardware which is very costly singularrly never mind in multiple room satelites.
There is no need to do anything but capture and play audio and be a 1st tier KWS detection system with maybe a simple protocol to hand back things like LED status, volumes and the likes.
A satellite is in its name and it ends in lite and usually is microscopic in respect to the body of orbit.
This is definately not currently so.
The best way forward for satelites is to drop much of the bloat and create just the basics keep it simple and have AEC of some sort.
The current Rhasspy satellite is probably wrongly named, feature rich and probably more of a Mesh than sattelite.
Seems to me much delight has gone into what python functionality can go into it without respect on how it may be used, how much cost that use will be and who will use it.
Personally I love what synethesiam did originally and had a brilliant insight into partitioning voiceAi infrastructure and that has been derailed almost completely by unesscary bloat across the whole project whilst missing some essential audio processing to create functional living space voiceAI with the exception of the assumption of expensive hardware use.
I started with Mycroft, jumped ship to Rhasspy but since you mentioned Linto and I had another look, probably going to jump ship again as much here has individual merit but as a whole it seems to be going off course as often opensource is prone to.
Are you checking version 2.5 or 2.4?
2.5 is what you describe actually
The only thing is, both satellite and server or a Rhasspy docker, but with specific settings to act as a master or satellite.
Rhasspy is still a work in progress. You have obviously much experience in audio processing, so for you AEC and cheap audio satellites are important, but most people are just fine with using a Raspberry Pi that they have lying around anyway. This doesnât mean that what you want (cheap satellites with very lightweight software) doesnât fit in Rhasspy, it just means that it doesnât exist yet. Hopefully you or other people experienced in these matters can contribute to it. Thatâs what open source is about, people joining and contributing their expertise to create better software than one person is able to create.
It already exists and this is another thing that is switching me off is the encapsulation of function just to make it Rhasspy.
Works great and only causes load when media is playing when barge in is needed.
Snapcast already exists and for me is just brilliantly simple for use.
Being opensource the json messages in snapcast prob could do with expanding for some simple additional sattelite roles and negate the addition for any further protocol.
It has a webserver that is any webroot put in its root folder of each server.
Mic/Speaker->Rhaspy server
Rhasspy server->Speaker/Mic
Thats it apart from KWS and probably VAD but nothing else is needed.
I really like what synethesiam did with intent2json but from my viewpoint anything beyond that is the premise of something else, what I call an âintent processorâ or just merely external app.
An intent processor may request TTS and includes the source method as a prefix to the following voice capture intent if needed or null if its a to return to a first stage intent.
There is a whole glut of stuff going on at the moment like skills for rhasspy that will be rhasspy only and in my mind its the wrong way to go.
That is an external skill provider that should really be able to work with any intent stream and would be a great app/project to provide.
I am no audio expert but you donât have to be to envisage how distribution of mics means close proximity to at least one is possible and if you want to keep things simple just sum the inputs in a asound.conf on the server.
But to be honest guys think I am going to jump ships and only posted info in response to a request by @ulno that my opinion is satellites should just be audio processors and RTP streamers to a voice AI server because that seems to provide the best cost effective use.
Rhasspy lacks its own KWS, it lacks dataset collection and model training and also it lacks audio processing such as AEC. But its not for me to set priorities just judge what others see as priorities and weigh up if there may be better alternatives elsewhere.
ASR has to be imported into the project because its so damn complex but the rest is fairly simple and specific and guess the big giant robot is just not for me.
It really sounds like you should look at Rhasspyâs little sister voice2json a little more.
You can cherry pick the wakeword and stt and tti components with none of the protocol overhead and build your own minimal structure around it using snapcast and mqtt or anything else really to tie it together.
Maybe worth a look?
Johannes
I guess I could do it all myself as all that is required is a mixer based on the peak VAD of last KWS time frame.
Its not rocket science really, in fact its quite simple and obvious in implementation.
However I am not interested in recreating the wheel alone whilst there are existing projects that are innovative, showing loads of promise and likely to garner herd support.
Voice2json looks extremely good like most things synesthesiam lays his hands on but its looking rather lonely at the moment.
The lack of innovation here with zero ideas apart from hijacking a project to resurrect a dead one verbatum, means I do have a destination and even though great its not Voice2json.
It doesnât have to be snapcast or streaming sattelites, but it does far more than just rehashing what was once before.
That replies here in terms of needed audio processing of whatever innovation they will be are âmaybe find another projectâ speak volumes in direction and thought.
Prob will still be around sharing hardware info but yeah I will spill the beans, maybe even voice2json dunno yet.
The KWS is looking like Linto as havenât tested but like what they are doing with datasets, but again wondering why they seem to of attached a display to a voice sattelite
But the base https://pypi.org/project/pyrtstools/ might be the KWS solution for Rhasspy due to how they are approaching datasets.
Actually uses the Mycroft MFCC and dunno why there optimised neon version is missing?
Thank you for the compliment Iâm thinking âsimple geniusâ would make a great epitaph and/or t-shirt!
I see the audiences for Rhasspy and voice2json as quite different. Rhasspy was intended for a less technical user, and I want it to reach as many people as possible (hence the Snips compatibility and multiple ways of doing the same thing).
voice2json is for command-line junkies like me, and is much more opinionated about tooling. I have no intention of âbloatingâ voice2json with a GUI or any kind of messaging infrastructure like MQTT. Unix pipes are the ultimate composable tool, but theyâre unfortunately not for everyone.
@JGKK is working on a node-RED plugin for voice2json that should help people caught between Rhasspy and voice2json
With Rhasspy feeling guilty about my opinion but just have this gut feeling there is this Wordperfect Vs MSoffice analog going on there. To be honest I am equally critical of Mycroft for it as well.
I see a clear partition via intent from VoiceAI to an intent processing project sort of Almond like and specific uses have much advantage to partitioning. Even media should not be part of a VoiceAI as it has a logical platform and that is a media player.
We all know what happened to Wordperfect with the all-in-one approach as infrastructure wise in use many people will be agnostic to what is underneath.
Its just opinion and gut feeling so who knows as often its very different, so enough of that.
In terms of Dev and implementation for the technical users whats your opinion on https://github.com/JuliaDSP/MFCC.jl especially RASTA-PLP or the jaw dropping array of audio processing methods it can provide.
It grabs https://labrosa.ee.columbia.edu/matlab/rastamat/ and more and dunno if my question about implementing SAD in the same way as feacalc(:energy) for MFCC apart from it creates and envelope similar to ALC from âsilenceâ to âsilenceâ for sentance part splitting.
Is just plain stupid or not.
