I am looking for a set of satellite microphones to be used in each room of my home. Voice output will be handled through Sonos speakers in the same room.
Based on your experience, what ESP32 based hardware can you recommend for satellites? The best wake word detection and voice recording is more important to me than the size of the device. Is this something a tiny esp32 based device can deliver? Or should I aim for USB based microphones connected to something like a pi?
If you can hold on and use a zero for now or with the price of a $5 esp32 maybe use and consume for now.
The new esp32-s3 is practically built for networked voice & vision AI where they are even doing libs for aec & BSS.
There probably isn’t a best esp32 hardware for satellite as always on streaming via MQTT is an extremely bad idea for quite a few reasons.
The esp32-wrover dev kits has a little extra sram that makes them a little bit more fexible and interfacing I2s mics is quite easy.
You might want to wait for the ESP32-S3 but there are a whole range of libs by esspressif such as esp-sr/audio_front_end at esp32s3_dev · espressif/esp-sr · GitHub that are not implemented in the above but maybe might be at one stage.
I only have the Matrix Voice and the M5 Atom.
The matrix voice has some nice feature like the led ring but does not have a nice case. Audio output is not that great and needs work (which I am currently doing)
The M5 Atom seems to have a low gain, but the form factor is very small and work quite well in my experience (and setup)
If formfactor does not really matter, you have more options with a device like a Pi.
A reason to use a Pi as well is that the ESP32 streamer does not have local hotword detection (yet), so it is streaming audio over your wifi network. This may or may not be an issue for you.
Regardless of rolyan’s yet another rant about the MQTT streaming, this is currently the way Rhasspy works. I am waiting for porcupine to release a library for the esp32, but I do not know if this will happen.
If it does, I want to implement local hotword detection so constant streaming might not be needed anymore
Both espressif and tensorflow provide NN models for KWS on esp32 and have done some time.
MQTT is a broadcast protocol supposedly for lightweight messaging and even one heavy binary stream isn’t a great idea but as you add rooms its gets even heavier and there is no rant that obviously that is a bad idea.
Also not having a local KWS means it is streaming all the time and all nodes are subject to this broadcast traffic.
But as said waiting out for the esp32-s3 as can use a better model type and its looking like it can also do beamingforming.
S3 is just the next gen XTensa LX7 MCU will eventually be priced about same as esp32 and espressif are also providing beamforming / aec and with the new vector instructions and that bit more oomf it will prob be the perfect satellite platform.
Hm, taking advantage of the ESP32-S3 certainly sounds tempting. My plan is to have a first installment ready by the end of this year, but if I understand you correctly, this might still be a little bit early for a ESP32-S3 based solution. I saw that the chip has been announced back in January already, but it is not yet available and neither are the libraries you mentioned, right?
Sounds like using a Pi might be a solid interim solution for now.
I have two Matrix Voice (without ESP32 though) and a Respeaker 4 mic array which I will probably use to start out until we have a more polished ESP32 based solution ready. As you already mentioned, the Matrix is quite large and I have not tried it with Rhasspy yet, but as far as quality goes, it should do the trick.
The Respeaker 4 works well when I am not too far away, but otherwise (5 meters or more) getting it to trigger the hotword (porcupine) takes a few attempts most of the time. I am actually curious to see how it compares to the Matrix Voice with its higher number of microphones. I also tried a Jabra 510 which works very well when close, but not at all when I am further away (which is probably what it has been designed for). I have to admit however, that I did not do any fine tuning of the microphone sensitivity yet on either device.
This situation is actually what triggered my initial question: I am thinking of putting more than one satellite mic in a room so that I am close to one most of the time. Especially the small ESP32 based combos could simply be plugged into an outlet.
Alternatively, I might put Zigbee based buttons around the house and trigger the hotword through pressing the button. This would also allow me to lower the sound on any speakers or TV within the same room (this of course I could do already without using a button by subscribing to the hotword triggered events).
From what I gather so far, however, I will give it a few more months and see where we are at then when it comes to solid satellite options.
Either way, keep up this great work @romkabouter - I will follow along!
Yes, when attached to the Pi it is a good Mic in my experience.
Microphones is an importent issue for voice assistants.
If you want more than one (small) mic in a room hidden away, the M5 Atom is very small. A bit like a cube of 1.5cm sides. The main issue with a lot of mikes is the network load I guess, at least untill there is good local hotword detected.
My original idea was to create audioswitches. A small device in a room entrance, triggered by a local hotword/command (like lights on). So when you walk into a room you can call “lights on” and the light in that room turn on.
Without local hotword detection, that is still an idea and not implemented
Far field really is all about AGC (Auto gain) and that the AGC is not so high that the SNR (signal to noise ratio) means much is now noise.
You get reverberation and the doppler effect of frequency of larger rooms and spaces but much is that sound levels diminish quickly over distance.
So check you have some form of AGC running and if not supported try software AGC.
Dunno how many times I have to say this but microphones are not an important issue for voice assistants its the DSP processing of array microphones that is important.
I can stick a $0.5 electret on $3 usb soundcard and have a good microphone in fact one that will likely work better than both of the above.
I have repeatedly stated that a KWS server is all that is needed using websockets as its lightweight and supported on ESP32 btu also allows string and binary bi-directional communication.
Your limited to using the same hardware in a room but all mics should 1st send KW probability hit and then start to broadcast.
The KWS server will just process the best signal and kick the others.
A Distributed Array just needs a co-ord server and that can output to std-out or even MQTT if you where so strangely inclined.
This topic has been raised on multiple times for quite some time and bemused to why its not been implemented as yes rather than single long range all encompassing mics you can do it simply by positioning distributed room mics.
Also what if probably more troublesome for VoiceAi is noise from anything from a washing machine to TV and with distributed mics, by numbers you are increasing the chance of at least one to be voice=near / noise=far.
But as for matrix its old news and an old post says it all.
PS as an update I think the esp32-s3 will be on sale November time.
Initial prices about double the price of the ESP32 wrover chip which from dim memory is about same as when the 1st esp32 got released.
Radxa just released a interesting quad core A53 1.8Ghz called a Radza Zero as its amazingly Pi Zero size but an Amlogic s905y2 on 12nm hence the higher clock speed than the Pi3.
Still really early days as devs are still putting together working images.
Interesting though as it does have 8 channel I2S/PDM
Thanks for this great insight @rolyan_trauts. Looks like a bright future is ahead of us in this regard (expect for the supply shortage).
On the software side, getting local hotword detection working will probably be key to reach the next milestone for that kind of setup.
In the meantime, I will put a few raspi/mic setups around my flat and collect some experience. Also, will try to optimize what I got by tweaking the settings.
Moving to a new place in December. Hopefully by then I will know which route to take. On the other hand, it would be ok to start out with what I got and improve the setup next year as long as I manage to get to a “usable” state for our smart home use case.
Yeah also because the SP32-S3 is the newer XTensa LX7 that has vector instructions to accelerate AI it should be able to provide more complex models such as ds-cnn which is a good combination of 100% tf embedded compatibility and accuracy.
The esp32 has been able to do a CNN for some time with little else for any other processing even if it could be a simple satellite KWS.
The extra the LX7 dual core gives with the pre written audio front-end algorithm’s should make it a no-brainer for satellite KWS as Espressif have focused strongly on that application.
Any I2S mic can connect to a ESP32 and forgot how many channels it has thinks its x2 (so x4)
Still prob not till November and at first prices will be a premium until many of the clone boards and market starts to sell in bulk.
They did demo vid dunno how it will be on reciept.
This is a really nice idea ! - like it!
Finally, it would be amazing if you could maintain the hotwords for all satellites centrally (e.g. in Rhasspy or so).
Already implemented it with a default yes/no because I was lazy.
What I need is a simple tool to upload and train data, which above mentioned seems to provide.
You still need to provide a lot of data for the keywords, but basically “lights on” and “lights off” should be a good start.
It does not seem to perform well in my satellite code, but stripped down to the core is works on my M5 Atom Echo
Maintaining in Rhasspy is a good idea, but I have not given that much thought yet.
But like everything we have without audio processing libs of beamforming or blind source separation they return pretty poor results in the presence of noise.
Still not sure what the state is with the esp32-s3 which is far more powerful vector math than just simple microcontrollers that is specifically aimed at a complete voice package.
There are some kits out ÂŁ20 and they seem to be going through revisions and mainly engineering samples, but think everything is being effected by the silicon shortages.
Also now the big thing is federated learning that solves huge problems in privacy a general purpose global model can be trained and have its weights effected by a locally trained small model so that user data can garner specific user accuracy without problems of privacy.
Its all part of Android 12 and especially the new pixel 6 phones where through use and correction a local model begins to learn user regional accent and expressions. Which all part of the new Tensor chip they have provided with quite a strong embedded TPU (vector math).
Works as AEC, BSS & NS finally arrive on a platform that makes extremely cost effective satellites but unless Hermes audio and IMO the manner of satellite implementation has changed. we have the satellites but Rhasspy still is in need of major change.
Still though to have mics included, case & display, $50 is hard to beat.
I ordered from aliexpress cost ￡45.46 delivered to the UK
I ordered one as well, see if I van get local wakeword running and have something left for other processes.
At least support for the esp32 satellite. With local wakeword, audio streaming over mqtt is still not a big issue in my opinion.
Local wakeword for a pi is already easy, so I still agee that mqtt is not the best for audio streaming alone, but is still a good choice as glue between al the different functionaliteit. And streaming audio also works well actually, so I still disagree that Rhasspy needs a major change on this.
Yeah it should run any tensorflow model @romkabouter and I keep meaning to have a look at the newer federated learning that is in there pixel phones.
In the pixel phones it combines a standard model with local training model, but how and what exactly dunno as haven’t been active with tensorflow lately.
I got one just to see how espressif have it set up and delve into the code as its much easier to hack away then build up from scratch.
I dunno if the mics are pdm or I2s and also inetrested in what they have used.
Also with Rhasspy haven’t paid much attention but still calling for audio input/output zones so multiple satellites and rooms can be provided for by a single brain.
Uncompressed raw audio over any protocol in the 2020’s is a horrendous solution in its own right but to encapsulate that in a lightweight control broadcast traffic is only glue as s hit apparently sticks.
When there is local KWS on the satellite, there is only a short while of actually sending uncompressed audio.
I assume that does not outweigh the major change needed for a different solution atm.
So while you have a good point in theory about raw audio, the actual problem with it is not so very big in my opinion. Maybe Mike is going to make that major change you so much desire in the future, I do not know and do not really care either
Again Hermes audio and the satellite implementation of Rhasspy did not have local KWS on a satelilite and like the Matrix and ESP products you seem to support this is your firmware we are talking about here they are purely always on broadcast satellites.
Even if a local KWS satellite was used you seem to forget about the all important ASR that contains the intent rather than purely a keyword to start broadcasting.
I am aware you do not know and don’t really care even though you often comment on the matter.
Zones is something else fundamental that Rhasspy misses call it Zones, Rooms or whatever but its absolutely essential to be able to easily group and associate input/output which audio is one of those.
Its sort of sad to see what was a very busy and active community turn into the trickle that it is today that can not be attributed to some of the community not caring, but hey at least they tried.
Maybe you should put more effort in actually contributing to Rhasspy to resolve the issues you are so passionatly about instead of always typing the same long texts regarding the matter.
Would be more productive, you post a lot of negativity but have zero contribution on change
I did at one stage, but things are so far out of whack and certain members are far too ready to turn to ad hominem than discuss or state they don’t care.
Rhasspy may have an over complex programmatical method of providing groups that like skills really there should be a plethora of uses for in existence, but the absence speaks volumes to current implementation needs and requirements.
Every time I hear Hermes Audio mentioned its a red flag to not waste my time, so I no longer do.
Occasionally as in the 1st reply of this thread, I do post on what has been some long term voids in an adequate voice /smart AI.
I cannot recall that I saw any pull request from you. Ever. But if you did, sorry about that and keep up the good work.
Also, keep up the good work for Rhasspy with regards to implementing the missing voids so we can move away from raw audio streaming. Hopefully some pull requests can be seen from you
I do not disagree with you on raw audio streaming, but I am not a Rhasspy developer and unlike you I do not have those big red flags on it. I would otherwise most probably have done something trying to change it. Instead my firmware just supports Rhasspy as it is, like it did a long time ago for Snips (where the whole mqtt audio shizzle is coming from in the first place)
If Rhasspy changes, the firmware will change along with it.
If I am going to be honest @romkabouter Rhasspy sort of doesn’t fit my needs anymore and I also am not a developer even though I have done quite a bit of work with tensorflow my main interest has been getting the working audio engineering and how on a Raspberry Pi platform we are near totally absent of the critical audio processing stages.
The esp32-S3 could be a game changer but how well it works has still to be seen as it does have AEC (Acoustic Echo Cancellation), BSS (Blind Source Seperation) & NS (Noise Suppression).
BSS is an alternative to Beamforming and it will be interesting to results as the best solutions use both and the combination of beamforming & BSS is probably how at least the Google devices work.
Amazon might use multiple mic beamforming alone with 6 array microphones whilst google used 2 but has I think a 3 array linear in its Nest Audio products.
More microphones increases resolution and accuracy of beamforming but each additional microphone exponentially increases process load.
The other advantage to the ESP32-S3 is its a vector math enabled microprocessor of low cost. The dev-kit board alone has arrived @ $15 and is not far off what the original ESP32 did so if a success then economies of sale should see similar prices maybe as low as sub $5 ESP32 boards.
Due to low cost this should allow the possibility of distributed wide array microphones where multiple 2mic array microphones can use KW hit probability to select the best signal source for a voice sentence.
This means you garner a hybrid of a conference wide array microphone system with low cost distributed array microphones to rival and better commercial alternatives.
You have been providing firmware for microphones that are absent of any audio pre-processing that have all the show stoppers that noise, distance and echo provide to recognition.
What you have been providing is essentially pointless as it provides no better results than a single el cheapo mic on a sound card.
You have disliked my posts informing others that apart from being a remote microphone do not expect anything more or anything like Google or Amazon performance as certain critical essential processes of voice AI are missing from your firmware.
MQTT is a control protocol that is lightweight and should be encrypted and by embedding audio into its payload because its broadcast traffic nodes that are not audio specific have to decrypt heavy audio traffic to just reject as not needed.
This is fine on the audio sender/receiver but completely kills over the use of lightweight control nodes in that MQTT network.
Audio does not need a broadcast traffic it has a sender and a destination and metadata and websockets is far more appropriate as its very simple to detect binary and string packets (audio & metadata) providing a very efficient way to transmit and separate the two whilst only occurring between the sender and recieve nodes than a whole MQTT network.
Also because only command sentence is sent after KW authorisation its very likely no encryption is needed or at least encryption is choice to create even lighter audio system transmit.
Where many room/zone systems can work over fairly low bandwidth WiFi/ethernet but transmission is sperate from MQTT as its pretty essential for any control protocol to be lightweight & encrypted and filling with unnecessary audio binaries creates huge load whilst there is not need for MQTT on a microphone/audio satelite like most commercial RTP protocols work, so MQTT can remain lightweight & secure and a control protocol.
That is incorrect. I actually stated myself in various occasions that the firmware is not more than a remote microphone.
Indeed it does not better results than a single cheap mike, but that is also not ever what I have advertised in any way.
The firmware offers a way to have a mike away from a server hardware having any sound input.
If that is pointless in your opinion, then that is what it is, your opinion.
I also created it for myself in the first place, because my server is hidden in a closet somewhere and I wanted a remote mike. But I saw interest from others for exactly that reason so opened it up on Github.
If you or someone else can provide good AEC and NS for my firmware, I would very much appreciate that because I have about zero knowledge about that audio processing stuff.
For MQTT and audio I agree that it not ideal, but you really should blame Snips (and now Sonos)
Rhasspy did nothing more than jump in the hole when Snips was sold. Also MQTT is not a control protocol, it is a message protocol for large variety of use-cases.
And basically, if you have Rhasspy setup with Raspberry’s or a NUC, you would never have to stream audio over the network. Everything can be local and no MQTT audio is not needed for that at all.
The firmware I wrote does sadly not support local KWS atm, so the only way to achieve a remote mike system was to connect an audio stream. I already had that working for Snips via MQTT so it was quickly done for Rhasspy.
I found other features more important than to find another way for audio broadcast. And hopefully I can find some time in the future to do local KWS.
That will result in a very short time of audio broadcasting, so that is why I think that this whole MQTT audio issue is blown very much out of proportion by you.
I had a branch for websockets audio as well, but stopped working on it due to time issues.
If you really need AEC and NS, which I can very much understand, then do not use the ESP32 firmware. But that is true for every mike you blindly connect to Rhasspy, because a simple USB mike does no AEC or NS either.
So can we please leave this pointless discussion behind us and spend our energy in a positive way?
This is exactly what I am telling you as AEC & NS has been part of the ESP32 software stack for 4 years.
Like MQTT its the manner you are using it as like many AEC audio in & out needs to be critically in sync for it to work.
That is with the older lesser powered ESP32 which supposedly it wasn’t that great like all the AEC & NS software we have for any platform unless using dedicated and expensive hardware.
The ESP32-S3 has an update of these libs and also more power so that we might not have the only option of dedicated and expensive hardware.
Multiple remote mics if via a local KWS which also for several years can run on ESP32 but has had a major boost on the vector math and improved performance of the S3.
So on all other platforms we are stuck in the exact same position of expensive hardware whilst with the introduction of the S3 we also gain BSS and a vendor who is actively supporting low cost solutions for this very specific area and its ESP32-S3.
The pointless discussion is the continuation of the cul-de-sac that is Snips and there methods as it is no longer, is not available and the remnants we have left are extremely poor in performance.
There is no way I can be positive of input that is going to purely Ostrich from the essential start of a VoiceAI system of audio processing because they have zero knowledge and continually dragging the community down that cul-de-sac.
We are nearing a low cost and importance of a singular microphone of use and its time to be positive and discuss the best ways forward to implement that.
I know that, but I do not have the knowledge to implement it at the moment. If you have a good repo I can use as example I will check that out. I hope with that S3 and the code examples I can figure those out
It mainly includes AEC (Acoustic Echo Cancellation) , BSS (Blind Source Separation), NS (Noise Suppression). ESP-SR encapsulates the above algorithms into simple APIs. Without undersatanding details of these algorithms, developers only need to arrange and combine the algorithms that need to be used for specific application scenarios, and input the voice data to be processed according to the API format requirements and can get the results.
wakenet strangely has opted for a in-house and charges for custom wakewords whilst tensorflow embedded has been posted on this forum several times in alleged long winded posts.
Its all been posted before but was a tight squeeze even on the 8mb PSRAM ESP32.
I presume they are using unidirectional microphones on the ADC as all I2S/PDM mics seem omnidirectional but there must be a rationale to the ADC route than a lower cost I2S/PDM route.
this is soon - looking forward to it!