Hi all and thank-you for putting into my brain a great amount of things that I would never have thought of even if I don’t always know what the hell you are on about.
Anyway, I am embarking on this new to me RPi project to displace a small herd of Alexas which is probably more than I am capable of so need all the help I can get for starters I would very much like a kind of rundown on where I can expect to get to and with what hw/sw I would have the best chances of getting there. I have searched under various terms from OS to microphones and have learnt a lot of stuff and I do understand that the knowledgeable may not want to promote a particular item/brand whatever but an updated 2024 guide for the willing but less-than-expert who wants to order some stuff and get started but doesn’t want to end up with a drawer full of junk that he never used…
Thanks for everything you have already done and thanks in advance for that which you might do in the future.
Hi Johnny, and welcome to the rabbit warren that is voice assistants 
Firstly, You will not get the same experience from any Open Source project as you do from Alexa. They have invested huge $ in developing their hardware and software, and especially in keeping their key algorithms proprietary. Rhasspy (and other open source projects) are getting closer … but their big advantage is local control if you can put up with a few rough edges.
Secondly, if you are using Home Assistant I recommend you investigate HA Voice Assist instead. It has been built by Mike Hansen who developed Rhasspy; and is the latest version, closely integrated with Home Assistant - which seems to be what 75% of users were using Rhasspy for anyway.
Thirdly, voice assistant devices is a hot topic. When Rhasspy was developed, Raspberry Pi’s were cheaper and readily available, and with reSpeaker (and clone) HATs made a good voice assistant satellite … but I no longer recommend them unless you already have a RasPi sitting around unused. Also note that there are plenty of multi-mic boards which do not have firmware or drivers available which take advantage of more than one mic.
Over at Home Assistant, they have done work improving the Espressif ESP32-S3-Box as a voice assistant, and there are hints that Nabu Casa are “looking at” making their own satellite hardware devices. Conference mics can be good, but tend to be rather expensive and assume no background noise.
Basically there is no clear winner in the voice satellite hardware category at the moment.
Thanks for your reply, yes I am happy to put up with a few rough edges in exchange for the ability to program out bad behaviour which I experience with Alexa. I find myself regularly shouting “Alexa, ALEXA !! STOP !!” Because she “misheard” my song choice and went ahead and played whatever the hell song without any confirmation. I hope that I would be able to program into the behaviour of “RaspBarry” (what my kids have started calling the as-yet non-existent Alexa replacement that I keep mentioning) a quick confirmation before playing if there was any doubt, I could program to avoid that kind of thing couldn’t I ?
Yes, I have a Raspberry Pi bought for a small kiosk task displaying some solar panel stats but I think the Raspberry can cope with a voice assistant task as well ?
I am not currently using HA but if it is a good thing to use for voice assistant then why not. It is able to operate local only and that’s what I want.
The main stumbling block currently is the ASR engine which uses OPenAi’s Whisper.
The above gives some info on the WER (Word Error Rate).
Whisper is a conversational ASR (Automatic Speech Recognition) based on a LLM (Large Language Model) and this is where the rated WER and what users might find start to depart.
It uses a LLM that works with a beam search that I have forgot how many branches and also the input length. 10 branches over 20 sec audio.
The more branches, more possibilities Whisper will consider and the longer the audio chunk the more context the LLM part of whisper has on what results it will choose.
To squeeze it into what was a Max of a Pi4 at the time much of the parameters have been shortened to reduce computational complexity at the cost of accuracy and WER starts to climb as the smaller models with very short beams searchs and audio chunk sizes where likely the worst scenario is in streaming use of very small audio chunks.
Also when it comes to language generally its native language speaker and regional or non native can also increase WER.
The good thing about Whisper is it was trained on a typical microphone situation similar to Streamers and gamers would use, which really is near-field <0.3m, but is tolerant and can be quite tolerant of room echo out to <1m but fails farfield.
So you can pick up a cheap USB mic or use onboard as likely they where in the training dataset as where audiobooks and ted talks and things that likley had a transcription.
You don’t need the confusion of Rhasspy to get to the 1st hurdle of where opensource departs from Commercial home assistants.
You can grab a copy of Whisper likely GitHub - ggerganov/whisper.cpp: Port of OpenAI's Whisper model in C/C++ is a good start as it has a numerous dev community that likely will help you along, and is littered with references and 3rd party code.
As for Alexa I have an Alexa 4th Gen echo as from time to time I do purchase the latest and greatest to review. That replaced my Nest Audio which I consider superior, lost interest with the Gen5 as even previous versions are still on a different level to what we have here and as far as I know there is no music player even though mentioned in the api.
Rhasspy and HA for ASR currently use Whisper which with the above with a simple desk mic for Dev trials you can get going with very little outlay, in fact none as likely you have a headset that you can plug into a Pi.
Currently there is no AEC or input audio processing and the playing audio will be above what Whisper can tolerate and no matter how many times you shout, it will not hear you.
Whisper is a good start if you want to get a quick introduction to Voice assistants that very much can be a rabbit warren.
For your case scenario of ‘Barge in’ Rhasspy doesn’t even have anything implemented and is hoping for magic hardware or conference mics.
Also with the weird and wonderful band names and tracks we have this can make for some very unusual sentences for a LLM.
Likley having a play will help you think about context as “Play me some music by the band The Beatles” is laden with context and likely will work much better than “Play me The Beatles” as an example for all bands with weird and wondeful titles.
“Alexa who is playing” annoys me the most than telling me what the current radio track is its even stranger to get the scores from the Bundesliga especially whilst I am English.
“Alexa stop” always works for me, but does often get the wrong band, but it would really annoy me to have to listen to an announcement of what was about to be played than just play it.
Again I am presuming its about context as notice short band names such as ‘Squid’ or ‘Shame’ are often the most problematic, but you have whisper which is the ASR of use and you can dev and test away, without much waste of time.
Home Assistant is very focussed on controlling and automating household operations - such as closing blinds and turning on lights at sunset, and reminding you that you have left the garage door open at 7pm.
Voice Assist is only one tiny component of the Home Assistant project, and so I suggest would be overkill for your requirement. However that is where I have come from, so I haven’t paid much attention to the other Voice Assistant projects out there. I expect other people can give better recommendations.
Right. Quite complex then. I’ve read on some of the threads on here about mics spread around the room, would that significantly improve the long-range ? I will also separate the mic(s) and speaker(s) by a couple of metres.
My Alexa does already announce what is about to play followed by “…by Amazon Music” so if this was more of a question rather than a statement plus a half-second wait for a “yes” or “no” I would be ok with that. I probably only want a confirmation if the certainty of understanding was low, assuming there is some useable measurement of that.
On another topic: I want to have a voice-operated intercom like Alexa should provide but doesn’t anymore. Is this easy enough to do with Rhasspy or HA or am I opening a whole new rabbit warren ?
Haven’t tried but from the options of Whisper.cpp
usage: ./main [options] file0.wav file1.wav ...
options:
-h, --help [default] show this help message and exit
-t N, --threads N [4 ] number of threads to use during computation
-p N, --processors N [1 ] number of processors to use during computation
-ot N, --offset-t N [0 ] time offset in milliseconds
-on N, --offset-n N [0 ] segment index offset
-d N, --duration N [0 ] duration of audio to process in milliseconds
-mc N, --max-context N [-1 ] maximum number of text context tokens to store
-ml N, --max-len N [0 ] maximum segment length in characters
-sow, --split-on-word [false ] split on word rather than on token
-bo N, --best-of N [5 ] number of best candidates to keep
-bs N, --beam-size N [5 ] beam size for beam search
-wt N, --word-thold N [0.01 ] word timestamp probability threshold
-et N, --entropy-thold N [2.40 ] entropy threshold for decoder fail
-lpt N, --logprob-thold N [-1.00 ] log probability threshold for decoder fail
-debug, --debug-mode [false ] enable debug mode (eg. dump log_mel)
-tr, --translate [false ] translate from source language to english
-di, --diarize [false ] stereo audio diarization
-tdrz, --tinydiarize [false ] enable tinydiarize (requires a tdrz model)
-nf, --no-fallback [false ] do not use temperature fallback while decoding
-otxt, --output-txt [false ] output result in a text file
-ovtt, --output-vtt [false ] output result in a vtt file
-osrt, --output-srt [false ] output result in a srt file
-olrc, --output-lrc [false ] output result in a lrc file
-owts, --output-words [false ] output script for generating karaoke video
-fp, --font-path [/System/Library/Fonts/Supplemental/Courier New Bold.ttf] path to a monospace font for karaoke video
-ocsv, --output-csv [false ] output result in a CSV file
-oj, --output-json [false ] output result in a JSON file
-ojf, --output-json-full [false ] include more information in the JSON file
-of FNAME, --output-file FNAME [ ] output file path (without file extension)
-ps, --print-special [false ] print special tokens
-pc, --print-colors [false ] print colors
-pp, --print-progress [false ] print progress
-nt, --no-timestamps [false ] do not print timestamps
-l LANG, --language LANG [en ] spoken language ('auto' for auto-detect)
-dl, --detect-language [false ] exit after automatically detecting language
--prompt PROMPT [ ] initial prompt
-m FNAME, --model FNAME [models/ggml-base.en.bin] model path
-f FNAME, --file FNAME [ ] input WAV file path
-oved D, --ov-e-device DNAME [CPU ] the OpenVINO device used for encode inference
-ls, --log-score [false ] log best decoder scores of tokens
-ng, --no-gpu [false ] disable GPU
If I remember rightly Rhasspy uses FasterWhisper but when it gets embedded you lose much of the api that Whisper.cpp has.
Its why I suggest Whisper.cpp as with the above cli options from number of best candidates to keep to word timestamp probability threshold likely the answer is yes.
Also running the biggest and best model with a headset and seeing the results of some of the command sentences Alexa fails on with Whisper should give you a good idea.
Unless your going to dedicate a MacMini or GPU to Whisper then I am pretty sure only the Large model was peer reviewed, but the WER on the smaller ones is extremely optimistic. (Could be my Brit English vs American)
You don’t have to think about Mics as you can just use a headset and test what would likely be the very best results you will ever get.
I think if you play with the CLI options of Whisper.cpp and the json output options, you can get something like you are asking for.
Wasn’t the voice-operated intercom part of the frigate/camera setup and just used the 2way audio of security cams?
It should be very easy to implement if we had a zonal mic and audio system.
PS multiple distributed zone mics is simple physical position that you will always be nearer to one of the mics, so you don’t need to try to do long range, or at least minimise the range.
So a Pi 4 with 8gb ram is probably not powerful enough for the best version of Whisper ? I don’t think I would get anything bigger just yet so if that is the case I will have to make do for the moment.
So I need to look into a zonal mic and audio system.
If I remember rightly with a Pi4 really we where talking the tiny or base models that have been quantised and also gain a bit more WER through quantisation.
Likely you can run the large model but the delay for transcription will be substantial, but for testing use possible.
memory-usage
Due to the newer instructions of ArmV8.2 the Pi5 & Opi5 with ML can be x6 faster than a Pi4 but still for use the small or medium models.
The newer CortexA55 little cores due to mat/mul vector instructions are faster than the CortexA73 big cores of a Pi4.
Its always confused me that with a emphasis for Raspberry Whisper was chosen especially this was before the Pi5.
Whisper with large models post SotA scores but when your running heavily quantised tiny or base models WER is no longer the same by quite a long way.
Have a play as you can still run the models with a 8GB especially with zram but transcription will be slow.
With music and wonderful band names, these are quite nonsensical sentences to the likely training data of whisper, give it a test and see how you go.
There are 2 great zonal audio systems Snapcast and Squeezelite which are likely a bit simpler than DiY with a Pulseaudio or another audio server.
My fave is Snapcast that you create streams and you can join as many clients to that stream as you wish. Generally a stream is a Zone (Room) with a single client.
I think it will run on a Pi0 but with such little difference in cost to a Pi02 whch will have no problem.
Squeezelite is even lighter and also runs on ESP32 boards, but as said have a pref for Snapcast.
Dunno about zonal microphones but its something I have been going on about for a while that Keyword and server activated zonal microphones should be a thing.
PS I have tried a few budget pan-tilt cams with bi-directional audio and so far they have been pretty useless with big latency.
I do have a Anker C300 and its dual mic is better than the Anker Powerconf I have.
The advantage of the Powerconf ‘conference mic’ is the built in audio and AEC.
You can do the same with SpeexAEC that is opensource with a webcam and likely better than a pan tilt as the servo’s are extremely noisy and probable cause for the bad audio.
I guess there are premium product that is much better, but out of my testing price range.
I have loads of questions: I will go away and do some more study and come back when I’ve got a better idea of what I need to ask you.
Thanks for your input so far.
Quick question though: talking zonal, do we mean zones within the same room for mic response or zones within a house as in for the intercom function ?
Edit: Any difference between 32 bit and 64 bit Pi OS for my purpose ?
for me the whisper error rate is too high., its like the old pocketsphinx libs from Carnegie Mellon. my success rate is near 10%
one of my projects uses the google speech reco, while cloud, is pretty responsive and accurate.
i made a wyoming asr using it
wyoming uses centralized asr, with satellite speech capture and playback.
but reco is done at the end of the speech not inline word by word
no one has built a good person speaker location dependent engine yet. alexa timers etc are alexa echo box specific… not where i am.
there are some that are trying, but human speaker location is a problem, as you currently have to announce yourself. speaker dependant is also a ways off. (recognize joe vs sue vs bob)
and then we need the skills type interface.
what words invoke what service to get you the results you want. ( and where you are)
Its not just the accuracy it inheritting a huge model like that for training is even worse.
The resources needed to finetune Whisper as not many have a RTX3090 24gb spare.
Wenet are quite a bit ahead by deliberately choosing older tech that they can create on-the-fly language models and also create multi-modal ASR.
Certain operations aka Domains don’t need a general purpose complete language ASR.
A Domain such as control that has a small number of predicates (Turn on, set to…) and entities (light, curtain…) can have small language models specific to that domain.
This allow much smaller models to run on lesser hardware because the language model is purely for a specific domain.
Posted this before but they explain this eloquently.
https://wenet.org.cn/wenet/lm.html
LM for WeNet
WeNet uses n-gram based statistical language model and the WFST framework to support the custom language model. And LM is only supported in runtime of WeNet.Motivation
Why n-gram based LM? This may be the first question many people will ask. Now that LM based on RNN and Transformer is in full swing, why does WeNet go backward? The reason is simple, it is for productivity. The n-gram-based language model has mature and complete training tools, any amount of corpus can be trained, the training is very fast, the hotfix is easy, and it has a wide range of mature applications in actual products.
Speechbrain also and many others such as the likes of Vosk.
Also Bigdata doesn’t want your voice in the cloud as its expensive and Google has halted all Assistant Dev unless its ondevice, where there Tensor TPU in Pixel phones and tablet uses the users electricity and hardware for free.
Apple apparently have cancelled the Mac Mini M3 as the next release will be a M4 with Siri and generative AI ondevice.
Amazon just leaks $ like a sieve and makes a loss on some great sounding Gen4 Echo devices 
Each realises revenue comes from the services they provide and they aim to provide the best devices to trap you to there services as surveillance capitalism has and is failing to create revenue as we know the articles published for some time now.
This is old news but just to add offline on-device makes little difference if you are going to use the services that the provide such as Amazon or Youtube music.
Bigdata realises it doesn’t need surveillance as that info is easier to capture via service API’s that are vastly more efficient to run.
Its even crazy to embed a singular engine in the fast world of AI and this branding and cloning consumer smart speakers is not smart at all. We need a opensource Linux Speech Framework not a collection of hoovered permisive licenced opensource, refactored as branded as own.
ASR does a single job, its presented with audio (speech) that is transcribed (text) the ASR module in a Linux Speech Framework is purely a container for any engine that has various methods for collecting audio and sending the transcription to the next container of some form of LLM or Skillserver.
A Smart Voice Assitant is a serial chain of containers all providing specific function and its incredably simple and must processes are completely agnostic of protocol as all they need to do is pass a binary/text pair of audio and a yaml of collected data of that command session.
You don’t even need a skills type interface as that is another container that is merely a transcript router that uses transcript predicate as a choice of what skill server to route to…
Depending on audio out such as Snapcast or Squeezelite, or the zonal audio in system currently we don’t have.
A zone is just a logical collection of devices, likely a room, but its your choice what you define.
With audio out you might have individual channels of a surround sound system that are all seperate or it could be a single amplifier.
Audio in is the same its just a logical collection of mics that may represent a room or even part of its area.
It just allow logical collections of audio in that you can associate to audio out that the basic assumption unless stated is Assistant responses return to the same zone they where initiated in.
With ML unless your using a Microcontroller 32bit halves the SIMD instructions the CPU can make in one pass.
Much faster engines use quantisation and the smaller multiple words passed to a large databus to be process via SIMD (Single Instruction Multiple Data) the faster it is near expotentially.
64bit ML on CortexA (pi) is near 2x faster than 32bit as you can pass 2x as many words with a single data instruction.
I think the Neon is 256bit from memory and can take 4 data instructions before processing the SIMD instruction ( I forget :).
Microcontrollers also have 32bit (some are 64 bit) but yeah its halves the amount of vectors you can pass in a single data instruction.
well, you got to have SOMETHING that matches gramar to actions.
whats the weather is a lot different than turn on x
i currently support smart mirror, a voice driven info panel.
each extension can register its gramar so that the router knows where to send it to.
on my list is to replace the hard coded snowboy/reco with wyoming. but its all js, so
I meant it doesn’t need to be part of a system, its very simple to link predicates (The action gramar) to an IP where transcription and metadata is sent.
The router can be standalone and is a router not an interface to a system as transcript and metadata is the protocol.
Routers can be simple text matching solutions to LLMs but a seperate container, that is the next stage in a Linux Speech Framework, so you can have choice than a dedicated system interface.
PS as for hardware likely state of the art is
https://www.kickstarter.com/projects/smartaudio/ankerwork-s600-all-in-one-speakerphone-0
Its the only hardware outside of a smartspeaker that does targetted voice extraction based on user profiles.
There could be problems with Whisper as certain filters create flat reverb free voice, again depends on alg and the signature of artifacts.
Whisper could work fine but after $138.00 pledge that is just a disclaimer as that is Whisper not the Speakerphone.
Google with voice-filter published the 1st paper on targetted voice extraction.
The key tech is that rather trying to cancel the unknown of noise, they extract voice by creating a voice profile.
This is the direction most on-device voice tech is going with even ASR learning user voice patterns via a small trained model that shifts the weights of a larger pretrained model.
I stopped buying tech and Dev but my last will be to review the S-600 as I can still publish how well it extracts voice and publish a few samples with other tech recording at the same time.
I will post some recordings when I have reviewed mine.
@rexxdad Have you tried Online ASR with Emformer RNN-T — Torchaudio 2.2.0.dev20240503 documentation as for embedded it might be a better fit for a Pi than Whisper and if you read up on rnn-t then personal LM & maybe ondevice training might be possible.
Or wenet as if you search both do have pretrained models but are very geared up for devs.
Products such as the above S600 or filters for optimum ASR need the ASR dataset to be processed so the ASR becomes optimised for that product or filter.
So lighter ASR with ready training frameworks can be a big advantage, especially with dev communties the size of Pytorch.
$138.00 obviously is a ridiculous amount just for a microphone and amp, but targetted voice extraction doesn’t seem to have equilvalent open source, so Anker have got me paying a premium.
It will be interesting to see how well the Anker alg works in the realworld as it has all the secret sauce that opensource is missing, $138.00 though!
Yes, and one for each room…
Yeah I know, I have been trying to advocate for a lower cost system on a ESP32-S3 that for some reason from the direction others have taken they don’t understand.
I keep repeating how important opensource datasets are for a number of years and we still don’t have them and could be collecting through use…
The Anker is SotA and I have been banging on about targetted & on device training for some time, so posted as thought sod it I will buy one.
Some are prepared to pay $100 for any speakerphone and guess they will but for me also far too much.
As said when recieved I will post results and if it works with Whisper, but really that is down to Whisper and not all ASR.
So the overall audio input and output component quality is paramount or is it more about the software inside these speaker phone devices ? I have a “UE Boom 2” speaker that has a mic in it and the output sound quality is pretty good but I have never used it for a phone call so I have no idea what the microphone is like but if the amp/speaker is anything to go by it should be not too bad. From what I can gather no-one has managed to do something with the alexa hardware have they (yet) ? I have been looking at ESP32 boards but I haven’t got a very good grasp of what is required.
I thought I’d found something that would do for a start-up piece of kit, the Korvo Wroover which was suggested to be at 22 euro but after reading the article and getting ready to buy, the link led to a “no longer available” Aliexpress page.
Its noise where the commercial units excel, not just what they are playing with but third party noise.
This also includes the reverberation that distance to a mic causes that harmonic mixing and can make a waveform look totally different in structure.
Humans have a natural DSP and we filter this automatically and how we focus on directional audio of interest is quite exceptional.
Alexa the bootloader is locked but much is the software algs and models they have.
I think the Korvo is another weird one where kit was produced for algs to be developed that never materialised and hence why “no longer avail”
Sort of similar to the respeaker stuff where they liberally supplied mics and flashy pixel rings without realising multiple mics causes nMics^ computation needs.
The esp32-s3 for much DSP can process near x10 faster that a esp32 ( ESP32-S3-Korvo still exists) but even then is still optimistic as Espressif have only ever released a 2 mic technology demonstrator in the esp32-s3-box, that uses a software approach.
However the esp32-s3 was highlighted as its a lowcost <10$ microcontroller but when it starts to become $30-$50 dev kits it has lost all value as apllication SBC such as Pi compete in that price space.
There is likely enough ‘opensource’ to do both esp32-s3 or Pi like SBC, likely the biggest hindrance is lacking opensource datasets and knowledge to co-ordinate.
Respeaker microphone array and Respeaker Core.
I’ve just picked up these items for next to nothing and I know they will probably be rubbish but that said, what can I do with them ? A quick search and I find everything on the internet is about the V2 whereas mine are V1. Are they any use for anything, (my first goal is a voice controlled wifi intercom).
I seem to remember all the way back to Mycroft days that the ver 1.0 was pretty weak, hence the Ver 2.0 and even then the algs struggled with full channel audio, but that is from memory @fastjack might comment as its vague.
State of the art is prob the Anker S600 that uses a combo of tech, but what it does is use voiceprint voice extraction than what was back then fairly simple beamforming.
I hacked together a delaysum 2 channel that if someone looked at the sources more channels could be added easily.
I paste that as all algs seem to suffer that computation is raised by nMics array. The delay sum part is just that a sum and no load but the TDOA delay is near all the calcs and fairly intensive.
Every mic adds an extra TDOA to get that mic delay for the calc and that is a basis for most.
More mics is computaionally heavy and think that was a prob with the above that even the Ver2.0 was an element of fail.
They also lack co-ordination with the KWS that can give the TDOA of the keyword and in essence lock on to target direction for a command sentence.
So here I am going to likely do one of those baffling long replies that you can stop here if you wish.
SotA (State of the Art) is likely voice extraction than trying to cancel every possible unknown.
Its trained to extract a voice embedding out of a recorded brief silence and then a paragraph, recorded on your phone.
Bluetooth has much going for audio with full 44.1Khz 2channel and multi channel audio.
You get a 5.3 dongle which I did deliberate as the Anker is V5.3 and pair, which means additional units are a another dongle. The extended range of BT 5 and LC3 codec is good, devices are expensive but likely the way.
Opensource and a shout again as the Pi5 would likely blitz the above beamformer as it does with much ML. Its near x6 of a Pi4 due to its V8.2 instruction set especially vectors.
My C/C++ is just pure hacking so someone could and the Radxa Zero with A55 little cores with V8.2 slugs it out with a Pi4.
DietPi - Lightweight justice for your SBC! as a OKdo ROCK ZERO 3W 1GB with Wi-Fi/BLE without GPIO - OKdo will give an often Pi4 beating ML £18.60
The vector instructions in Arm v8.2 is no longer SotA, but is a big improvent (x3?) over A53 counterparts, making them great for ML/DSP.
If your interested in opensource then there are a lot of models especially for RK35xx SBC such as EK3566 (Zero 3) and the RK3558 (OrangePiZer0) that really apart from ecosphere outdoes a Pi because Broadcom seem to of lost there way, with that type of application Soc.
Anything less especially the age of that ver1.0 just remind of a long rocky road.
You can but having the same algs that scale on bigger computers, so you can create and manipulate datasets, training and finetuning much faster than realtime is a dev godsend.
If opensource collated even a modicum of datasets much would be very easy and it constantly blows my mind that opensource with an option to turn on local data collection.
You can capture Keyword and Command Sentences so easily through use that it would be very easy to contribute data and metadata. Some Metadata that would be handy is the region you associate your accent to as that region would be able to finetune in that region.
Age bracket and Gender would be great info again to cherry pick data for finetuning.
You don’t need more and you could but apart from that we have a demographic than any identity, that is for user finetuning and if you opt in to share that with others.
Also I don’t get the current implementation of GitHub - rhasspy/wyoming: Peer-to-peer protocol for voice assistants with it being peer2peer with a protocol that IMO is TWAIN (Technology With an Interesting Name).
It just seems so logical to make Wyoming a Websockets server and have clients that are associated with zones or at least an addition.
What you want is a server that is different to a HomeAssistant one and have the schedulers to max performance in a Race-Till-Idle config.
It slows down when idle but any queues are ASAP.
You can have a pi5 like or above upstream server that can connect via Peer2Peer Wyoming and because the Assistant path is serial its a simple route or queue.
Websockets the underlying bit of Wyoming transmits to distinct datapackets Text/Binary and the API can detect which.
Its a file (json dict?) with markup and your audio without need of much that is current.
Simple route and queue handling are part of the websockets API or at least busy can be detected of stream 1, persist 1, busy states.
Zones are just collections of connections associated with a Room. A Zone is its ID and a Room is the abstraction of whatever collection of connected devices.
Same with BT devices as an array of dongles with BT 5+ coping better with congestion.
Its another radio, client server of just BT audio streams that with no Wyoming or Websockets but still uses the same route and queue of the file and binary data.
The current input audio processing its just more absent than not good.
No, please carry on with the long answers, I learn stuff through them but there should be an emoji with the “it went straight over my head” kind of thing.
Regarding the V1.0 Respeaker, I know it’s mickey-mouse by today’s standards but 7 years ago it was presented as the bee’s knees so I just want to try and do something with it to start my journey without spending anything much. It seems that Seeed have basically ditched the V1.0 and the V2.0 isn’t exactly supported either but back the day, 2017 ish, somebody must have done a few things with these items…
Mine sits on the table and the leds light up in the general direction of where the noise is coming from, it’s quite cool really, but I haven’t managed to get any sound out of it yet.
Seriously its sort pile of poop as when it did work it didn’t work that well.
The respeaker range IMO seemed to be devices to look the part and leave the voice algs for you the dev.
It was a bit like the matrix device as was a multi-mic looking the part, without a big enough cpu to do the beamform compute and so is just multi-mic
Seriously the V2.0 was a big upgrade and that didn’t get great reviews, never mind the V1.
I guess if you root around the respeaker site the orig firmware will be there somewhere
I guess you can usb connect it if you have the firmware but as for audio don’t expect much more than any mic.
As far as I know no-one did do very much with them but the github is still there.
If your talking SotA then Google do actually publish things and sometimes drop a few hints.
Is likely what the use on device in there Pixel tablet.
As a later might hint to that.
https://arxiv.org/pdf/2401.08864
What they are doing is combing spatial cues of beamforming with speech enhancement or voice extraction based on a recorded embedding.
I haven’t been able to find a repo of ready made model of that as just papers seem to exist whilst source code is being withheld.
Sooner or later some source code or models will turn up likely when they have found alternative methods (Also maybe another Google search search might turn something up.
Those early devices had very simple multimics 360 degree sound whilst both Google Nest and the full blown echoes they realised hold on this is a speaker and it faces you unless for conferences.
That means they can cut the number of mics and have forward facing devices massively reducing compute.
Still many are using conference types that work OK but they bounce round to whatever predominant sound / voice they have whilst a assistant will lock onto and beam towards a voice for the full command sentance then reset.
Also the sound seperation in a domestic scenario on average only has 1 clashing noise source such as a vacuum cleaner, TV, HiFi or radio…
So binaural as we humans are and 2 mics is becoming more common on embedded devices.
If your talking a Pi4/5 then there are various algs avail but tend to be a very tight squeeze on a Pi3.
Thats why I mention the Radxa Zero3 as its near Pi Zero2 price but actually outdoes a Pi4 as its A55 small cores are 2 generations up from A53(pi3 ‘small core’) and A73 (pi4 large cores).
The Pi5 is 3 gen up with a A76 large core.
You get to that sort of level and quite a few options are avail but my memory is so foggy that the Respeaker v1.0 had a Xmos soundproceesor but no license for Xmos closed source ?.. I have forgot.
Xmos XVSM-2000 apparently…
I do eventually want SotA but I’ve realised that this is much more complex than I thought and a lot of this stuff is still very much experimental, much more so than casual googling led me to believe so I am not wanting to spend too much and find that the whole SotA thing has moved on before I even get what I’ve bought up and running.
Baby steps at the moment…
I say the next gen SotA might be out before you get that up and running and its still far off.
Only problem is its seems to take much time for SotA to filter down into opensource.
Likely some lateral thought of using what you have to wifi/bt the audio streams to a upstream bigger cpu and handle multiple connections.
Don’t stick your mics on top of a speaker and just have some wireless ones or at least wireless plugged into an amp.
I could post you a load of repos to have a look at for Pi type boards but microcontrollers tend to be much lower level and need quite a high level of knowledge.
Maybe use as is as a wifi streamer, dunno about audio it has but has pins but maybe line level.
Xmos XVSM-2000 you have, there closed source paywall SDK you don’t.
I don’t doubt that you are right but I am a pig-headed sort of a person and I can’t stop myself from wasting my time with this respeaker…
At the moment I am up sh*t creek because the firmware is no longer where it is supposed to be (on One drive apparently…) I am missing the “Juci frontend, PocketSphinx, Python libraries and Mopidy” but I don;t know how to get over this enormous obstacle.
GitHub - mkschreder/juci: JUCI JavaScript Webgui for embedded devices running OpenWRT just google but how they where implemented who knows.
Thanks for that but I am struggling to actually make sense of it all. I’ve been googling till my brain hurts…
It seems the original firmware has the missing packages but then the updated firmware that suits the Meow King speaker and battery unit doesn’t have these items to “save space” but without them I don’t know how it’s supposed to do all it’s magic. I am getting there slowly but I think I need the original firmware which is no longer to be found and the ReSpeaker page is no longer supported. If I can get something out of it with the original firmware and a separate speaker then I can try and get the Meow speaker working later then eventually use it as a satellite to the RPi 4 some day in the future.
I think ‘designed for’ means its a a 4 ohm 5 watt speaker in a housing that fits the case.
Thing is I don’t think it ever did its magic, I can remember a user had one on the Mycroft forum but that doesn’t exist anymore.
Enjoy as nothing I can add as its an old obselete unit that the demo video’s never show it working and used it as a fruit piano instead as the touch buttons did, which speaks volumes at least.
I just can not remember, but have a vague recollection it got a bad review by the Mycroft user, but what was said … dunno can not remember.
I know it’s a total pile but it did have some sort of functioning directional mic thing going on before i started messing. It would light up the leds in the general direction of the person speaking…
I would be just looking to get it doing that again and outputting sound to the Meow King thing and it would give me the inspiration to do something “proper” with the Pi after. Then the ReSpeaker can be a satellite: I definitely need at least one satellite as the first thing I want to do is have the voice-operated intercom that used to work on the Alexas.
Edit: It still does the directional mic but I had the 7-way mic thing disconnected so it wasn’t doing it… DOH!
I assume that function is built in to the Mic array ?
Yeah that must be the xmos the TDOA (time difference of arrival) 2ch_delay_sum/tdoa.h at main · StuartIanNaylor/2ch_delay_sum · GitHub is usually calculated and if delay sum, it is just the sum of the calculated delay.
That is with broadside arrays, when you have a mic behind a mic (endfire) I think its sum again but delay seignal is inverted.
Haven’t a clue with the geometry you have and maybe uses a more complex beamformer such as GSC or MVDR.
Its so long back that I remember very little.
Don’t laugh but I’m looking at a Seeed Snips now. Apparently Seeed sold it to Sonos and they blocked it or something. Anyway, I can buy it for peanuts but is it any better/easier to make use of than the Respeaker ? Sorry for the idiot questions and I know this thread has become the exact opposite of “State of the Art” for which I apologise.
Its ok as Rhasspy or HA doesn’t have any input audio processing, its completely lacking SotA.
Seed snips is just a Pi3 B+ and a Respeaker 2 mic hat and there used to be some C libs and api documentation, that I am not sure if it exists anymore. Again it was another thing about licensing, but my memory fades.
You have gone full circle now, where you can use a Pi3-B+ or even a PiZero2, but a £18 Radxa Zero 3 will give Pi4 like performance.
You don’t have to have Snips, it was just software in fact the Respeaker 2 mic hat as due to it being a hat its really awkward to isolate the mics with many users having them pointing upwards.
The 2 channel ADC from Plugable USB Audio Adapter – Plugable Technologies which allows you to mount x2 https://www.aliexpress.com/item/1005005348094991.html as the Max9814 has a great analogue AGC as they mount using a rubber grommet.
I can do all this with various methods the problems really are upstream, but from beamforming, BSS to filters, much is avail and possible just not particularly SotA but far better than the complete absense of audio processing Rhasspy and HA have.