STT Speeds on local pi setup

Help
1

I just finished setting up my environment including homeassistant pocketsphinx and a deconz interface. However the decoding times of the Pocketspinx Decoder are arround three to five seconds, while the rest just takes milliseconds. Terminating homeassistant and deconz running on the pi doesn’t seem to change much.
I know i could include a home server for these computations for speed up but since I’m trying to do everything on one pi I was wondering if there are any ways of speeding up the process.

Are my times average or unusually high, what about your computation times?
Running on a Pi 3B+ btw…

2

By my experience I have similar times with Raspberry Pi 3b+ but It could depend from microphone or environment noise and size of wav registration (more seconds, more size).
I am trying two different setups:

  • Respeaker v1 (low cost) microphone, 2/3 seconds
  • Respeaker v2 (optimized) microphone, 3/5 seconds (probably slow for built-in algorithms or usb)

I have similar times with Raspberry 2 but I will try with Raspberry 4.

Often It happens that I stop to speak but the registration Is still Active becouse of environment noise. To resolve this I tried to decrease the timeout of the “command” section.

Suggestions are appreciated :slight_smile:

PS for good comparation we should specify our environment, I am using docker

3

I get the same with my RPi3b. Takes three / four seconds…

4

I’m on a RPi3b running Hassio - as Mic I’m using the PS3 Cam - Decode Times are 1.5 - 2.5 seconds

5

Regarding decoding time, are you using Kaldi? Using a smaller acoustic model (TDNN-250 instead of TDNN-F) reduced decoding times for me without any impact on accuracy.

@synesthesiam Maybe the smaller acoustic model can be provided as default?

Online decoding should also help to speed things up when Rhasspy supports it natively.

6

How to change the acoustic model?

7

On my rpi 3b+ with Kaldi I have to wait 4 seconds…
But here are some good news :nerd_face:

8

Until @synesthesiam updates the model provided for Kaldi profiles with smaller ones (as I think he will :wink: ) and if your language has such model (english, german and french do), you can simply replace the files in the {profile_dir}/kaldi/model/model folder by the TDNN-250 model files:

  • cmvn_opts
  • den.fst
  • final.mdl
  • normalization.fst
  • tree

The models should be available here:

Hope this helps.

1 Like
9

It does. Thanks a lot @fastjack

10

Maybe you could create a new topic showing how to install kaldi, and set it in Rhasspy ?
Not sure to have it all :wink:

11

There is not much to do I afraid… When your profile is all setup using the Kaldi Rhasspy profile, simply download the model from the link and swap the mentionned files. No config changes necessary (maybe retrain though?).

I pretty sure @synesthesiam will include these models natively in the near future as the narrow and specific language model trained by Rhasspy mitigates the “lightness” of the acoustic model.

12

To install Kaldi you just do that, then ?
https://kaldi-asr.org/doc/install.html

13

Ah I see what you mean… I did not build Kaldi as I’m using the Docker image so Kaldi is already installed with Rhasspy. The model files are downloaded by Rhasspy into the profile folder.

If you are not using Docker, you will have to build Kaldi using the link you provided (can be pretty complex though as Kaldi does not easily build on ARM).

14

Ah didn’t know that !! I use docker, so I will give it a try ! Thanks :beers:
Does number (1…100) builtin works with kaldi ?

15

Everything works with Kaldi :wink:
Much better than Pocketsphinx.

3 Likes
16

Thanks again, now I know what to do this evening :rofl:

17

Hi @fastjack
Thanks for sharing… will try it
But do I need to replace Kaldi files after every rasspy update?

18

I’m afraid so… If you download the Kaldi profile using the web UI, you’ll have to replace the file again. This is a quick fix to speed up the ASR transcription.

1 Like
19

Does kaldi support italian language?

20

Not now as it seems: https://rhasspy.readthedocs.io/en/latest/reference/

21

I haven’t found a Kaldi model for Italian yet, unfortunately. If we can get a big enough speech corpus, we can train one. The Zamia English model was trained on about 1200 hours of audio.

22

That’s certainly the truth. Using Kaldi, I get about 98-to-99% accuracy.
With pocketsphinx, it was barely 50% at best. :frowning:

23

Strange, I got very good intent catching with pocketsphinx, near 100%.
wakeword false positive is another story and my main problem with rhasspy actually … :sleepy:

24

Sounds like you have sensitive mic
Do you use snowboy?
Can you try to set audio_gain less than 1?
Or just play with sensitivity parameter

25

Pocketsphinx works fine as long as the intents and slots are simple.

From what I experienced, as soon as you start adding complexity it breaks down and the WER decreases drastically.

26

I’ve found that Piero Cosi used italian model in his Kaldi thesis.
I can try to contact him, maybe he has a model.

27

Based on a quick look at the paper, his model was trained on about 10 hours of children’s speech. Mozilla Common Voice has about 91 hours available. Maybe Piero is aware of other Italia speech corpora (preferably adult speech)?

28

I’m asked to him…waiting for an answer :slight_smile:

2 Likes
29

No fr model then ? :sleepy:

30

Don’t despair… here you go for the French model:

:wink:

31

Thanks ! Got Kaldi working now.

Not convinced anyway that it is better than pocketsphinx after a few try, but will keep it for a few days of testing.

Will see, still fighting with snowboy actually, really have to find some good settings.

32

Tried your Suggestion while running in Virtual Environment. Kaldi in general takes arround 8 Seconds for processing. With your tweaks the processing time is 2 Seconds but no text is recognized :confused:
Guess I’ll keep using Pocketsphinx (4 Seconds Processing) until the mentioned speed bump for Kaldi is released.

33

Piero answers me that he has a lot of models and he asks to me what of those models/files we need:

HMM
tri1 : first triphone system (delta+delta-delta features)
tri2 : an LDA+MLLT system
tri3 : Speaker Adaptive Training (SAT) system

SGMM2 Training
SGMM2 + MMI Training

DNN
Hybrid System (Dans DNN)
Combination SGMM + Dans DNN
Hybrid System (Karel’s DNN)
Hybrid System (Karel’s DNN), sMBR training

What of those does we need?

34

I’ve just tried it and have the same result((
@fastjack could you explain what you did step by step?
Btw, in my original kaldi folder I haven’t found normalization.fst and den.fst… so looks like I’m missing something🤔
The folder where I put the new files is rhasspy/profiles/en/kaldi/model/model

35

Which language model did u download? a german one? Got this issue as well.

36

Little bit of OT, but is there a guide for setting up kaldi with rhasspy as hassio addon? :slight_smile:

37

No, I’m using en model

There is nothing special… just select Kladi in Rhasspy settings. Then you will be asked to download files, and after that you will need to train. That’s it :nerd_face:

38

The option is greyed out in my case - i guess because i’m using a german profile?

39

I’ve heard about this bug, but I can’t reproduce it. The German profile definitely supports Kaldi, but the profile setting seems to be flipped in the web interface unless you start with English and then switch over.

40

i did not start with an english profile and had no problem selecting Kaldi for speech to text on a german profile. It works way better then pocketsphinx and almost always recognizes my senteces correctly. But quite a bit slower then pocketsphinx on a pi4 (nearly immediatly vs 2-3s)

41

@Sikk
Could you try add this in advanced settings?

    "speech_to_text": {
        "system": "kaldi"
    }
42

hey @frkos
i have it enabled by now :slight_smile:

but struggling with other problems right now

KaldiDecoder	Missing HCLG.fst	Graph not found at /share/rhasspy/profiles/de/kaldi/model/graph/HCLG.fst. Did you train your profile?

Already tried to delete everything and let it download again…

maybe i’m doing something wrong with the paths

    "kaldi": {
        "base_dictionary": "base_dictionary.txt",
        "custom_words": "custom_words.txt",
        "dictionary": "dictionary.txt",
        "kaldi_dir": "/share/rhasspy/profiles/de/kaldi",
        "language_model": "language_model.txt",
        "model_dir": "kaldi/model",
        "phoneme_examples": "phoneme_examples.txt",
        "phoneme_map": "espeak_phonemes.txt",
        "unknown_words": "unknown_words.txt"
    },
43

Hi @Sikk

Did you train your profile?
Btw, I have no settings for Kaldi, just the code I posted above.
Try to delete your settings, Rhasspy will use default ones in that case

1 Like
44

Thats the solution :slight_smile:

It’s working now - seems to work really well - but takes ages to transcribe now, on raspi 3B :slight_smile:

45

Great!
Yes, it takes about 9 seconds for me too

I’m waiting for Kaldi online encoding option, which I believe will be implemented sometime :nerd_face:
Hi @synesthesiam I remember it was on a roadmap, is it still there? :nerd_face: :see_no_evil: all other thing in Rhasspy works absolutely perfect for me :partying_face:

46

Yes :slight_smile: This has been successfully implemented in version 2.5. I’m also switching all the Kaldi models to the faster 250 versions, so this plus online decoding should make things much faster :monkey_face:

3 Likes
47

The best morning news I have ever read :upside_down_face:
Thank you

48

That seems to be a german language problem. I saw this several times now and had this error, too.

49

Jeah I guess so now. I’m having the same issue. Will try to figure out tomorrow how to reproduce it so it’s easier to fix.

50

Basically this is an issue with saving and loading settings I guess. Almost every time I’m saving something in my settings and rhasspy restarts afterwards the en profile is shown and all default settings are set. The only way I’ve figured out to work around is to restart the docker container. At first I thought that my browser is caching something, but opening the webUI using a private mode window, closing it and open a new one isn’t changing anything after the restart.