Mimic 3 TTS Preview

Announcements
1

Hi everyone,

Today, Mycroft is opening up a beta test for Mimic 3 TTS. This is a spiritual successor to Larynx, with better performance and more supported languages than Larynx. It runs about 2x faster than real-time on a Pi 4 (64-bit OS), and I’m hoping this will replace Google TTS for a lot of people :slight_smile:

I’m looking for feedback, specifically on the non-English voices (there are a total of 25 languages). You can sign up for the beta using the link above, or just listen to some voice samples. PM me if you’d like to get access to the code or Debian packages to try out locally. There will also be a Docker image available when it officially launches (probably next month).

If you’d like to learn more, the documentation is already up. And here is the list of currently supported languages:

  • Afrikaans
  • Bengali
  • Dutch
  • English
  • Farsi
  • Finnish
  • French
  • German
  • Greek
  • Gujarati
  • Hausa
  • Hungarian
  • Italian
  • Javanese
  • Kiswahili
  • Korean
  • Nepali
  • Polish
  • Russian
  • Setswana
  • Spanish
  • Telugu
  • Ukrainian
  • Vietnamese
  • Yoruba

Thanks,
Mike

8 Likes
2

Good work, the Dutch voices are pretty good. I’ll check on the documentation!

1 Like
3

Hi there, also the German examples sound interesting, so I’d also like to take part in testing.

What kind of additional information shall be provided in the PM (beside the info one wants to participate)? Rhasspy is running as debian package here, so I’d need access to the (amd64) deb package…

2 Likes
4

@synesthesiam Awesome! :+1:

Small issue on the french voices for the word “de” though.

C'est un arc de cercle is read as C'est un arc dam cercle.

In the Rhasspy French Kaldi base_dictionary.txt, the word “de” has these pronunciations :

de d a m
de d e
de d ə
de d e ø

The first one (d a m) looks incorrect to me and probably is the cause of this issue.

Hope this helps.

3 Likes
5

I don’t know about the other languages, but some of the English ones are really good. My favorite was hifi-tts_low ID 2. But cmu-arctic_low 7, 9, and 11 were also great, in my opinion. I didn’t listen all of the last group. There were a ton of them.

1 Like
6

Thanks everyone for testing and the feedback!

One thing that seems consistent is problems with gruut for non-English languages (French especially). Because Mimic 3 is licensed under APGLv3, I’m considering re-training some voices with eSpeak phonemes instead. @fastjack do you think eSpeak pronunciations are better for French than gruut?

No additional info, I’ll just send you a Google Drive link. The project is open source, so there’s nothing to hide. I’m just trying to keep the feedback at a level where I can respond and fix things quickly :slight_smile:

Thank you! I will get this fixed right away.

Thanks! Let me know if you’d like to try it locally. It runs pretty well on a Pi 4 (64-bit), and very fast on a desktop/laptop.

2 Likes
7

I’ve never noticed any specific issue with IPA phonemes (using the Rhasspy Kaldi French profile).

What kind of problems did you encounter with Gruut for the french language?

8

The is mispronunciation of “de” is one of the problems, but I’m sure there are others. I was just curious if it would be worth it to use eSpeak as a phonemizer instead of gruut.

1 Like
9

Listening to Mimic 3 french samples the only other issue I can hear is the missing liaison between the C'est (s e) and the un (œ̃'') in C'est un (s e / œ̃'') which should vocalize the t (s e t / œ̃'') when it is followed by a vowel.

2 Likes
10

Sure, I’ll try it on my Chromebook Tablet.

1 Like
11

Hm, what happened to the harvard-glow_tts voice? That’s the one I’m using and it’s a bit difficult to compare quality between different voices.

I found a couple of possible voices and they seem to have improved where Larynx had trouble, like “thirteen” coming out as “thirthy” and the name “Maria” coming out as “May”, but each of them is somewhat unique and they might have other issues. Also some of them seem to still have trouble with “thirteen” and “Maria”.

I think when the Docker image comes out I’ll do further tests with my actual Rhasspy. I tried putting the beta URL as Remote HTTP but it didn’t like that at all. It crashed on restart until I removed the URL from profile.json.

1 Like
12

Nice work ! Well done !
I tested french voices.

I agree there are some issue on phoneme due to Gruut. A try with an other phonemizer could be a solution.

Also, the first voices, m-ailabs_low, are ending too fast. The sound of the last letter is cut.

Else, voices are really nice.

1 Like
13

I hadn’t trained that one just yet. I can put it on the list, though :+1:
Thanks for testing the other voices. I wonder what it is about “thirteen” and “Maria”.

Thanks! Dare I ask your favorite French voice :wink: ?

Yeah, I’m having a lot of trouble with those voices for some reason. I think they may just work better split out as single speaker models, maybe with eSpeak.

14

This is certainly very interesting. Although I’m not a Russian native speaker, I did take a little listen to the first Russian sample. I’m just a student, and the monologue was beyond my level, but it sounded in the ballpark to me.

I am curious, is the eventual plan to provide this as one of the options in Rhasspy, or is it something that would be used externally?

15

“thorsten drunk” - best voice ever :laughing: :heart:

1 Like
16

Using Pi4 64bit mimic3-server.

m-ailabs_low:

  • On short sentences or just single words, most of the times there is a strange noise at the end of the voice output. Not sure, why this happens. If the text gets longer, everything is ok (most of the times).
  • There is a nice and realistic speaking-break after a comma. But no break after the end of a sentence. That makes the text hard to understand.

Should the API of mimic3-server already be functional?

17

I tested German and English voices on a smartphone running Mobian (Pocophone F1). Impressive speed and quality!

18

I think Siwis_low is the best one. But this voice is very close to voice use in public transport in Toulouse. So I don’t like it much. In fact it sounds like an actress reading a book. So good voice but not natural.

Else, I would say zeckout, I feel like I’m listening to an old science teacher.

Edit: I re-listen voices with a good headset, Siwis low is better than the other in terms of quality.

19

I don’t think there is anything special about those words, it’s just that they happen to be in my voice outputs and Larynx has issues with them. There are probably many more words with issues, I’m just not using them in any voice prompts.

1 Like
20

I’d agree in terms of audio quality. Many of the voices I trained from the M-AILabs dataset don’t have great audio quality since they were recorded by volunteers for Librivox using whatever hardware they had.

Thank you! Was it difficult to get working on the phone at all?

This seems to be a general problem with the TTS model I’m using. If the dataset doesn’t contain the speaker saying single words or very short phrases, the model has a hard time producing them. For now, I think I’ll have to consider the M-AILabs voices as intended for reading long-form text only :confused:

I think I can at least fix the pausing issues after a period for now :+1:

Yes, if you’re running it locally you can check out http://localhost:59125/openapi/ to see what’s available. It should also be compatible with anything that’s meant to talk to MaryTTS. You just have to make sure your “MaryTTS voice” is something like “en_UK/apope_low”.

21

Was it difficult to get working on the phone at all?

No. It was as easy as for a Debian or Ubuntu computer.

1 Like
22

Hi there, got the server up and running (manualy for now).

Settings are saved (wrt. to maryTTS) as follows:

    "text_to_speech": {
        "marytts": {
            "voice": "thorsten_low"
        },
        "system": "marytts"
    },

The other keys mentionned in docu / text-to-speech/#marytts are not explicitely stored in the JSON, but visible in the Rhasspy UI (de_DE, thorsten_low).
Putting that combined in the “Voice” field doesn’t help changing back leads to the locale also beeing stored in the JSON, but still this results in

TtsException: file does not start with RIFF id

What did I miss or could do better?

Tests with “http://external-ip:59125/” work quite good, calling with “openapi” postfix results in 404 error…

23

@synesthesiam
I like the announcement! :slightly_smiling_face:
Is there already a date when the mimic3 repository will be online in the Github? I would like to test the Debian packages.

Greetings, Jens

24

Wow, how could Portuguese (Brazilian) not be on the list?

25

I’ll have to check this myself. The Mimic 3 server should also work with Rhasspy’s “remote TTS” option, but I need to double check I haven’t broken anything with that either!

Hopefully next month, but I sent you a link with the beta packages :slight_smile:

It was, but people told me that the voice I trained wasn’t understandable. I used this dataset: https://github.com/Edresson/TTS-Portuguese-Corpus

Do you know of any other TTS Portuguese datasets?

2 Likes
26

sorry no. I’m clueless about lang models, data, etc.

27

did you look here? Hugging Face – The AI community building the future.

28

Nice! :+1:

    "text_to_speech": {
        "command": {
            "say_arguments": " --ssml --voice 'de_DE/m-ailabs_low#rebecca_braunert_plunkett' ",
            "say_program": "mimic3"
        },
        "satellite_site_ids": "default",
        "system": "command"
    },

Das ist ein Test in deutsch <voice name="en_US/vctk_low#p236">and this is an test in english.</voice>

… and Rhasspy speaks two languages in one sentence - cool. :sunglasses:
It runs a bit slow on my old machine without GPU. With enough power and cache it will definitely get better.

Greetings, Jens

4 Likes
29

I didn’t, but I don’t see any useful data there :frowning:

Awesome! The way to speed this up is to run mimic-server as a service (check the source code for a systemd unit example), and then use mimic3 --remote ... so it will use the web server instead.

30

Calling it up via the web interface wasn’t faster either. Now I have to pimp my base a bit first…

31

Btw, I think Mycroft should link to some demo’s in their Mimic 3 blog post announcement.
If people could hear presumably how good the TTS sounds they’d be more likely to sign up and get involved. My 2 cents.

1 Like
32

Will this be a drop in replacement?

33

Hello,

unfortunately i cannot sent PM as a new user and therefore cant test RTF’s for different architectures. Can someone give hints about RTFs, maybe for ARM?

Thanks

34

Hi @The1And0, on 64-bit ARM you can get an RTF of around 0.5. 32-bit ARM is slower, around 1.2 or 1.3. If you’re on a 64-bit x86/64 machine though, it can be 10x faster than ARM :slight_smile:

Try it out for yourself: https://github.com/mycroftAI/mimic3

35

Oh, there’s a Docker image now. :+1: (Although apparently without harvard-glow_tts yet?)

Is it compatible with the “Remote HTTP” TTS option of Rhasspy?

36

It is! Just set this as the URL: http://localhost:59125/api/tts

You can change the voice like this: http://localhost:59125/api/tts?voice=en_US/vctk_low#p236

2 Likes
37

I think I’m missing something with the docker image.
After running the image I can access the web server but I cannot synthesize voices.

My guess is I have to still manually download the voices. But how do you do it using the docker image?
I tried locating the mimic3-download command in the image but no luck.

ERROR:mimic3_http.synthesis:Error during inference
Traceback (most recent call last):
File “/home/mimic3/app/mimic3_http/synthesis.py”, line 125, in do_synthesis_proc
result = do_synthesis(item, mimic3)
File “/home/mimic3/app/mimic3_http/synthesis.py”, line 81, in do_synthesis
raise e
File “/home/mimic3/app/mimic3_http/synthesis.py”, line 61, in do_synthesis
mimic3.speak_text(params.text, text_language=params.text_language)
File “/home/mimic3/app/mimic3_tts/tts.py”, line 368, in speak_text
voice = self._get_or_load_voice(self.voice)
File “/home/mimic3/app/mimic3_tts/tts.py”, line 579, in _get_or_load_voice
voice = Mimic3Voice.load_from_directory(
File “/home/mimic3/app/mimic3_tts/voice.py”, line 283, in load_from_directory
onnx_model = Mimic3Voice._load_model(
File “/home/mimic3/app/mimic3_tts/voice.py”, line 403, in _load_model
onnx_model = onnxruntime.InferenceSession(
File “/home/mimic3/app/.venv/lib/python3.9/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py”, line 335, in init
self._create_inference_session(providers, provider_options, disabled_optimizers)
File “/home/mimic3/app/.venv/lib/python3.9/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py”, line 370, in _create_inference_session
sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
RuntimeError: /onnxruntime_src/onnxruntime/core/platform/posix/env.cc:183 onnxruntime::{anonymous}::PosixThread::PosixThread(const char*, int, unsigned int ()(int, Eigen::ThreadPoolInterface), Eigen::ThreadPoolInterface*, const onnxruntime::ThreadOptions&) pthread_setaffinity_np failed, error code: 0 error msg:

38

Connect to the docker image
sudo docker exec –it nginx-test /bin/bash as if the container name was nginx-test but change name to the container name
Its like ssh as you connect to the docker image and logon as by default your will be root
then do as you would do on a host wget and you may need to install wget as you would do ‘apt-get install’

Docker to connect its always sudo docker exec –it <container-name> /bin/bash

39

That’s what I meant by running the mimic3-download command ‘from the image’ :smile:
Should’ve specified I was running it through docker exec /bin/bash

40

@tipofthesowrd You may need to run this before running the Docker image:

mkdir -p "${HOME}/.local/share/mycroft/mimic3"
chmod a+rwx "${HOME}/.local/share/mycroft/mimic3"

The Docker image runs as an unprivileged user for security, so it may not have permission to download voices.

41

I can see it downloading the voices and afterwards throwing the following errors & logs. When running docker exec -it /bin/bash I can enter the folder and cat the phonemes.txt file for example. So read permissions seems to be OK

ALIASES: 100%|███████████████████████████████| 25.0/25.0 [00:00<00:00, 45.0kB/s]
LICENSE: 100%|█████████████████████████████| 6.88k/6.88k [00:00<00:00, 16.6MB/s]
README.md: 100%|███████████████████████████| 11.2k/11.2k [00:00<00:00, 6.89MB/s]
README.md.in: 100%|█████████████████████████████| 186/186 [00:00<00:00, 367kB/s]
SOURCE: 100%|█████████████████████████████████| 52.0/52.0 [00:00<00:00, 332kB/s]
VERSION: 100%|███████████████████████████████| 6.00/6.00 [00:00<00:00, 30.7kB/s]
config.json: 100%|█████████████████████████| 3.57k/3.57k [00:00<00:00, 39.0kB/s]
generator.onnx: 100%|██████████████████████| 59.9M/59.9M [00:27<00:00, 2.26MB/s]
phoneme_map.txt: 100%|███████████████████████| 21.0/21.0 [00:00<00:00, 34.4kB/s]
phonemes.txt: 100%|█████████████████████████████| 329/329 [00:00<00:00, 296kB/s]
ERROR:mimic3_http.synthesis:Error during inference
Traceback (most recent call last):
File “/home/mimic3/app/mimic3_http/synthesis.py”, line 125, in do_synthesis_proc
result = do_synthesis(item, mimic3)
File “/home/mimic3/app/mimic3_http/synthesis.py”, line 81, in do_synthesis
raise e
File “/home/mimic3/app/mimic3_http/synthesis.py”, line 61, in do_synthesis
mimic3.speak_text(params.text, text_language=params.text_language)
File “/home/mimic3/app/mimic3_tts/tts.py”, line 368, in speak_text
voice = self._get_or_load_voice(self.voice)
File “/home/mimic3/app/mimic3_tts/tts.py”, line 579, in _get_or_load_voice
voice = Mimic3Voice.load_from_directory(
File “/home/mimic3/app/mimic3_tts/voice.py”, line 283, in load_from_directory
onnx_model = Mimic3Voice._load_model(
File “/home/mimic3/app/mimic3_tts/voice.py”, line 403, in _load_model
onnx_model = onnxruntime.InferenceSession(
File “/home/mimic3/app/.venv/lib/python3.9/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py”, line 335, in init
self._create_inference_session(providers, provider_options, disabled_optimizers)
File “/home/mimic3/app/.venv/lib/python3.9/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py”, line 370, in _create_inference_session
sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
RuntimeError: /onnxruntime_src/onnxruntime/core/platform/posix/env.cc:183 onnxruntime::{anonymous}::PosixThread::PosixThread(const char*, int, unsigned int ()(int, Eigen::ThreadPoolInterface), Eigen::ThreadPoolInterface*, const onnxruntime::ThreadOptions&) pthread_setaffinity_np failed, error code: 0 error msg:

ERROR:mimic3_http.app:/onnxruntime_src/onnxruntime/core/platform/posix/env.cc:183 onnxruntime::{anonymous}::PosixThread::PosixThread(const char*, int, unsigned int ()(int, Eigen::ThreadPoolInterface), Eigen::ThreadPoolInterface*, const onnxruntime::ThreadOptions&) pthread_setaffinity_np failed, error code: 0 error msg:
Traceback (most recent call last):
File “/home/mimic3/app/.venv/lib/python3.9/site-packages/quart/app.py”, line 1512, in full_dispatch_request
result = await self.dispatch_request(request_context)
File “/home/mimic3/app/.venv/lib/python3.9/site-packages/quart/app.py”, line 1557, in dispatch_request
return await self.ensure_async(handler)(*request_.view_args)
File “/home/mimic3/app/mimic3_http/app.py”, line 225, in app_tts
wav_bytes = await text_to_wav(
File “/home/mimic3/app/mimic3_http/app.py”, line 102, in text_to_wav
wav_bytes = await future
File “/home/mimic3/app/mimic3_http/synthesis.py”, line 125, in do_synthesis_proc
result = do_synthesis(item, mimic3)
File “/home/mimic3/app/mimic3_http/synthesis.py”, line 81, in do_synthesis
raise e
File “/home/mimic3/app/mimic3_http/synthesis.py”, line 61, in do_synthesis
mimic3.speak_text(params.text, text_language=params.text_language)
File “/home/mimic3/app/mimic3_tts/tts.py”, line 368, in speak_text
voice = self._get_or_load_voice(self.voice)
File “/home/mimic3/app/mimic3_tts/tts.py”, line 579, in _get_or_load_voice
voice = Mimic3Voice.load_from_directory(
File “/home/mimic3/app/mimic3_tts/voice.py”, line 283, in load_from_directory
onnx_model = Mimic3Voice._load_model(
File “/home/mimic3/app/mimic3_tts/voice.py”, line 403, in _load_model
onnx_model = onnxruntime.InferenceSession(
File “/home/mimic3/app/.venv/lib/python3.9/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py”, line 335, in init
self._create_inference_session(providers, provider_options, disabled_optimizers)
File “/home/mimic3/app/.venv/lib/python3.9/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py”, line 370, in _create_inference_session
sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
RuntimeError: /onnxruntime_src/onnxruntime/core/platform/posix/env.cc:183 onnxruntime::{anonymous}::PosixThread::PosixThread(const char, int, unsigned int ()(int, Eigen::ThreadPoolInterface), Eigen::ThreadPoolInterface*, const onnxruntime::ThreadOptions&) pthread_setaffinity_np failed, error code: 0 error msg:

42

Are you running this on a shared (cloud) machine, by chance?

43

Wow, how could Portuguese (Brazilian) not be on the list?

Wow, how could you be any more obtuse?

44

Greetings sir,
Is there any chance to access the training source code of mimic-III model?

45

I’ve uploaded the code here: GitHub - MycroftAI/mimic3-train: Mimic 3 Training Code

I can’t guarantee that it’s in a working state, unfortunately. It was being refactored when Mycroft went under, and I never got a chance to finish it.

46

Greetings sir,
I have another request to make, if it’s OK.

I intended to know the way to add extra words and phonemes to the dictionary files of mimic3. Can you please guide me with that?

47

Mimic 3 uses gruut for phonemization. You can see the structure of the English data here, for example: gruut/gruut-lang-en/gruut_lang_en at master · rhasspy/gruut · GitHub

Adding words involves modifying the lexicon.db sqlite3 database. There is a word_phonemes table that you can add words/phonemes to. Each language is installed as a separate Python package, such as gruut-lang-en. You can create a new package with your updated lexicon.db or modify the one in your virtual environment.

48

Sorry to interrupt，As a Chinese, I’m glad that rhasspy supports Mandarin, but I’m frustrated that its TTS part only supports pocketsphinx, and its recognition accuracy is very low, so low that it can’t be used properly. I want to know if there is any way to solve this problem, can I change the model or train or what, I hope there are good people who can teach me by hand, thank you very much.