Collaboration with Jaco-Assistant project

Development
1

Hi @synesthesiam,

first of all, I would like to congratulate you for the great progress Rhasspy did make since last year.
With the end of Snips I was looking for a replacement, which should have been able to be extended with custom skills, but a that time I wasn’t satisfied with the existing solutions, so I decided to build my own assistant called Jaco.
Like with Rhasspy, the code is completely open source, but Jaco has the focus on usage and extension with skills, in difference to Rhasspy’s focus on SmartHome integration.

Currently Jaco can run on almost and Linux computer or a Raspberry Pi 4 and understands English, German, Spanish and French. It also did show great results in some benchmarks I did run, like this one:

compare_barrista
compare_barrista1920×1440 200 KB

1 Like
2

I did check out Rhasspy recently again and with the plans you described in your Master Plan and with the ideas @maxbachmann and @koan did mention for secure skills, I thought it would be a good idea if we could work together on problems we both face.

Regarding the different modules like STT, NLU and TTS required for an assistant, I did focus on one solution for each. For TTS I’m using PicoVoice, like Snips did, and for NLU I’m using Rasa which I also did port to work on RasPis. For STT I did train own models using DeepSpeech. All of those come with a prebuilt container image, for easy installation.

For the skills I did try to make installation as well as creation as easy as possible, without restricting possibilities. You can add utterances in a simple markdown style and run any code you want. Communication with the rest of the assistant is done over a small python interface, using MQTT. To get rid of the restriction Snips had, that you can only use python code, and to still keep things secure, all skills run in a separated container, built locally on the device (with Podman that containers don’t run with root permissions).
The skills also use a permission system, so that they can only access special hardware features or external topics if they mentioned them in their config file.
Training of STT language model and the NLU model is done locally on the device (normally does take a few seconds on a computer or a few min on a RasPi).

3

For easy access to skills and sharing them with others, I did build a skill store (can also run offline if needed):

store
store1708×967 264 KB


You can find the project here:

I did try to make the setup as easy as possible, but I believe it’s a bit more complex than the one of Rhasspy, which has a nice GUI:) The installation does take about 30-60 min, depending on your hardware and internet speed. Jaco also should be able to use multiple satellites, but I didn’t test this yet.


Regarding a collaboration, I would suggest, that we first try to join our utterance definition syntax, that we can more easily interchange the different modules of the assistants. It’s already quite similar, but not to the full extend. Maybe we then can make the skills (or at least some of them) runnable with Rhasspy too.
I’m also interested in your larynx TTS project, while I can offer support with STT trainings in exchange.


What do you think about it?

Greetings,
Daniel

4

I’m maintaining a Docker image for Rasa on ARM, I’m using it on my Raspberry Pi:

For now still pre-Rasa 2.0 and only 32-bit.

How are you running Rasa on a Raspberry Pi?

This all sounds awesome, exactly like I envisioned how a secure app architecture should look like! I’ll try to have a closer look at your project next week.

5

Yes thats what I envisioned for the app architecture aswell. I really like, that the store shows warnings for these things (access to different topics, internet connection, use of specific devices, memory usage …).

6

In about 180 lines Container file. This did take me a lot of time :see_no_evil:
Thanks for your bazel-on-arm project by the way:)
You can check it out here: https://gitlab.com/Jaco-Assistant/Jaco-Master/-/tree/master/nlu-parser or download the prebuilt image from there: https://gitlab.com/Jaco-Assistant/Jaco-Master/container_registry

1 Like
7

Dear @DANBER,

First thing to say… this skill store looks awesome :+1:

I would love to see a collaboration of Rhasspy and Jaco with the goal to fuse the knowledge and create very flexible and secure offline voice assistants with skill stores. I think this is a win-win situation and the result could inspire a lot of programming beginners to give it a try and start with their first voice project without the fear of having to create all the skills by themselves or risking the security of their privacy.

1 Like
8

This is some incredible work, @DANBER!

I would definitely be interested in joining forces (perhaps @sepia-assistant would be interested as well). If we can make our utterance syntaxes compatible and our containers interoperable, this could be quite an impressive system :slight_smile: Your skill store also looks awesome; this is definitely something that could benefit the whole community.

Do you have a document describing your utterance syntax? Perhaps we could author a shared standard here. Part of Rhasspy’s syntax complexity comes from not having a very sophisticated NLU component (basic graph search). I went this direction to keep things more language independent, but it wastes a lot of time with things like numbers and dates.

If possible, how would you propose making our containers interoperable? It looks like Jaco is using something similar to the Hermes protocol. Maybe a translation service?

Your use of podman here is especially intriguing; I’d like to transition Rhasspy to it eventually and use podman compose instead of having one big container. I have some working code already that uses the settings from Rhasspy’s GUI to generate a Docker compose file. I imagine this could be made to work with podman too.

Larynx uses MozillaTTS, but I’ve been able to shrink down the models by using a small phoneme set per language instead of one big set across all languages. My hope is to train TTS models for all of Rhasspy’s STT languages.

It’s been a while since I’ve looked at DeepSpeech, and 0.9 with your models seems quite fast and accurate. I may be able to train some models myself; I’ve collected quite a bit of speech data, and have been training Kaldi STT models.

Let’s keep the discussion going here and see what we can accomplish.

@koan is the keeper of our Hermes messaging library. @maxbachmann wrote the very fast rapidfuzz NLU library and is our resident C++ badass. @RaspiManu has recently been helping verify German sentences for Larynx TTS so we can have volunteers contribute their voices.

Lots of great talent here :smiley:

9

You can find all the instructions to create new skills in the demo skill. I also think this is a good start for you to checkout things we can combine.

There is some explanation in the riddles demo skill: https://gitlab.com/Jaco-Assistant/Skill-Dialogs/-/blob/master/dialog/nlu/en/nlu.md. I’m using the style Rasa suggested, but with some restrictions. I do like the style because you can see the utterances in a nice layout when looking at the files in the skills repository. Currently I’m not supporting named slots for slots of the same type like “start” and “destination” locations, but this should be easy to integrate.
With numbers and dates, I am using Duckling, which does all the conversion stuff and is integrated into Rasa.

It’s quite similar, mainly using different topic names. But most importantly, all topics are encrypted by default, because I did want to ensure, that skills can only access their topics or external topics they mention in their config file. So I think a translation service (we could create an extra skill for this) would be the best way. For encryption and decryption I’m using a file with keys from which the required keys are written to another file for each skill. So it’s easy to access them as user or a separate program (which is intended), but not from inside a skills container (a weather skill shouldn’t be able to access the microphone streaming topic).

Really would love to be able to use Rhasspys GUI to setup even easier. Currently you have one global setup file, where you have to fill out details like microphone index by hand.

For running Jaco, I’m automatically creating a podman-compose file which can start all the modules and skills.

One important requirement for me would be that we can run the model on a RasPi 4, maybe 2x faster than real time.
I already integrated a MozillaTTS voice (in German), but it’s quite slow, even on good computers (not using the gpu).

3 Likes
10

Our initial idea for was to use Mosquitto Access Control List for this. This raises the question how restrictive we want access to be.

Using Mosquitto ACL the most we could do is restriction at the topic level. However we might want to restrict read/write aswell, since e.g. each skill will require to a topic like hermes/dialogueManager/endSession, since it is used to end an dialog. Should each skill be able to read the content of this message when it is sent by a different intent? If not, this would require us to encrypt messages. Even when using Mosquitto ACL to prevent subscribtions to certain topics and encrypting messages on topics like hermes/dialogueManager/endSession it would still be possible for skills to check when these messages are sent, which could be used to monitor the presence of a person.

So I would be really interested in the way Jaco handles this.

Some of the requirements I personally have for the solution:

  • users should be able to have a simple way to bypass all of this when they do not need this security. E.g. a common use case would be to install some skills from the store, but at the same time have some own skills outside of these containers, e.g. because you need to receive messages on a microcontroller (encryption might not be well supported and the device is trusted)
  • We should try to keep the performance impact as small as possible
  • It should be very simple to implement (or at least the complexity should be hidden away from the user e.g. in the Python skill library)
11

You’re in luck! My German Larynx container runs around 2X realtime on my Pi 4 (4 GB). Ignore the timing of the first sentence you try, as it’s loading the lexicon into memory. It’ll also cache sentences, so repeats will skip synthesis.

I’ve thought about ways to make the GUI more modular. It would be nice to have a description of options in a service or skill, and have HTML controls created for them. I’ve been wary to go down that rabbit hole, though, because there are going to be lots of corner cases and I’m not really a fan of web programming anyways :laughing:

This is an interesting approach. I like it overall, but for Rhasspy I think we’d need to lock topics down at the broker level by client. Lots of people use Rhasspy with NodeRED, and getting encryption keys into that would be tough for newcomers.

Your system seems secure and well thought out, though :+1:

12

Currently Jaco encrypts only the messages, so skills can’t read the content, but they would be able to subscribe to topics and see that something is sent. Not sure which metadata is included in the messages, but that would be readable too. Each skill has its own topic to send text which shall be spoken to the system. The sessions are also handled differently, in a way, that I skipped them completely. I didn’t like the approach of Snips here, because I have some skills that require long running sessions, because they did some web or hardware request which could take half a minute. So currently I’m just blocking wake-word detection while the user or the system are speaking.

The encryption itself is done with python’s cryptography library, and you just need to share the key string to decrypt it. So I think it’s quite easy to implement, if you don’t want to use Jaco’s tools which do everything automatically. I also don’t think this results in a big performance drop.

But I know this is not perfect and Mosquitto ACL could be a good, or maybe better solution for this, I’m open to a more elegant suggestion here.
My plan for some time in the future would be to drop MQTT completely and use some messaging system which runs in peer-to-peer mode, removing the central mqtt server. If you have some external system like HomeAssistant my idea was that you use a skill which serves as communication interface. What do you think about this?

For me this was more a problem for the future, because my current priority is to further improve recognition performance, as well as supporting more languages.

Did you test if it’s also able to run on a Pi3? And how long did training the voice roughly take?
Your larynx container has a nice web interface, really good for a short test :+1:


Regarding the intent definition syntax, what do you think of something like this:

## lookup:city
city.txt  <- text file with one city per line 

## intent:travel_duration
- How long does it take to drive from [Augsburg](skill_travel_city:city_name_1) to [Berlin](skill_travel_city:city_name_2) by (train|car)?
`Comments like this`
`Rasa, and I think Rhasspy too, use something with {} brackets for named slots, but I think this breaks the style somewhat, because you don't get the blue link coloring then`

## synonyms
syn.json  <- Currently I'm using a list like with the intents above, but I think this would be more readable.

I would say we are quite free with creating a syntax and should make another conversion script for the different tools. I like your (option1|option2) syntax, but its not supported by Rasa, and I would have to replace it with two sentences anyway.
We can focus on readability this way.

Currently Jaco does collect all dialog files from the skills and merges them into one single place, with skill-name prefixes you can use globally.
Those intent files are used later on to create a text file with possible sentences users can say, which is used to train the language model. I think Rhasspy uses a similar approach here, which should make replacing the STT service with a different on quite easy.

13

I did just recheck how mosquitto ACL works and it appears to do exactly what we need. We could use MQTT over TLS and username/password, where each skill gets his own pair of username and password. Username is the intent name used for the hermes topic in hermes/intent/<intentName>. I remember we talked about the behaviour, when multiple skills have the same name, but I do not remember how we decided to handle this (@synesthesiam, @koan, do you remember where this was discussed).

Using ACL it is possible to set read, write and readwrite permissions, so the basic rules for a skill could be something along the lines of

user <skill_name>
topic read hermes/intent/<skill_name>
topic write hermes/dialogueManager/startSession
topic write hermes/dialogueManager/endSession
topic write hermes/dialogueManager/continueSession
...

What I like about this solution is:

  • TLS for MQTT is required anyways, so nobody else on the network can listen to the traffic (already tracked in issue 29)
  • we can derive usernames from the skill name and easily automate the generation of the ACL rules.
  • for self made skills the ACL rule can simply give readwrite permission to all topics
  • skills do not even know a message was sent, when they only have write access to a topic
  • All encryption components used are standard in MQTT -> most MQTT libraries support it
1 Like
14

This should be quite easy to implement into Jaco. Instead of automatically distributing the encryption keys (which look like this: "Jaco/Intents/JacoMusicmasterNextSong": "64uLChtXiB8n00yZoBzzbPcfYSK_BKt17spuTn2jbtE=" we just have to share passwords and create a rules file for the broker. Skills use an interface for communication, so there should be no need for changes in the skill code.

Currently they would share the same keys if their intents have the same names too (I don’t check for upper or lower case, drop all special signs and convert them to SkillNameIntentName format). Maybe we should do a short check in the skill installation process and print a big warning.

15

Ah now I remember what we discussed. It was not about having multiple skills with a similar name, but multiple intents with a similar name. We decided to go for the syntax

hermes/intent/<skill_name>/<intent_name>

so it is possible to subscribe e.g. to all topics of a skill aswell using hermes/intent/<skill_name>/#

I guess in the skill store this could not occur anyways, since skills would be forced to choose a unique name. So it should be enough to warn the user about this (this would probably mean he created a local skill himself with a similar name)

16

Currently it’s possible, there is not check for name uniqueness right now. The skills can also have different names in different languages. What Jaco uses later as name in the scripts is the folder name of the skills. You could have two skills with the same names here too, because you can download the skills from any git repository and you theoretically could have one TheSkill on github and another TheSkill on gitlab.

Would you be interested in adding this yourself into Jaco, as a merge request? With some help of course.

17

In the next weeks I still have to get my string matching library RapidFuzz to v1.0.0 (getting closer :tada:) and add SSML support to Rhasspy. Possibly afterwards, but I would not count on it.

18

@synesthesiam what do you think about it?

19

(Sorry for the delay in a response. I’m spending a bit less time on the computer during the holidays :slight_smile:)

I think this would be a good start; there is a lot of overlap with Rhasspy’s existing format. I would probably keep this “cross-assistant” format separate and just convert it to Rhasspy’s internal format during training.

Can you provide some examples of synonyms? Are these any different than Rasa’s?

20

Currently the synonyms follow rasa’s format:

## lookup:city
...

## intent:travel_duration
...

##synonym:red
- light red
- dark red

##synonym:blue
- light blue
- dark blue

In my opinion moving the definitions into an extra file (like with the lookup.txt) would improve readability if there are many synonyms.
In this file I would create a simple json structure:

{
  "red": ["light red", "dark red"],
  "blue": ["light blue", "dark blue"]
}

This is a good approach I think. I would handle this the same way for rasa. This should make switching nlu or stt services quite easy.

1 Like
21

We could also use another syntax which is quite close to rhasspy’s and define the synonyms directly in the lookup.txt files:

(red|light red|dark red):red
green
yellow
(blue|light blue|dark blue):blue

And I would update the intent’s slot definition a little bit:

## intent:travel_duration
- How long does it take to drive from [Augsburg](city.txt?city_name_1) to [Berlin](city.txt?city_name_2) by (train|car)?

This should make the links which are displayed in the git repository usable.

1 Like
22

I was quite busy the last weeks, but finally found some time to update Jaco’s dialog syntax.

The syntax is almost similar to the one in my above post. Intents are defined in a nlu.md file like this:

## lookup:city
city.txt

## intent:book_flight
- Book (me|us) a flight from [Augsburg](city.txt?start) to [New York](city.txt?stop)

Entities are defined in an extra file (here city.txt):

Augsburg
(New York|N Y)->New York
Berlin

A longer example can currently (until I merged it) be found here: https://gitlab.com/Jaco-Assistant/Skill-Riddles/-/blob/some_upds/dialog/nlu/de/nlu.md

To train the NLU and STT services, Jaco collects all the dialog files from the installed skills and injects the different entity examples into the utterances. The output consists of two files like this:

# sentences.txt (for STT)
book me a flight from augsburg to new york
book me a flight from augsburg to berlin
book me a flight from berlin to new york
book us a flight from berlin to new york
...

# nlu.json (for NLU)
{
  "lookups": {
    "skill_flights-city": [
      "augsburg", 
      "(new york|n y)->new york",
      ...
    ]
  },
  "intents": {
    "skill_flights-book_flight": [
      "book me a flight from [augsburg](skill_flights-city?start) to [new York](skill_flights-city?stop)",
      "book me a flight from [augsburg](skill_flights-city?start) to [berlin](skill_flights-city?stop)",
      "book me a flight from [berlin](skill_flights-city?start) to [new york](skill_flights-city?stop)",
      "book us a flight from [berlin](skill_flights-city?start) to [new york](skill_flights-city?stop)",
      ...
    ]
  }
}

The sentences.txt file can directly be used to train the language models for DeepSpeech. The nlu.json is afterwards converted to rasa’s train data style.

A sample nlu output for the query book me a flight from augsburg to new york would then look like this:

{
  'intent': {
    'name': 'skill_flights-book_flight',
    'confidence': 1.0
  },
  'text': 'book me a flight from augsburg to new york',
  'entities': [
    {
      'entity': 'skill_flights-city', 
      'value': 'augsburg', 
      'role': 'start'
    },
    {
      'entity': 'skill_flights-city', 
      'value': 'new york', 
      'role': 'stop'
    }
  ]
}

What do you think?

And @maxbachmann, do you think you can build a conversion script for your RapidFuzz library?
I would like to benchmark it against rasa then, to check if there is a trade-off between the training duration and recognition performance, and how big it is in that case.

1 Like
23

I’ll take a look tomorrow, @DANBER. Thanks for the update!

I’ve been busy as well trying to improve the performance of Larynx (TTS) and train more Kaldi models. I’d like to talk to you at some point about expanding the language support in DeepSpeech-Polyglot. I’m curating datasets, and have GPUs available for training.

24

I would recommend we wait a few weeks with that, I’ve been working on improving the performance here, but it’s not yet finished.
Generally it’s very easy to add support for new languages, you just have to add a new alphabet and extend the preprocessing rules here: https://gitlab.com/Jaco-Assistant/deepspeech-polyglot/-/blob/master/data/langdicts.json

With the upcoming changes I will update the datasets .csv format a little bit, but there is already an importer for the old DeepSpeech format. The content in langdicts.json stays the same.

If your datasets are public, it would be great if you could add them to the corcua library, where I already did collect a lot of German and multilingual datasets:

25

Seems that your days are quite long, don’t forget to sleep for some hours in between :wink:

1 Like
26

My days are quite long :wink:
Really, though, I just have too many things I’m trying to do at once.

I’ve written a rudimentary converter from your proposed format to Rhasspy’s ini format. It should be pretty easy to turn it into a command-line tool that can convert a set of Jaco skills.

Are the ## lookup: sections needed anymore if the entity links use the file names (city.txt)?

27

Sounds familiar to me :smile:

Not necessarily, but I did keep them for a better overview in the web readme, you don’t see the entity names there, if you don’t hover over the links.

28

Now it’s finished:)
The project got a new name: Scribosermo and now can be found here:

The new models can be trained very fast (~3 days on 2x1080Ti to reach SOTA in German) and with comparatively small datasets (~280h for competitive results in Spanish). Using a little bit more time and data, the following Word-Error-Rates on CommonVoice testset were achieved:

German English Spanish French
7.2 % 3.7 % 10.0 % 11.7 %

Training is even simpler than with DeepSpeech before and adding new languages is easy as well. After training, the models can be exported into tflite-format for easier inference. They are able to run faster than real-time on a RaspberryPi-4.

Only downside is that the models can’t be directly integrated into DeepSpeech bindings (technical possible, but I had no need for it) and doesn’t support streaming anymore (at least until someone has the time to implement it). I don’t think the missing streaming feature should be a problem, because our inputs are quite short, they usually are processed in 1-2 seconds on a Raspi.

1 Like
29

I already did update Jaco and run the benchmarks again, which show that the new models perform really well:

compare_barrista_432
compare_barrista_4321920×1440 203 KB


compare_smart_lights
compare_smart_lights1350×1500 77.2 KB

30

@maxbachmann, if you have some free time left, I have an idea for a project, where your experience with RapidFuzz might be helpful.

Currently the different STT modules of Jaco and Rhasspy are using n-gram language models (in form of arpa or scorer files), which are based on plain sentences, to improve the predictions.
With the recent update of Jaco’s STT network to Scribosermo, the base-line performance of the model is much better than the Kaldi model which was used in Snips. But in above SmartLights benchmark the recognition performance of Jaco is only slightly better than the one of Snips.

I think we should be able to improve the performance further if we replace the n-gram language model with a more task specific language model. In the Snips paper (https://arxiv.org/pdf/1810.12735.pdf) a combination of a n-gram model with a pattern-based model is described, which I think would be a good idea to start with.
In general this would replace the training sentences “turn on the light in the kitchen” and “turn on the light in the living room” with “turn on the light in the [ROOM]” . The resulting sentence can be used to build a n-gram model (which should be much smaller than before) and for the ROOM slot an extra matcher would be required.

The input into such a model would be in our case the direct output of the STT models. In the case of CTC-based STT models (Scribosermo, DeepSpeech, and I think Kaldi too) this would be, before rescoring with a LM, letter based probabilities for a specific time step. The word “hello” might look like this: “hhellll-lllllllooooo” (using the letter with highest probability for each time step). To get the original text, letters with multiple occurrences have to be merged to a single letter, and then the blank-symbol (-) is removed.


What do you think?
31

I have absolutely no idea about machine learning, so I am probably unable to help you on that. For the character deduplication I would personally try to go with a relatively naive implementation first and see whether it is fast enough. It could probably be improved performance wise using a concept similar to https://lemire.me/blog/2017/04/10/removing-duplicates-from-lists-quickly/ if it is really required.

32

This wouldn’t require any knowledge of machine learning, but experience in fast C++ implementations for text based problems:)
Merging “hhellll-lllllllooooo” to “hello” and combining it with a n-gram language model is already solved, for example by DeepSpeech’s ds-ctcdecoder. What I didn’t find is a library that can work with slot patterns like ROOM.

33

There are some papers/libraries which combine regex with edit distance measurements like the Levenshtein distance. See e.g. https://github.com/laurikari/tre

34

Very impressive! I will see if I can train a new model using Scribosermo :slight_smile:

A few questions:

  • Do you think accuracy could be improved using phonemes instead of orthographic characters?
  • Would streaming require significant changes to the Quartznet architecture?

For Kaldi, Rhasspy also supports directly generating an FST with all of the possible sentences. This is the default, and is actually faster and more accurate than an n-gram language model. The downside, of course, is that it can never recognize sentences outside of the training set.

This sounds very close to Kaldi’s grammars, which you’re probably already familiar with. Instead of rolling our own, I wonder if we could take the output phoneme probability distributions from your DeepSpeech models and run them through a modified Kaldi FST. This would essentially swap out the acoustic model layer of Kaldi with DeepSpeech, but leave the rest intact.

Another idea I’ve had is to use a GPT-2 model in place of a traditional n-gram language model. GPT-2 is far better at tracking long-range dependencies, and may be fast enough to use as a DeepSpeech scorer. Some big challenges are:

  • How can the model be tuned quickly for a specific domain?
  • Can new models be reasonably trained for other languages?
  • Are separate slots possible (e.g., “turn on the light in the [ROOM]”)?
35

No, that shouldn’t be very complicated, Nvidia already did implement an example script here, but I found that recognition is fast enough that you wouldn’t notice a great speedup with streaming.

It might be, but most of the recent STT papers use graphemes (letters) directly, so I’m not sure about that. Some use a subword approach instead of single characters, which seems to bring a small improvement.
If you want to experiment with that, you would need to retrain the English network, I did use the one published by Nvidia, which made training much easier…
And I think that we will get more performance improvement per invested development time if we optimize the language model rescoring and the nlu extraction.

Oh, didn’t know about that, never trained a model with Kaldi, will take a look into it …

That sounds interesting, but as you already mentioned in your challenges, this has to be fast enough to be trained on a Raspi, that we can build domain/skill specific models directly on the device.

1 Like
36

This seems to be what Snips used. @fastjack has pointed me to the kaldi-active-grammars project, which uses them to do dynamic decoding.

If we went with this approach, we could pre-generate grammars for numbers, dates, etc. like Snips did and stitch them into the final graph at runtime.

2 Likes
37

Thanks for the links:)
Kaldi’s grammar syntax seems to be quite similar to what is described in the Snips paper. They also did use Kaldi models, so this looks like a project we should try out

1 Like
38

I found a project which already did implement something similar, a WFST decoder for CTC models, which is called Eesen, but I’m still trying to understand it …


I also have another suggestion, in the threads of this forum I’m often seeing users that are trying to build custom skills for Rhasspy, but often with different approaches. What do you think about making Jaco’s skills directly compatible with Rhasspy and vice versa? This would give all skills a similar structure and also would allow to share Rhasspy skills with other users through the skill store.

Since you already have built a conversion script for the dialog syntax I think this should be very easy. I know that not all features will be supported, but I could add an extra flag with which users can disable the automatic topic encryption and we could restrict the additional requirements of those skills to python libraries only (Jaco would build a container for each skill, that they can have arbitrary requirements, but I think running containers in Rhasspys docker container doesn’t work).
We only need to find an approach for intent to topic mapping. Jaco currently uses Jaco/Skills/SkillName/IntentName and Jaco/Skills/SayText for answers.

39

I suppose this could be abstracted away behind a common API. @koan already created a helper library for Rhasspy skills which can be found here. The library already abstracts away all of the MQTT topics, since they are not relevant for the skill author anyways.

40

Jaco does something similar with skill owned topics (you can listen to them with the intent name only: assist.add_topic_callback("get_riddle", callback_get_riddle)) and specific other topics like text outputs (assist.publish_answer(result_sentence, message["satellite"]), but skills can listen to other topics (system or other skills) or write to them too, and I’m not sure if it makes sense to abstract those arbitrary topics

Example can be found here: https://gitlab.com/Jaco-Assistant/Skill-Riddles/-/blob/master/action-riddle.py

41

Thanks for the pointer, Max.

And here’s the forum discussion where you see the evolution of its development:

If you look at the example code in the repository and the Usage page in the documentation, you see that this abstracts away all MQTT topics indeed. The main goal of the library is to make it as easy as possible to create Rhasspy apps in Python, without having to know low-level details and with as less boilerplate code as possible.

42

This is very similar to what Jaco is doing. The time example for Jaco would look like this:

"""Example app to react to an intent to tell you the time."""
from jacolib import assistant
from datetime import datetime

file_path = os.path.dirname(os.path.realpath(__file__)) + "/"
assist: assistant.Assistant

def callback_get_time(message):
    now = datetime.now().strftime("%H %M")
    assist.publish_answer(f"It's {now}", message["satellite"])

assist = assistant.Assistant(repo_path=file_path)
assist.add_topic_callback("get_time", callback_get_time)
assist.run()
43

Hi @synesthesiam, I made some progress using customized FSTs with the Scribosermo models. Building custom FSTs also allowed an integration of the nlu information directly into the decoding graph, which means that is’s possible to combine the STT+NLU steps into a single SLU model. I already tested it on some benchmarks, with very good results (competitve to Jaco’s current solution with Rasa, as well as some other SLU approaches). And, what’s very interesting for Jaco and Rhasspy, the models can be built in a few seconds, even on a RaspberryPi.

A hopefully understandable description of the approach can be found in the readme of the project:

In the next time I will update Jaco to use the new SLU model (needs some additional features for number+date parsing in multiple languages, presumably with the help of Duckling)

2 Likes
44

This is some incredible work! I’m still digesting it all.

How will you incorporate numbers and dates into the FST? Will you pre-generate all of the possibilities during training, or use grammar FSTs?

By the way, you may be interested in the “unknown words” feature I’ve added to Rhasspy 2.5.11 (Docker preview is available). For each word in the grammar, I have a low-probability branch that goes through a collection of ~100 frequently-used words from the target language (and not used elsewhere in the grammar). This branch outputs <unk>, and acts as a way of “catching” misspoken words since (presumably) frequently-used words will be phonetically diverse. Pretty easy to code up, and seems to be effective!

1 Like
45

I would say it’s a mix with both. The process for numbers looks like this:

  1. all number possibilities are pre-generated in text form
  2. the numbers are converted to a (grammar) Slot-FST
  3. the FST is optimized → this makes it much smaller because different paths are merged (twenty-two and twenty-one start both with twenty and the path after optimization splits at one and two instead of twenty)
  4. the Number-FST is inserted into the Intent-FST which contains the intent examples (what is [number1] plus [number2])

I did test it with the Timers-And-Such benchmark, where finstreder could outperform the paper’s baseline. Optimizing about 1M numbers (from -1000.00 to 1000.00 in different wordings) takes about 10 seconds. Benchmark code is here.
For the benchmark I did write a custom word2num function for the opposite way, but I think switching to Duckling makes things much easier, especially when supporting multiple languages.

Very interesting. What do you think is a use-case where the transcription benefits from it? I think it could be integrated into finstreder too, but I’m not sure where it could help.

1 Like
46

With a restricted vocabulary/grammar in Kaldi, it seems to go to great lengths to match even garbage audio to a known sentence. I don’t know if this is the case with Jaco’s ASR; maybe you already have a good way to reject bad sentences :slight_smile:

47

Finally integrated finstreder into Jaco. Number+Date parsing is done with Duckling (the RasaNLU module already did this too, but I had to rebuild the container and this was more complicated than planned).
Another change is that podman got replaced with docker, because podman doesn’t support Raspi’s arm32 architecture anymore. One benefit of the replacement is that I could integrate portainer, which creates a simple local website to interact with the module and skill containers.


In general Jaco has the same problem, but I found that often just the yes/no intents are matched which normally trigger no reaction of the assistant. I will try to observe this further in the future. Maybe a simple option for Jaco would be to add a garbage intent with such words which doesn’t trigger any actions.

48

You might be interested in the new Conformer models I’ve recently added to Scribosermo and Jaco. In general I’ve converted the pretrained models from Nvidia’s NeMo to tensorflow and tflite to make them work on a RaspberryPi. I hope I can finetune them further in the future. In comparison with the old QuartzNet models the WordErrorRate on greedy free speech recognition was reduced by half, with a language-model about 20-40%. Maybe the models can help you with some ASR tasks as well. But unlike before, they now output sentencepiece-based tokens instead of alphabet-based single characters.

In combination with an updated finstreder SLU decoder, Jaco now achieves state-of-the-art recognition accuracy in most SLU benchmarks I’ve found.
As always all models can be found in Jaco’s repository: Jaco-Assistant / Jaco-Master · GitLab

1 Like
49

Awesome work as always, @DANBER! I’d like to include some of these models and finstreder in the next version of Rhasspy. Have you encountered any problems getting sentencepiece running the pi?

1 Like
50

Have you encountered any problems getting sentencepiece running the pi?

No, running is no problem, the tflite STT models run about 3x faster than realtime on a 64bit Raspi-4. Training the finstreder SLU model also takes only a few seconds. The only problem might be installation of the dependencies, it’s quite resource intensive. I think the easiest way would be using my prebuilt docker images and run them as an external service.

Can you start other containers with Rhasspy? Or else do you think it would be complicated to add?
In this case I could help you with the interface.

1 Like
51

I can, but I’ve been considering other options for Rhasspy going forward. Rather than requiring containers, a more general approach may be something like nix or guix.

I’d be interested in building a few CLI programs for your STT models and finstreder, and packaging them with nix/guix. One program for training, and one program of inference. What do you think?

52

I’d be interested in building a few CLI programs for your STT models and finstreder, and packaging them with nix/guix. One program for training, and one program of inference. What do you think?

Never worked with nix before. As far as I’ve read in my short research now, the main feature would be reproducible builds, which work quite well with docker too. And since the Jaco images are prebuilt, every user gets the same build. Where do you see the benefits of nix vs the docker images?

53

With the installation of plugins/extensions. If you go the container route, you need an orchestration mechanism like pods or docker-compose to bring up all the right containers. Additionally, you need a network-based IPC method like mqtt or HTTP for communication.

My (currently untested) idea is have a single Rhasspy container with nix installed and the nix store mounted externally. Plugins/extensions would be installed as nix packages, and accessed as regular programs by Rhasspy (Python subprocesses, etc).

The dream would be for these packages to be highly reusable outside out Rhasspy, so we can get out of the N by M problem where you need N plugins (wakeword, STT, etc.) for M voice assistants (Jaco, Rhasspy, Mycroft, SEPIA, etc.).

54

I don’t think containerization is the main reason for the incompatible modules problem. I would say it’s more the different interfaces the modules and assistants use. For example Rhasspy uses MQTT following Snips’ hermes protokoll, and Jaco uses MQTT as well but with different topic names and encrypted topic contents.

Regarding the SLU module, if I understand this correctly, you would basically package the python packages of tensorflow + finstreder and some voice assistant specific scripts together into a single nix package. Then it could be called with a specific interface, which results in the same behavior as calling it as containerized service…
And even if there are already packaged modules, like WW detection with porcupine we still package them again to match our custom interfaces.

After reading some more about nix, I think the idea itself is quite elegant, but it also seems to be more complicated than using modularized docker images. I’m not sure it this should be the first problem we need to solve. So I think in a first step it would be easier and faster to add an interface translator between our MQTT topics.

One problem I don’t understand yet, is how you solve the communication between Satellites and Master if they run on different devices? You would still need a network communication like MQTT here?

Another problem I’m not sure about how nix solves this is sandboxing, in case of Jaco it’s mainly required for the skills. I would like that skill devs can implement almost anything they want with any dependencies they need in a preferably simple manner, and currently I solve it by letting each skill build a custom container if they need it. All containers (modules+skills) are then started via an auto-generated docker-compose file. How would this look like in the nix architecture?

55

nix/guix by itself doesn’t directly solve the problems you mentioned, but I think it would be a more flexible and composable starting point. For example, packaging up a nice command-line interface for porcupine with nix/guix would allow for it to be installed on most any Linux distribution, inside a Docker/Podman container, etc. And it would still be usable standalone.

My idea here is trying to reduce the number of “commitments” made upfront by each packaged module, which include things like a particular container technology, MQTT/HTTP/Websockets, and the specific protocol (Hermes, etc.). At a minimum, if a packaged module is a program with a “simple” command-line interface using stdin/stdout for communication (more on this if you’re interested), then it can used standalone or wrapped for something more specialized (Jaco, Rhasspy, etc.).

I think models and other artifacts would also work well as nix/guix packages, since everything is automatically hashed, and you can download files in the build process. I have a weak form of this in Rhasspy now, where STT models, etc. are downloaded in chunks from Github and their sizes (but not hashes) are checked.

56

Some dev news again:
In the last days I had some time to continue working on a better cross-assistant integration and found a way to make Jaco’s skills usable with Rhasspy:

The basic concept was to create a mapping to Rhasspy’s sentences.ini file and between the different mqtt topics. To use a skill from Jaco’s skill store you download and install it with the tools from Jaco-Master. Then run the preprocessing script which builds the sentences.ini and slots files and retrain Rhasspy. At last you start all skill containers with the docker-compose file Jaco generated (running Jaco-Master as well is optional). You can find this interface skill here:


As I already wrote at the beginning of this thread, the skill concept is a bit similar to the old Snips’ skills, but with several improvements regrading skill functionality, with less developer restrictions, and an improved security concept with topic encryption, a permission system and container isolation.

It would be great if you could test this out yourself, and if you like it, maybe add it as official skill concept. In my opinion skill interoperability between the open source assistants (adding support for Mycroft, Sepia and Alice through another interface skill shouldn’t be complicated) is a quite important feature for everyone.


Did you continue working on the nix/guix packaging of the assistant modules? How is your progress?

1 Like
58

@synesthesiam, what do you think?

59

This looks really cool! I’ll try it out and let you know how it works :+1:

I agree that skill interoperability is something we should strive for. It seems like for most cases you’d still need to run both systems, though. With Mycroft, for example, skills embed intents and dialogue responses as well as code for interacting with the microphone and TTS.

I’m still working on that, but for now I’m focusing on how Rhasspy (v3) talks to programs. I have a small protocol that works over stdin/stdout, and a set of “adapter” scripts that let you call existing programs. For example, you can use any STT program that accepts a WAV file and outputs the transcription as text.

I’ll be writing a variety of adapter scripts, and I want to make it easy for others to contribute them.