Add intent tests to avoid breaking assistant

To test my assistant and avoid repeating the same utterances all day long, I’m using an in-house Node.js script based on Snips exported unit-tests.

It would be nice to also have a “tests” feature (like Snips had).

Tests can be JSON files located in a tests folder of the profile with the WAV file bytes encoded as base64 and the expected utterance, intent and slots.

{
    "bytes": "",
    "expected": {
        "text": "...",
        "intent": "...",
        "slots": [
            {
                "name": "...",
                "value": "...",
                "raw_value": "..."
            }
        ]
    }
}

A new tab in the web UI can be added to manage tests. This tab should have the ability to record an utterance, play it back and match it with an intent and slots.

Using the API, it should be easy to execute tests in batch.

This would clearly help working on intents and slots and ensure nothing is broken.

What do you think?

3 Likes

This feature actually exists in voice2json. I think the major work in porting it would be the web UI.

I’m all for adding it. As always, I would welcome a web developer to take on the Rhasspy web UI :slight_smile:

I’ve posted a python script to do this. It test sentences from a json and tell you if the right intent was matched.

Was thinking about adding this into the Jeedom plugin, but if it’s in interface, even better.

-> rhasspyBatcher : fine tune your intents / slots

Anyway, not sure it is high on the list. Builtins slots (duration, datetime, number etc) are really needed and for me the top priority actually. Could not use any assistant in production without this, it is a no go.
Base/satellite setup comes second, even if I could run several pi4 with full rhasspy and duplicates profiles after training on one of them (not a solution but would work).
LED control/feedback also comes to mind, etc.

I think making a snips replacement with these features would make a lot of users switch to rhasspy. And more users means more feddback, more ideas, more helpers, all the best for rhasspy :wink: Cheaters ever know how to run such batch tests :kissing_smiling_eyes:

2 Likes

The web UI is pretty awesome! You did a great job already. :clap:

The tests in voice2json do not seem to take the extracted slot values into consideration. Is this correct?

2 Likes

No, they do. There are a number of accuracy metrics computed, including an intent_entity_accuracy that is 1.0 when every intent matches and every slot value matches, with no extra slot values.

1 Like

This will definitely be my focus for the next few weeks.

2 Likes

Awesome! I highly await the builtin slots :blush:

I’ll take a look at the tests in voice2json when I have more time.

I have some expertise in web app dev so I might try to port the web UI as a separated service when the communication layer and protocol is defined more clearly and if my free time allows it.

In the meantime, I’ll try to write some ideas in a dedicated topic so we can all discuss it.

1 Like