Support grammar based slots

Continuing discussion of https://github.com/synesthesiam/rhasspy/issues/92

What does everything think of making it so that slot files (current text) could also be programs that generate values?

So, maybe slot files that end in .py will be executed during training and could generate their values. And maybe .sh slot files would be executed as a shell script.

This would let us encode numbers, date ranges, etc. as programs instead of large FSTs. To be efficient, I think it would make sense to allow arguments to be passed in when using the $slot. The syntax may get complex, though. Something like $number,1,100 could generate “one” to “one hundred” in the current profile language (using num2words), for example.

Interesting… For slots like artist names it would allow fetching Spotify charts…

I would prefer to use grammar based slots though. The other type of slots (gazeteer slots) can already be generated via a script anyway.

Grammar based slots would be easier to work on and share with others (no code skill or dependencies required).

If both are possible then it would be even more awesome :blush:

1 Like

Here the one I use from Snips, and that I won’t be able to run an assistant in production without equivalent:

duration
dateTime
year
age

At least knowing if there is solution in reasonable future for Rhasspy would help :upside_down_face:

1 Like

I see grammar based as the default. Using the doit library, msybe we could pre-compute programmatic slot values and generate GrammarFSTs for later use.

This is what Snips was doing. Their grammar slots are provided with each assistant download as FSTs (e_#.snips) with the words.txt file (named w.snips). I’m sure these FSTs were generated once (using a grammar or a corpus, I don’t know) and are then included with every assistant that uses them.

They used a very small acoustic model and the Kaldi GrammarFST system to optimize the language model footprint during decoding (since the slots are not repeated).

The problem with generating all the values is that there can be millions of possible values (7 weekdays for 30 days for 12 months for hundreds of years, plus relative dates, glue words like “and”, etc).

A grammar would be easier to maintain.

Here is what I came up with in french for these slots:

# Number
two_to_nine = ( deux | trois | quatre | cinq | six | sept | huit | neuf )
one_to_nine = ( un | une | <two_to_nine> )
teens = ( dix | onze | douze | treize | quatorze | quinze | seize | dix sept | dix huit | dix neuf )
tens = ( vingt | trente | quarante | cinquante | soixante )
tens2 = ( soixante | quatre vingt )
one_to_hundred = ( <one_to_nine> | <teens> | <tens> [ <one_to_nine> ] | <tens2> [ ( <one_to_nine> | <teens> ) ] )
hundreds = cent
thousands = mille
number = [ [ <two_to_nine> ] <thousands> ] [ [ <two_to_nine> ] <hundreds> ] [ <one_to_hundred> ]

# Date
weekdays = ( lundi | mardi | mercredi | jeudi | vendredi | samedi | dimanche )
months = ( janvier | février | mars | avril | mai | juin | juillet | août | septembre | octobre | novembre | décembre )
monthdays = ( premier | <one_to_nine> | <teens> | vingt [ et ] [ <one_to_nine> ] | trente [ et un ] )
date = ( <weekdays> | le ) <monthdays> <months>
relative_date = ( ( demain | hier ) [ ( matin | midi | soir ) ] | <weekdays> [ ( dernier | prochain | matin | midi | soir ) ] )

# Time
hour = ( <one_to_nine> | <teens> | vingt [ et ] ( une | deux | trois ) ) heure
one_to_sixty = ( <one_to_nine> | <teens> | ( dix | vingt | trente | quarante | cinquante ) [ et ] [ <one_to_nine> ] )
minute = <one_to_sixty> [ minutes ]
time = ( minuit | <hour> ) [ <minute> ]

# Datetime
datetime = [ ( <relative_date> | <date> ) ] à <time>

# Duration
duration_seconds = <number> secondes
duration_minutes = <number> minutes
duration_hours = <number> heures
duration_days = <number> jours
duration_weeks = <number> semaines
duration_months = <number> mois
duration_years = <number> années
duration = ( <duration_years> | <duration_months> | <duration_weeks> | <duration_days> | <duration_hours> | <duration_minutes> | <duration_seconds> )

# Percent
percent = <number> pourcent

# Temperature
temperature = <number> degrés [ celcius ]

It’s basic but it works quite well (apart for the sentence weighting issue…).

I Just wanted to put these in slot files to ease maintenance and improvements like so:

# File: slots/time
# This is rules used by the slot and shared with other slots like all rules
hour = ( <one_to_nine> | <teens> | vingt [ et ] ( une | deux | trois ) ) heure
one_to_sixty = ( <one_to_nine> | <teens> | ( dix | vingt | trente | quarante | cinquante ) [ et ] [ <one_to_nine> ] )
minute = <one_to_sixty> [ minutes ]
time = ( minuit | <hour> ) [ <minute> ]
# This is a value used for the slot
<time>

Hope this helps.

1 Like

So you have duration working in rhasspy ?? Saying ‘let the light of kitchen on for 24 minutes’ you get the slot duration at 24 ?? Could you elaborate on this ?
Really miss this one for deeper testing once multiple intents file bug is fixed.

I’m thinking GrammarFST slots should be a main feature for version 2.5. Let’s decide on a core set of slots that we can create for most of the supported languages (maybe the ones @KiboOst identified).

For Pocketsphinx, I’m hoping I can use class-based language models to lower the training time. I still think restricting ranges will be necessary for performance on the Pi with Pocketsphinx.

Version 2.5 FTW :blush: This will be a game changer feature for Rhasspy.

I propose these slots to start as I think they are the really essential ones:

  • number
  • datetime (absolute and relative)
  • duration

If these work well, we can add additional slots like:

  • ordinal
  • currency
  • temperature
  • amountOfMoney
  • distance
  • volume

All of these can be parsed in multiple languages by duckling so the NLU can even provide them with their real actionable value.

2 Likes

4 posts were split to a new topic: Slot syntax with rules

Having done some tests with duration. Works great with rules into the intent but duration slot is for example “vingt minutes”.

Problem is we can’t do anything with that. Duration slot is used to calculate stuff with the slot value, programming something later for example.

If I say “Let the kitchen light on for twenty minutes”
slot duration = “twenty minutes”
sure I can remove " minutes" (in one language, this is another problem)

But then I endup with duration = “twenty” but we need duration = 20
Can’t see how we could translate number representation string to int.

duration slot should always return an integer in same unit. This is how snips does it, and it works perfect as we can actually use the number in all sort of calculation.

That’s the same for number slot.
How could I use “un” + “deux” ? number must be … a number.

dateTime can be a string, but in a standard format so we can convert it to date, in python or php or whatever. But always exact same format.

So actually, it works, recognition at least, but we can’t do anything with it :confused:
Dunno how snips handle it. I guess we would need a parser that does the opposite of the rules (convert word to number) depending on the language used.
Here is an example in english for php: https://stackoverflow.com/questions/4189195/convert-string-ten-to-integer-10

We would also support “Let it for two months three days and two hours” :face_with_raised_eyebrow:

I haven’t used it yet, but Mycroft’s Lingua Franca should be able to handle this.

Numbers:

from lingua_franca.parse import extract_number

assert extract_number("two million five hundred thousand tons of spinning "
                      "metal") == 2500000
assert extract_number("1 and 3/4 cups") == 1.75

Durations:

from lingua_franca.parse import extract_duration
from datetime import timedelta

assert extract_duration("nothing") == (None, 'nothing')

assert extract_duration("Nineteen minutes past the hour") == (
    timedelta(minutes=19),
    "past the hour")
assert extract_duration("wake me up in three weeks, four hundred ninety seven"
                        " days, and three hundred 91.6 seconds") == (
           timedelta(weeks=3, days=497, seconds=391.6),
           "wake me up in , , and")

Dates:

from datetime import datetime
from lingua_franca.parse import extract_datetime, normalize

def extractWithFormat(text):
    date = datetime(2017, 6, 27, 13, 4)  # Tue June 27, 2017 @ 1:04pm
    [extractedDate, leftover] = extract_datetime(text, date)
    extractedDate = extractedDate.strftime("%Y-%m-%d %H:%M:%S")
    return [extractedDate, leftover]

def testExtract(text, expected_date, expected_leftover):
    res = extractWithFormat(normalize(text))
    assert res[0] == expected_date
    assert res[1] == expected_leftover

testExtract("What is the day after tomorrow's weather?",
            "2017-06-29 00:00:00", "what is weather")
testExtract("Remind me at 10:45 pm",
            "2017-06-27 22:45:00", "remind me")

This is just parsing. It also supports the other direction: generating pronounceable strings based on numbers, durations or dates.

By the way, it’s not visible in these examples but the library is multilingual, as can be seen in the test scripts.

Ah, sounds nice !!!

Snips always return duration slot in total seconds, seems the same here.

I also guess this is why they need to be builtins slots, not user slots. So rhasspy know the slot and can extract/parse it into usable values before returning found slots.
Say “twenty one minutes”
-> rhasspy see it as duration slot
-> extract_duration(“twenty one minutes”) -> 1260 sec
-> return slot {duration: 1260}
:+1:

Can’t wait to play with such builtins slots, extremely powerfull with snips.

Just in case, here is the list of snips builtins slots, in snips-nlu repo that have many interesting stuff (maybe fork it in case sonos delete everything ?)

Regarding duration, I though the result was the sum of minutes, but that’s not the case. I wrote some python app for snips and had to write some function to parse the result to add duration to actual date etc.
Here is a dummy duration class I used for test:

class duration(object):
			def __init__(self):
				self.years = 0
				self.quarters = 0
				self.months = 3
				self.weeks = 2
				self.days = 14
				self.hours = 0
				self.minutes = 0
				self.seconds = 0
				self.precision = "Exact"

The conversion to total minutes happens in the jeedom plugin to program cron easily. So once rhasspy got duration builtin slot, I will handle that in the plugin.

LOL! Great minds think alike.
I’ve just created a new topic on the subject :slight_smile:

Will Rhasspy 2.5 enhance built-in slots or will this get worked later ?

duration : Got similar working use with Number Ranges {0…120}[minutes] for example. Harder for “In one year, two months and three weeks”
dateTime ?
year ? ( Guess could be handled with number ranges like [en|année]{1900…2050} )
age ? (same)