Continuing discussion of https://github.com/synesthesiam/rhasspy/issues/92
What does everything think of making it so that slot files (current text) could also be programs that generate values?
So, maybe slot files that end in .py
will be executed during training and could generate their values. And maybe .sh
slot files would be executed as a shell script.
This would let us encode numbers, date ranges, etc. as programs instead of large FSTs. To be efficient, I think it would make sense to allow arguments to be passed in when using the $slot
. The syntax may get complex, though. Something like $number,1,100
could generate âoneâ to âone hundredâ in the current profile language (using num2words
), for example.
Interesting⊠For slots like artist names it would allow fetching Spotify chartsâŠ
I would prefer to use grammar based slots though. The other type of slots (gazeteer slots) can already be generated via a script anyway.
Grammar based slots would be easier to work on and share with others (no code skill or dependencies required).
If both are possible then it would be even more awesome
Here the one I use from Snips, and that I wonât be able to run an assistant in production without equivalent:
duration
dateTime
year
age
At least knowing if there is solution in reasonable future for Rhasspy would help
I see grammar based as the default. Using the doit
library, msybe we could pre-compute programmatic slot values and generate GrammarFSTs for later use.
This is what Snips was doing. Their grammar slots are provided with each assistant download as FSTs (e_#.snips) with the words.txt file (named w.snips). Iâm sure these FSTs were generated once (using a grammar or a corpus, I donât know) and are then included with every assistant that uses them.
They used a very small acoustic model and the Kaldi GrammarFST system to optimize the language model footprint during decoding (since the slots are not repeated).
The problem with generating all the values is that there can be millions of possible values (7 weekdays for 30 days for 12 months for hundreds of years, plus relative dates, glue words like âandâ, etc).
A grammar would be easier to maintain.
Here is what I came up with in french for these slots:
# Number
two_to_nine = ( deux | trois | quatre | cinq | six | sept | huit | neuf )
one_to_nine = ( un | une | <two_to_nine> )
teens = ( dix | onze | douze | treize | quatorze | quinze | seize | dix sept | dix huit | dix neuf )
tens = ( vingt | trente | quarante | cinquante | soixante )
tens2 = ( soixante | quatre vingt )
one_to_hundred = ( <one_to_nine> | <teens> | <tens> [ <one_to_nine> ] | <tens2> [ ( <one_to_nine> | <teens> ) ] )
hundreds = cent
thousands = mille
number = [ [ <two_to_nine> ] <thousands> ] [ [ <two_to_nine> ] <hundreds> ] [ <one_to_hundred> ]
# Date
weekdays = ( lundi | mardi | mercredi | jeudi | vendredi | samedi | dimanche )
months = ( janvier | février | mars | avril | mai | juin | juillet | août | septembre | octobre | novembre | décembre )
monthdays = ( premier | <one_to_nine> | <teens> | vingt [ et ] [ <one_to_nine> ] | trente [ et un ] )
date = ( <weekdays> | le ) <monthdays> <months>
relative_date = ( ( demain | hier ) [ ( matin | midi | soir ) ] | <weekdays> [ ( dernier | prochain | matin | midi | soir ) ] )
# Time
hour = ( <one_to_nine> | <teens> | vingt [ et ] ( une | deux | trois ) ) heure
one_to_sixty = ( <one_to_nine> | <teens> | ( dix | vingt | trente | quarante | cinquante ) [ et ] [ <one_to_nine> ] )
minute = <one_to_sixty> [ minutes ]
time = ( minuit | <hour> ) [ <minute> ]
# Datetime
datetime = [ ( <relative_date> | <date> ) ] Ă <time>
# Duration
duration_seconds = <number> secondes
duration_minutes = <number> minutes
duration_hours = <number> heures
duration_days = <number> jours
duration_weeks = <number> semaines
duration_months = <number> mois
duration_years = <number> années
duration = ( <duration_years> | <duration_months> | <duration_weeks> | <duration_days> | <duration_hours> | <duration_minutes> | <duration_seconds> )
# Percent
percent = <number> pourcent
# Temperature
temperature = <number> degrés [ celcius ]
Itâs basic but it works quite well (apart for the sentence weighting issueâŠ).
I Just wanted to put these in slot files to ease maintenance and improvements like so:
# File: slots/time
# This is rules used by the slot and shared with other slots like all rules
hour = ( <one_to_nine> | <teens> | vingt [ et ] ( une | deux | trois ) ) heure
one_to_sixty = ( <one_to_nine> | <teens> | ( dix | vingt | trente | quarante | cinquante ) [ et ] [ <one_to_nine> ] )
minute = <one_to_sixty> [ minutes ]
time = ( minuit | <hour> ) [ <minute> ]
# This is a value used for the slot
<time>
Hope this helps.
So you have duration working in rhasspy ?? Saying âlet the light of kitchen on for 24 minutesâ you get the slot duration at 24 ?? Could you elaborate on this ?
Really miss this one for deeper testing once multiple intents file bug is fixed.
Iâm thinking GrammarFST slots should be a main feature for version 2.5. Letâs decide on a core set of slots that we can create for most of the supported languages (maybe the ones @KiboOst identified).
For Pocketsphinx, Iâm hoping I can use class-based language models to lower the training time. I still think restricting ranges will be necessary for performance on the Pi with Pocketsphinx.
Version 2.5 FTW This will be a game changer feature for Rhasspy.
I propose these slots to start as I think they are the really essential ones:
- number
- datetime (absolute and relative)
- duration
If these work well, we can add additional slots like:
- ordinal
- currency
- temperature
- amountOfMoney
- distance
- volume
All of these can be parsed in multiple languages by duckling so the NLU can even provide them with their real actionable value.
Having done some tests with duration. Works great with rules into the intent but duration slot is for example âvingt minutesâ.
Problem is we canât do anything with that. Duration slot is used to calculate stuff with the slot value, programming something later for example.
If I say âLet the kitchen light on for twenty minutesâ
slot duration = âtwenty minutesâ
sure I can remove " minutes" (in one language, this is another problem)
But then I endup with duration = âtwentyâ but we need duration = 20
Canât see how we could translate number representation string to int.
duration slot should always return an integer in same unit. This is how snips does it, and it works perfect as we can actually use the number in all sort of calculation.
Thatâs the same for number slot.
How could I use âunâ + âdeuxâ ? number must be ⊠a number.
dateTime can be a string, but in a standard format so we can convert it to date, in python or php or whatever. But always exact same format.
So actually, it works, recognition at least, but we canât do anything with it
Dunno how snips handle it. I guess we would need a parser that does the opposite of the rules (convert word to number) depending on the language used.
Here is an example in english for php: https://stackoverflow.com/questions/4189195/convert-string-ten-to-integer-10
We would also support âLet it for two months three days and two hoursâ
I havenât used it yet, but Mycroftâs Lingua Franca should be able to handle this.
Numbers:
from lingua_franca.parse import extract_number
assert extract_number("two million five hundred thousand tons of spinning "
"metal") == 2500000
assert extract_number("1 and 3/4 cups") == 1.75
Durations:
from lingua_franca.parse import extract_duration
from datetime import timedelta
assert extract_duration("nothing") == (None, 'nothing')
assert extract_duration("Nineteen minutes past the hour") == (
timedelta(minutes=19),
"past the hour")
assert extract_duration("wake me up in three weeks, four hundred ninety seven"
" days, and three hundred 91.6 seconds") == (
timedelta(weeks=3, days=497, seconds=391.6),
"wake me up in , , and")
Dates:
from datetime import datetime
from lingua_franca.parse import extract_datetime, normalize
def extractWithFormat(text):
date = datetime(2017, 6, 27, 13, 4) # Tue June 27, 2017 @ 1:04pm
[extractedDate, leftover] = extract_datetime(text, date)
extractedDate = extractedDate.strftime("%Y-%m-%d %H:%M:%S")
return [extractedDate, leftover]
def testExtract(text, expected_date, expected_leftover):
res = extractWithFormat(normalize(text))
assert res[0] == expected_date
assert res[1] == expected_leftover
testExtract("What is the day after tomorrow's weather?",
"2017-06-29 00:00:00", "what is weather")
testExtract("Remind me at 10:45 pm",
"2017-06-27 22:45:00", "remind me")
This is just parsing. It also supports the other direction: generating pronounceable strings based on numbers, durations or dates.
By the way, itâs not visible in these examples but the library is multilingual, as can be seen in the test scripts.
Ah, sounds nice !!!
Snips always return duration slot in total seconds, seems the same here.
I also guess this is why they need to be builtins slots, not user slots. So rhasspy know the slot and can extract/parse it into usable values before returning found slots.
Say âtwenty one minutesâ
-> rhasspy see it as duration slot
-> extract_duration(âtwenty one minutesâ) -> 1260 sec
-> return slot {duration: 1260}
Canât wait to play with such builtins slots, extremely powerfull with snips.
Just in case, here is the list of snips builtins slots, in snips-nlu repo that have many interesting stuff (maybe fork it in case sonos delete everything ?)
Regarding duration, I though the result was the sum of minutes, but thatâs not the case. I wrote some python app for snips and had to write some function to parse the result to add duration to actual date etc.
Here is a dummy duration class I used for test:
class duration(object):
def __init__(self):
self.years = 0
self.quarters = 0
self.months = 3
self.weeks = 2
self.days = 14
self.hours = 0
self.minutes = 0
self.seconds = 0
self.precision = "Exact"
The conversion to total minutes happens in the jeedom plugin to program cron easily. So once rhasspy got duration builtin slot, I will handle that in the plugin.
LOL! Great minds think alike.
Iâve just created a new topic on the subject
Will Rhasspy 2.5 enhance built-in slots or will this get worked later ?
duration : Got similar working use with Number Ranges {0âŠ120}[minutes] for example. Harder for âIn one year, two months and three weeksâ
dateTime ?
year ? ( Guess could be handled with number ranges like [en|annĂ©e]{1900âŠ2050} )
age ? (same)
Is there some news on this subject ? Or are we still need to generate tons of sentence for duration_minutes, duration_days, duration_months and so on ?
No news yet on the subject, sorry