After training or save sentences --> kaldi graph folder is empty

Bozor · February 15, 2020, 4:48pm

I’m having a weird problem. If I add something to my sentences.ini using the webUI I’m getting following error in the docker container and afterwards my graph folder inside the kaldi folder is empty.

INFO:quart.serving:172.17.0.1:50742 GET /api/sentences 1.1 200 343 12831
.  language_model:intent_counts
.  language_model:intent_model
.  language_model:intent_arpa
.  vocab
.  vocab_dict
DEBUG:vocab_dict:Forcing lower-case
DEBUG:vocab_dict:Loading dictionary from /profiles/de/kaldi_custom_words.txt
DEBUG:vocab_dict:Loading dictionary from /profiles/de/kaldi/base_dictionary.txt
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<WebsocketMixin.handle_websocket() done, defined at /usr/local/lib/python3.6/dist-packages/hypercorn/asgi/wsproto.py:60> exception=LocalProtocolError('Connection cannot be closed in state ConnectionState.CLOSED',)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/hypercorn/asgi/wsproto.py", line 91, in handle_asgi_app
    await asgi_instance(self.asgi_receive, self.asgi_send)
  File "/usr/local/lib/python3.6/dist-packages/quart/asgi.py", line 106, in __call__
    await _cancel_tasks(pending)
  File "/usr/local/lib/python3.6/dist-packages/quart/asgi.py", line 189, in _cancel_tasks
    raise task.exception()
  File "/usr/local/lib/python3.6/dist-packages/quart/asgi.py", line 157, in handle_websocket
    'code': 1000,
  File "/usr/local/lib/python3.6/dist-packages/hypercorn/asgi/wsproto.py", line 149, in asgi_send
    await self.asend(CloseConnection(code=int(message["code"])))
  File "/usr/local/lib/python3.6/dist-packages/hypercorn/asyncio/wsproto.py", line 93, in asend
    self.write(self.connection.send(event))
  File "/usr/local/lib/python3.6/dist-packages/wsproto/__init__.py", line 64, in send
    data += self.connection.send(event)
  File "/usr/local/lib/python3.6/dist-packages/wsproto/connection.py", line 101, in send
    "Connection cannot be closed in state %s" % self.state
wsproto.utilities.LocalProtocolError: Connection cannot be closed in state ConnectionState.CLOSED

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/hypercorn/asgi/wsproto.py", line 78, in handle_websocket
    await self.handle_asgi_app(event)
  File "/usr/local/lib/python3.6/dist-packages/hypercorn/asgi/wsproto.py", line 99, in handle_asgi_app
    await self.asend(CloseConnection(code=CloseReason.ABNORMAL_CLOSURE))
  File "/usr/local/lib/python3.6/dist-packages/hypercorn/asyncio/wsproto.py", line 93, in asend
    self.write(self.connection.send(event))
  File "/usr/local/lib/python3.6/dist-packages/wsproto/__init__.py", line 64, in send
    data += self.connection.send(event)
  File "/usr/local/lib/python3.6/dist-packages/wsproto/connection.py", line 101, in send
    "Connection cannot be closed in state %s" % self.state
wsproto.utilities.LocalProtocolError: Connection cannot be closed in state ConnectionState.CLOSED
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<WebsocketMixin.handle_websocket() done, defined at /usr/local/lib/python3.6/dist-packages/hypercorn/asgi/wsproto.py:60> exception=LocalProtocolError('Connection cannot be closed in state ConnectionState.CLOSED',)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/hypercorn/asgi/wsproto.py", line 91, in handle_asgi_app
    await asgi_instance(self.asgi_receive, self.asgi_send)
  File "/usr/local/lib/python3.6/dist-packages/quart/asgi.py", line 106, in __call__
    await _cancel_tasks(pending)
  File "/usr/local/lib/python3.6/dist-packages/quart/asgi.py", line 189, in _cancel_tasks
    raise task.exception()
  File "/usr/local/lib/python3.6/dist-packages/quart/asgi.py", line 157, in handle_websocket
    'code': 1000,
  File "/usr/local/lib/python3.6/dist-packages/hypercorn/asgi/wsproto.py", line 149, in asgi_send
    await self.asend(CloseConnection(code=int(message["code"])))
  File "/usr/local/lib/python3.6/dist-packages/hypercorn/asyncio/wsproto.py", line 93, in asend
    self.write(self.connection.send(event))
  File "/usr/local/lib/python3.6/dist-packages/wsproto/__init__.py", line 64, in send
    data += self.connection.send(event)
  File "/usr/local/lib/python3.6/dist-packages/wsproto/connection.py", line 101, in send
    "Connection cannot be closed in state %s" % self.state
wsproto.utilities.LocalProtocolError: Connection cannot be closed in state ConnectionState.CLOSED

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/hypercorn/asgi/wsproto.py", line 78, in handle_websocket
    await self.handle_asgi_app(event)
  File "/usr/local/lib/python3.6/dist-packages/hypercorn/asgi/wsproto.py", line 99, in handle_asgi_app
    await self.asend(CloseConnection(code=CloseReason.ABNORMAL_CLOSURE))
  File "/usr/local/lib/python3.6/dist-packages/hypercorn/asyncio/wsproto.py", line 93, in asend
    self.write(self.connection.send(event))
  File "/usr/local/lib/python3.6/dist-packages/wsproto/__init__.py", line 64, in send
    data += self.connection.send(event)
  File "/usr/local/lib/python3.6/dist-packages/wsproto/connection.py", line 101, in send
    "Connection cannot be closed in state %s" % self.state
wsproto.utilities.LocalProtocolError: Connection cannot be closed in state ConnectionState.CLOSED
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<WebsocketMixin.handle_websocket() done, defined at /usr/local/lib/python3.6/dist-packages/hypercorn/asgi/wsproto.py:60> exception=LocalProtocolError('Connection cannot be closed in state ConnectionState.CLOSED',)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/hypercorn/asgi/wsproto.py", line 91, in handle_asgi_app
    await asgi_instance(self.asgi_receive, self.asgi_send)
  File "/usr/local/lib/python3.6/dist-packages/quart/asgi.py", line 106, in __call__
    await _cancel_tasks(pending)
  File "/usr/local/lib/python3.6/dist-packages/quart/asgi.py", line 189, in _cancel_tasks
    raise task.exception()
  File "/usr/local/lib/python3.6/dist-packages/quart/asgi.py", line 157, in handle_websocket
    'code': 1000,
  File "/usr/local/lib/python3.6/dist-packages/hypercorn/asgi/wsproto.py", line 149, in asgi_send
    await self.asend(CloseConnection(code=int(message["code"])))
  File "/usr/local/lib/python3.6/dist-packages/hypercorn/asyncio/wsproto.py", line 93, in asend
    self.write(self.connection.send(event))
  File "/usr/local/lib/python3.6/dist-packages/wsproto/__init__.py", line 64, in send
    data += self.connection.send(event)
  File "/usr/local/lib/python3.6/dist-packages/wsproto/connection.py", line 101, in send
    "Connection cannot be closed in state %s" % self.state
wsproto.utilities.LocalProtocolError: Connection cannot be closed in state ConnectionState.CLOSED

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/hypercorn/asgi/wsproto.py", line 78, in handle_websocket
    await self.handle_asgi_app(event)
  File "/usr/local/lib/python3.6/dist-packages/hypercorn/asgi/wsproto.py", line 99, in handle_asgi_app
    await self.asend(CloseConnection(code=CloseReason.ABNORMAL_CLOSURE))
  File "/usr/local/lib/python3.6/dist-packages/hypercorn/asyncio/wsproto.py", line 93, in asend
    self.write(self.connection.send(event))
  File "/usr/local/lib/python3.6/dist-packages/wsproto/__init__.py", line 64, in send
    data += self.connection.send(event)
  File "/usr/local/lib/python3.6/dist-packages/wsproto/connection.py", line 101, in send
    "Connection cannot be closed in state %s" % self.state
wsproto.utilities.LocalProtocolError: Connection cannot be closed in state ConnectionState.CLOSED
DEBUG:vocab_dict:Loaded 27 word(s) from /profiles/de/vocab.txt
WARNING:vocab_dict:1 word(s) are unknown
WARNING:vocab_dict:swiss
DEBUG:vocab_dict:Wrote unknown words to /profiles/de/kaldi/unknown_words.txt
DEBUG:train:Guessing pronunciations for unknown word(s)
.  kaldi_train
arpa2fst --disambig-symbol=#0 --read-symbol-table=/profiles/de/kaldi/model/data/lang/words.txt - /profiles/de/kaldi/model/data/lang/G.fst 
ERROR (arpa2fst[5.5]:Input():kaldi-io.cc:756) Error opening input stream /profiles/de/kaldi/model/data/lang/words.txt

[ Stack-Trace: ]
/opt/kaldi/src/lib/libkaldi-base.so(kaldi::MessageLogger::LogMessage() const+0xb42) [0x7ffb33b7d642]
arpa2fst(kaldi::MessageLogger::LogAndThrow::operator=(kaldi::MessageLogger const&)+0x21) [0x56095a5b6459]
/opt/kaldi/src/lib/libkaldi-util.so(kaldi::Input::Input(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool*)+0xbe) [0x7ffb33d9fa70]
arpa2fst(main+0x596) [0x56095a5b49a0]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7ffb32a9cb97]
arpa2fst(_start+0x2a) [0x56095a5b432a]

kaldi::KaldiFatalErrorDEBUG:DialogueManager:training_sentences -> training_intent
DEBUG:FuzzyWuzzyIntentTrainer:Wrote intent examples to /profiles/de/intent_examples.json
INFO:DialogueManager:Training complete
DEBUG:DialogueManager:Reloading actors
DEBUG:SnowboyWakeListener:loaded -> stopped
DEBUG:SnowboyWakeListener:stopped -> stopped
DEBUG:KaldiDecoder:started -> stopped
DEBUG:KaldiDecoder:stopped -> stopped
DEBUG:FuzzyWuzzyRecognizer:loaded -> stopped
DEBUG:FuzzyWuzzyRecognizer:stopped -> stopped
DEBUG:DialogueManager:training_intent -> training_loading
DEBUG:KaldiDecoder: -> started
DEBUG:SnowboyWakeListener: -> started
DEBUG:FuzzyWuzzyRecognizer: -> started
DEBUG:SnowboyWakeListener:Loading snowboy model from /profiles/de/snowboy/snowboy.umdl
DEBUG:SnowboyWakeListener:Loaded snowboy model snowboy/snowboy.umdl ({'sensitivity': '0.5', 'audio_gain': 1.0, 'apply_frontend': False})
DEBUG:SnowboyWakeListener:started -> loaded
DEBUG:FuzzyWuzzyRecognizer:Loaded examples from /profiles/de/intent_examples.json
DEBUG:FuzzyWuzzyRecognizer:started -> loaded
INFO:DialogueManager:Actors reloaded
DEBUG:DialogueManager:training_loading -> ready
INFO:DialogueManager:Automatically listening for wake word
DEBUG:DialogueManager:ready -> asleep
DEBUG:SnowboyWakeListener:loaded -> listening
DEBUG:InboxActor: -> stopped
[2020-02-15 16:42:12,031] 172.17.0.1:50734 POST /api/train 1.1 200 36 6303457
INFO:quart.serving:172.17.0.1:50734 POST /api/train 1.1 200 36 6303457
DEBUG:InboxActor: -> stopped
[2020-02-15 16:42:12,080] 172.17.0.1:50734 GET /api/problems 1.1 200 389 2433
INFO:quart.serving:172.17.0.1:50734 GET /api/problems 1.1 200 389 2433

Is anyone having the same issue or knows how to fix this?

frkos · February 16, 2020, 6:17pm

It’s really weird… and I don’t know how to fix it)
But i would try to edit file manually, or at least with another browser. Maybe you have some special characters?
And what are you referring to add there?

Bozor · February 18, 2020, 6:57pm

This is my current sentences.ini:

[GetTime]
wie spät ist es
sag mir die uhrzeit
wie viel uhr ist es
wie ist die zeit
welche zeit ist [es]

[StartRadio]
(starte | spiele) [das] radio
ich möchte radio hören
lass radio laufen [bitte]
mach [das] radio an

[StopRadio]
stoppe [die | das] (Musik | Radio)
halte alles an
ruhe bitte
sei leise

[SetRadioStation]
(starte | spiele) (swr3 | radio swiss pop)
mach swr3 an
mach radio swiss pop an

I’ve thought as well about special german characters, but the second code line is there since the beginning and the troubles started like two days ago.
I’ll edit the file now from vs code and report.

Is someone having trouble to learn kaldi short german words as well? For example swr3 or other radio station names.

Whats the best way to teach them to kaldi? Or simpler the best way to get phenomens?

thomas_cologne · February 18, 2020, 8:31pm

Did you try that with an english profile instead german one?
Maybe there is some troubleshooting with the german kaldi folders

Bozor · February 18, 2020, 8:50pm

I’ve used a german profile. Do u mean I shall try it with an english kaldi?

thomas_cologne · February 18, 2020, 11:22pm

Would be worth to give it a try. I just noticed problems with kaldi and german profile several times in this community.