Hi @synesthesiam!
I have HASS running in an old Raspberry PI 3B+, and it wasn’t enough powerful to run “faster-whisper” (or any other ASR program) faster enough, so I take another approach…
I have created a fork of the Rhasspy3 project and added a new “OpenAI Audio API” ASR program, that uses the online audio/whisper API of OpenAI (requires and API key): rhasspy3/programs/asr/openai-audio-api at feature/programs-asr-openai-audio-api · gregoriohc/rhasspy3 · GitHub
Also, I forked the “wyoming-addons” repo to add the Docker build for that program: wyoming-addons/openai-audio-api at feature/docker-openai-audio-api · gregoriohc/wyoming-addons · GitHub
The python package used in the Docker image is currently published in TestPyPi (wyoming-openai-audio-api) and it’s source code it’s “conveniently” sitting 
in my forked Rhasspy3 repo (inside /programs/asr/openai-audio-api/server).
Do you think this could be interesting for other people? I can make a pull request if you want.
Thanks for your work!