Using External MQTT breaks microphone

I have rhasspy running on one pi, and home assistant on another. I have a matrix microphone on the rhasspy. It will wake and send an event to HA so long as I’m using the internal MQTT, but if I set it to use the mosquitto broker on HA, rhasspy resets PyAudio to default device and will not recognize any other device. Using the external broker is needed for many actions in HA; the broker is working there. Any help is appreciated.