[Rehansanjay]

Follow-up to #7913. That PR closed this hole in the three plugins #7900 names; I then audited every streaming STT plugin for the same thing, and these four are the rest of it.

The bug, same as #7900

aiohttp defaults heartbeat to None, so a socket that dies without a FIN/RST leaves the receive side parked on ws.receive() forever. Nothing raises, the stream never reconnects, and stt.FallbackAdapter never switches. The agent stops hearing the user with no error anywhere.

The audit

plugin	state on main	this PR
gladia	streaming connect unbounded	heartbeat=30.0
nabrah	unbounded	heartbeat=30.0
sarvam (stt.py)	unbounded	heartbeat=30.0
slng	unbounded, and the ERROR frame swallowed	heartbeat=30.0 + raise on ERROR
gladia (streaming receive loop)	the ERROR frame swallowed	raise on ERROR
baseten, fireworksai, gradium	also unbounded — see the correction below	left to a follow-up
deepgram, elevenlabs, inworld, meta, microsoft-ai, openai, simplismart, smallestai, soniox, telnyx, xai	already pass a heartbeat	untouched
assemblyai	unbounded	left to #7547

Two plugins have more than one websocket, and only one each needed this:

• gladia's batch connect already passes ClientWSTimeout(ws_receive=..., ws_close=10). Only the streaming connect in _run had no bound.
• sarvam ships two streaming STTs and exports both. stt_streaming.py's RealtimeSpeechStream already passes a heartbeat; stt.py's SpeechStream, reached through STT, did not.

nabrah and sarvam already raise on an unexpected ERROR frame, so the heartbeat is all they were missing. gladia did not -- see the correction at the end. slng's not TEXT branch continued past it, which is the same hole #7913 found in the other three — so it raises now, keeping the cause from ws.exception() and matching the close branch above it.

Tests

nabrah and slng get the test from tests/test_plugin_soniox_stt.py: assert the socket is opened with the heartbeat. Both mutation-verified — removing the kwarg fails that plugin's test.

gladia and sarvam are not tested, and I'd rather say so than imply otherwise. Their connects are inline in _run / _run_connection rather than in a _connect_ws method, so they aren't reachable by the patterns this repo uses. Each change is the same kwarg as a bounded sibling connect in the same file. If you want them covered, the way is to lift those connects into methods, which is a bigger diff than the fix — happy to do it if you'd prefer that.

One thing worth flagging from the audit: baseten, fireworksai and gradium bound the read with a flat 5-second wait_for on receive(). That does catch a dead socket, but #5717 is about why a read timeout is the wrong instrument for STT, since a quiet user is the normal state. I haven't touched them, but they may be worth a look separately.

Correction after review

I originally wrote that gladia already raises on an unexpected ERROR frame. It does not. Its only ERROR branch is in the batch path, which is also the connect that was already bounded by ClientWSTimeout -- my audit matched that branch by text and reported the plugin as covered. The streaming _recv_messages_task had none, so the frame fell through to "Unexpected message type", the task ended normally, and _run broke instead of reconnecting: the heartbeat would have found the dead socket and the stream would still have died quietly.

Fixed in a follow-up commit with its own test, in a new file because tests/test_plugin_gladia_stt.py is gated behind pytest.mark.plugin("gladia"). I re-checked nabrah and sarvam by enclosing function rather than by text match, and both do raise in their streaming loops as described.

Second correction: baseten, fireworksai and gradium are vulnerable too

I wrote above that these three bound the read with asyncio.wait_for(ws.receive(), timeout=5) and are therefore safe. They do call it, but the handler is:

except asyncio.TimeoutError:
    if closing_ws:
        break
    continue      # round again, forever

A polling timeout, not a liveness bound. Measured against origin/main with #7900's probe extended to every streaming STT plugin, all three sit at 75s without noticing, next to 46.4s / 46.9s / 47.0s for smallestai, telnyx and deepgram. They belong in the fix; they are coming as a follow-up rather than growing this PR a third time.

[devin-ai-integration]

Devin Review found 1 potential issue.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

[devin-ai-integration]

🔴 Gladia heartbeat failure ends transcription silently

When Gladia's new heartbeat times out, _recv_messages_task logs the ERROR frame and exits normally after the socket closes. _run then completes without raising, so the fallback adapter cannot switch providers.

Learn more

A heartbeat timeout closes the aiohttp socket and delivers an ERROR frame to the waiting reader. The Gladia receive loop logs that frame without raising and then finishes when the closed socket ends iteration in _recv_messages_task. _run treats normal receive-task completion as the end of the stream. No connection error reaches the base stream or fallback adapter.

Example: The Gladia peer stops answering without sending FIN or RST. After an unanswered ping, the receive task logs the ERROR, finishes, and the stream closes rather than signaling provider failure.

Recommended fix: Handle WSMsgType.ERROR in _recv_messages_task by raising APIConnectionError, using self._ws.exception() as the cause when available. Add a test that triggers an ERROR frame and verifies _run propagates a connection failure.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[Rehansanjay]

Right, and it caught a false claim in my PR description. Fixed.

I wrote that gladia already raises on an unexpected ERROR frame. It does not. Its only ERROR branch is at stt.py:415, in the batch path — which is also the connect that was already bounded by ClientWSTimeout, so it was never the one at issue. My audit scan matched that branch and reported the plugin as covered. The streaming _recv_messages_task had no ERROR branch at all: the frame fell through to logger.warning("Unexpected message type"), the task ended normally, _run broke out instead of reconnecting, and the base stream only reports a failure when _run raises. So the heartbeat would have detected the dead socket and the stream would still have died quietly — the same hole #7913 found in inworld.

It raises now, keeping the cause from ws.exception(). Test in a new file, since tests/test_plugin_gladia_stt.py is gated behind pytest.mark.plugin("gladia") and would skip by default. Mutation-verified: DID NOT RAISE APIConnectionError without it.

I re-checked the other two the same way, by the enclosing function rather than by a text match:

• nabrah — _recv_task, raises self._stream_failure(...) on ERROR ✓
• sarvam — _process_messages, raises APIConnectionError on ERROR ✓

So the claim held for those two and only gladia was wrong. Updating the PR description to match.

[Rehansanjay]

Correcting myself again, and this one I published as a reason to leave three plugins alone.

I wrote above that baseten, fireworksai and gradium are not vulnerable because they bound the read with asyncio.wait_for(ws.receive(), timeout=5). That is wrong. All three have the same handler:

except asyncio.TimeoutError:
    if closing_ws:
        break
    continue      # round again, forever

It is a polling timeout, not a liveness bound — the timeout fires every five seconds and the loop simply carries on, so a dead socket is never escalated. I read the wait_for and concluded "bounded" without checking what the handler did with it.

I extended @DEVJHAWAR11's probe from #7900 to every streaming STT plugin and measured it against origin/main at d4cc4a941. A local server accepts the upgrade, then answers nothing, pings included; each plugin is fed silence and timed until it raises. The harness agrees with his numbers — deepgram raises at 47.0s against his 45.4s.

	time to raise
smallestai	46.4s
telnyx	46.9s
deepgram	47.0s
baseten	never, 75s
fireworksai	never, 75s
gradium	never, 75s
assemblyai, elevenlabs, gladia, inworld, openai	never, 75s

So those three belong in the fix rather than in the exemption list. I will send them as a follow-up rather than growing this PR again — it is already at four plugins and two corrections.

Two other things the measurement turned up, neither of which is this PR's business but both worth recording:

soniox and simplismart detect the dead socket and never surface it. Soniox's own log:

 5001ms  connection established
51278ms  error frame: ServerTimeoutError('No PONG received after 15.0 seconds')
51279ms  WebSocket closed; requesting reconnect
51282ms  connection established     <- straight back into the dead state

It detects at ~46s, reconnects, and repeats indefinitely. conn_options.max_retry=0 does not bound it, because that is the plugin's own reconnect loop rather than the base class's retry, so a FallbackAdapter is never told and the caller stays deaf. simplismart logs "connection lost" at ~50.6s and likewise never raises. Whether an unbounded internal reconnect should ever surface failure looks like a policy question for you rather than a bug fix, so I have not touched either.

Six plugins could not be measured and I would rather say so than guess: meta's constructor requires an absolute wss://; slng's endpoint validation rejected every local shape I tried; sarvam's session handshake answers 403 against my local server; microsoft-ai, nabrah and xai expose no endpoint override at all.

[Rehansanjay]

The measurement behind this is now a repo rather than a number in a comment: https://github.com/Rehansanjay/livekit-stt-halfopen

It is @DEVJHAWAR11's probe from #7900 extended to all nineteen streaming STT plugins, with the harness, the per-plugin construction table and the raw results. Against d4cc4a941: three plugins fail over in ~46-47s, two detect the dead socket and never propagate it, eight never notice inside a 75-second watch, and six cannot be pointed at a local server from their public constructors — those are listed with the reason rather than guessed at.

The mechanism turns out to be one parameter. Every plugin that raised passes heartbeat= to ws_connect; every plugin that never noticed does not; and the two that detect-but-stay-silent pass one and then log the ERROR frame instead of raising it. The heartbeat column is read with a paren-balanced parse of each call, after an earlier text-search version of that audit got three plugins wrong.

Re-runnable from any livekit/agents checkout, so the numbers in this PR do not have to be taken on trust.

[davidzhao]

Please keep test_an_error_frame_raises_so_the_stream_can_fail_over and drop the two heartbeat-kwarg tests (nabrah, slng).

[davidzhao]

Thanks @Rehansanjay for the effort here! We're handling WebSocket ERROR frames centrally in the liveness monitor in #8147, so closing this in favor of that.