Option to start a reply on time when caller noise began while the agent was thinking (1.6.6 behavior) #7837

Activity

1. 31groot commented on Oct 15, 2026

   

   31groot

   on Oct 15, 2026

   Last edited by 31groot

   More actions

   hello, i want to take this
2. Rehansanjay commented on Oct 18, 2026

   

   Rehansanjay

   on Oct 18, 2026

   Last edited by Rehansanjay

   Contributor

   More actions

   Opened #7919 for this.

   @31groot you said three days ago you wanted to take it and I did not want to tread on that, so: if you have something in progress, say so and I will close mine in favour of yours. No hard feelings either way — I only picked it up because nothing had appeared and the measurement here made it straightforward.

   Implemented as the issue suggests: hold_reply_while_user_speaking, default True so nothing changes by default, and scoped to pauses recorded while the agent was thinking — an interruption during playback keeps today's behaviour.

   One detail for @assembled-caden, since you are monkeypatching _reconcile_playout_pause: resuming unconditionally there is not quite enough on current main. Releasing the pause and falling through lets the block below re-pause it, because the agent is still not "speaking" and the caller is still not silent. The PR returns straight after resuming.

   👍React with 👍2Reacted by Parv Agrawal and Caden Marinozzi
3. longcw

   pinned this issue on Oct 19, 2026
4. longcw commented on Oct 19, 2026

   

   longcw

   on Oct 19, 2026

   Contributor

   More actions

   The hold is a trade-off: before the first audio plays, a breath and the start of a word look the same, so we accept some dead air on noise to keep a reply from starting and then being cut off by the final transcript.

   Would a shorter false_interruption_timeout only for pauses that start while the agent is thinking cover your case? if so we may separate the timeout for speaking and thinking states.
5. longcw

   unpinned this issue on Oct 19, 2026
6. Rehansanjay commented on Oct 21, 2026

   

   Rehansanjay

   on Oct 21, 2026

   Contributor

   More actions

   I measured this before answering, because a shorter timeout is what I assumed too, and on 3c6e65124 there turns out to be nothing to shorten for most of the rows in the table.

   A pause recorded while the agent is thinking already carries timeout=0. Both paths that create one hardcode it:

   • on_start_of_speech → _update_paused_speech(current_speech, timeout=0) — agent_activity.py:2451
   • _reconcile_playout_pause → _update_paused_speech(speech_handle, timeout=0) — agent_activity.py:5010

   false_interruption_timeout reaches one of those only when _interrupt_by_audio_activity upgrades it at agent_activity.py:2404, and that needs on_vad_inference_done to have seen speech_duration >= min_duration — 0.5 s by default.

   Driving those hooks directly, agent in thinking, false_interruption_timeout=2.0:

   [on_start_of_speech, thinking]         timeout = 0      (configured: 2.0)
   [_reconcile_playout_pause, thinking]   timeout = 0
   [before _interrupt_by_audio_activity]  0   ->   [after] 2.0
   

   So separating the timeout by state acts on exactly one row of @assembled-caden's table — the 0.6 s one that lands at 5.35 s, where it would take out most of two seconds. The 0.1–0.3 s rows already run at timeout zero and still cost +0.57–0.77 s. That cost is the wait for VAD end-of-speech, because the pause is released from on_end_of_speech; their numbers are consistent with a min_silence_duration near the 0.55 s Silero default. No timeout value reduces it — it is already zero.

   That is the half I care about, because the sounds that actually show up near a reply's start on a phone line are breaths, line clicks and short "mm"s, and those are all under min_duration.

   Your trade-off framing is right, and I would rather concede it than argue around it. Before the first frame a breath and a word onset are the same signal, and the price of guessing wrong is a reply that starts and is then cut by the final transcript. That is why the option defaults to True — current behaviour unless someone asks for the other one — and when they do ask, the downside is bounded: a reply interrupted ~100 ms in, which is what happens anyway if the caller interrupts a moment later.

   On the PR's third change, in case it reads as unrelated: the authorization gather also waits on _user_silence_event (3207, 3743, 4171, 4402). That gate is not tied to EOS — on_vad_inference_done sets it once accumulated silence passes min_endpointing_delay / 2, at 2505-2512 — but it is clear while the caller is still making noise, so releasing the pause alone left the reply waiting there and the option still did nothing. Hence both.

   If you would still rather have a timeout than a flag, say so and I will open it; it is the smaller change. I would only want it described as covering the >= min_duration case, because the sub-0.5 s rows need the pause not to be taken, or released at forwarding, which is what the boolean does.

   the probe, if you want to re-run it

   # tests/test_timeout_probe.py
   import time
   import pytest
   from livekit.agents import Agent, AgentSession
   from livekit.agents.voice.agent_activity import AgentActivity
   from livekit.agents.voice.speech_handle import SpeechHandle
   from .fake_io import FakeAudioOutput
   
   pytestmark = pytest.mark.unit
   
   
   @pytest.fixture
   async def act():
       session = AgentSession(
           turn_handling={
               "interruption": {
                   "resume_false_interruption": True,
                   "false_interruption_timeout": 2.0,
                   "min_duration": 0.5,
               }
           }
       )
       session.output.audio = FakeAudioOutput(can_pause=True)
       a = AgentActivity(Agent(instructions="test"), session)
       session._update_agent_state("thinking")
       yield a
       await session.aclose()
   
   
   async def test_sos_while_thinking(act):
       act._current_speech = SpeechHandle.create()
       act.on_start_of_speech(None, time.time())
       assert act._paused_speech.timeout == 0
   
   
   async def test_reconcile_while_thinking(act):
       act._user_silence_event.clear()  # caller still making noise
       act._reconcile_playout_pause(SpeechHandle.create())
       assert act._paused_speech.timeout == 0
   
   
   async def test_only_an_interruption_upgrades_it(act):
       act._current_speech = SpeechHandle.create()
       act.on_start_of_speech(None, time.time())
       assert act._paused_speech.timeout == 0
       act._interruption_by_audio_activity_enabled = True
       act._interrupt_by_audio_activity()
       assert act._paused_speech.timeout == 2.0
7. Rehansanjay commented on Oct 21, 2026

   

   Rehansanjay

   on Oct 21, 2026

   Contributor

   More actions

   @assembled-caden a heads-up, since the design decision here is live and it determines whether you get to drop that _reconcile_playout_pause monkeypatch or keep carrying it.

   @longcw's counter-proposal above is a shorter false_interruption_timeout for pauses that start while the agent is thinking, rather than the boolean in #7919. I measured it against 3c6e65124 before answering, because it was what I assumed too, and it only covers part of your table:

   • A pause recorded while the agent is thinking already carries timeout=0 — agent_activity.py:2451 (on_start_of_speech) and :5010 (_reconcile_playout_pause) both hardcode it.
   • The configured value reaches such a pause only when _interrupt_by_audio_activity upgrades it at :2404, and that needs on_vad_inference_done to have seen speech_duration >= min_duration — 0.5 s by default.

   So separating the timeout by state acts on the 0.6 s / 5.35 s row of your table and does nothing for the 0.1–0.3 s rows, which already run at timeout zero and still cost you +0.57–0.77 s waiting for VAD end-of-speech.

   The thing that would settle this is your data, not mine. If the sounds that actually trigger this on your calls are mostly under min_duration — breaths, line clicks, short "mm"s — then the timeout split leaves your problem in place and the argument for a flag is straightforward. If they are mostly longer than 0.5 s, longcw's version is the smaller change and probably the right one, and I would rather implement that than argue for mine.

   Do you have a rough duration distribution for them? Even "most are short" or "most are around half a second" would be more persuasive than my synthetic numbers, since you are the one running this on real calls.

   Also worth flagging for your patch specifically, if you have not hit it yet: resuming unconditionally in _reconcile_playout_pause is not quite enough on current main. Releasing the pause and falling through lets the block below re-pause it, because the agent is still not "speaking" and the caller is still not silent. #7919 returns straight after resuming for that reason.
8. assembled-caden commented on Oct 21, 2026

   

   assembled-caden

   on Oct 21, 2026

   Author

   More actions

   Thanks both. To answer @longcw : no a thinking only timeout wouldn't cover most of it. I confirmed on 1.8.6 that pauses recorded while thinking already carry timeout=0 (both on_start_of_speech and _reconcile_playout_pause). The 0.1-0.3 s rows are the wait for VAD end-of-speech, not a timeout.

   @Rehansanjay : I don't have a measured duration breakdown from real calls yet. The table came from scripted VAD/STT runs. From listening, the common triggers are breaths, line clicks and short "mm"s, which would mostly be under min_duration. I'll post numbers if I get them.

   In terms ofthe patch: ours replaces _reconcile_playout_pause entirely and doesn't fall through, so the re-pause doesn't happen. A test checks first-audio timing with noise at reply start. Good catch for anyone copying it, though.

   I'm in favor of the boolean in #7919, default True, scoped to pauses taken while thinking.
9. Rehansanjay commented on Oct 22, 2026

   

   Rehansanjay

   on Oct 22, 2026

   Last edited by Rehansanjay

   Contributor

   More actions

   @assembled-caden thank you — "mostly under min_duration" is the part that actually decides this, and it is the one thing neither @longcw nor I could supply. Scripted runs can show the mechanism but not which sounds really turn up on a call.

   Noted on the patch: replacing _reconcile_playout_pause outright sidesteps the fall-through entirely, so that warning was not news to you. It only bites the narrower version that releases the pause and continues.

   So where this stands, as I read it: the thinking-state timeout split would help the ≥0.5 s case and leave the sub-0.5 s one where it is, and the sub-0.5 s one is what you actually hear. That is an argument for the flag rather than against the timeout — if you also want a separate timeout for thinking vs speaking states, @longcw, the two are not exclusive and I am happy to send that as well.

   #7919 now has current main (3c6e65124) merged in and still defaults to True, so nothing changes for anyone who does not ask. (Correcting myself: when I first posted this it was 49 commits behind and I said it was rebased, which it was not. It is up to date now.) The three gates are in the description above if it helps review: the forwarding pause, the authorization gather, and the block that re-creates the pause with timeout zero when start-of-speech lands before the handle becomes current.

   No rush from my side. If you would rather have the timeout shape than the boolean, say so and I will open it instead — I would rather the dead air go away than own the patch that removes it.
10. Rehansanjay commented on Oct 27, 2026

    

    Rehansanjay

    on Oct 27, 2026

    Contributor

    More actions

    @assembled-caden you said the duration breakdown was the missing piece and that you did not
    have one from real calls. I cannot get your calls, but the boundary itself is measurable, so
    here it is against 3a6248cb7 using this repo's own fake harness (tests/fake_session.py),
    with min_duration at its default 0.5s.

    A real turn, then a burst with an empty transcript beginning while the agent is thinking --
    VAD start/end-of-speech fire and STT emits nothing, which is what a cough or a door is.

    burst	reply starts	held vs baseline	registers as an interruption?
    none	3.03s	—	—
    0.05s	3.46s	+0.43s	no
    0.10s	3.50s	+0.48s	no
    0.20s	3.60s	+0.58s	no
    0.30s	3.71s	+0.68s	no
    0.40s	3.81s	+0.79s	no
    0.50s	5.91s	+2.89s	yes
    0.60s	6.01s	+2.98s	yes
    0.80s	6.22s	+3.19s	yes

    The sub-threshold rows are a pure VAD wait. Every one of them fits
    burst + min_silence_duration, measured from when the reply was ready: the harness VAD uses
    min_silence_duration = 0.5, the burst opens at 2.9s and the reply is ready at 3.03s, so the
    predicted hold is 0.37 + burst — 0.42, 0.47, 0.57, 0.67, 0.77 against 0.43, 0.48, 0.58, 0.68,
    0.79 measured. Within 20ms on all five. That is your "the 0.1-0.3s rows are the wait for VAD
    end-of-speech, not a timeout", with a number on it.

    This is what decides @longcw's proposal. A shorter false_interruption_timeout for pauses
    that begin while thinking cannot touch those five rows, because no timeout runs in them. The
    reply is held until the VAD says the noise stopped, and that is min_silence_duration, not a
    timeout. It is the same reason I gave on 21 October — a pause recorded while thinking already
    carries timeout=0 — now visible as a duration rather than as a code path.

    Where a shorter timeout would help is the bottom three rows: +2.89s at a 0.5s burst is the
    VAD wait plus false_interruption_timeout of 2.0, and each of those is 0.37 + burst + 2.0 to
    within 20ms. So the counter-proposal covers exactly the bursts long enough to register as real
    interruptions -- which, per @assembled-caden's listening, are not the common case. The common
    case gets nothing from it.

    Caveats, since this is synthetic. Fake VAD and STT, not real calls, so it shows the
    mechanism and the boundary rather than your population of noises. The timing proxy is the first
    agent_state_changed to speaking, not a literal audio frame. The hold scales with the VAD's
    min_silence_duration, so a deployment that tunes it away from 0.5 shifts every sub-threshold
    row with it.

    One methodological note worth stating, because it nearly produced the opposite conclusion. My
    first run opened the burst at 2.0s and reported +0.00s for every sub-threshold duration,
    which reads as "short noise is already free" and would have argued for the counter-proposal.
    It was an artefact: VAD end-of-speech lands at burst end + 0.5, so a short burst at 2.0s was
    already over before the reply was ready at 3.03s, and the two never overlapped. Five rows at
    exactly 0.00s was the tell. The burst has to still be open when the reply becomes ready, which
    is why it starts at 2.9s above.

    @assembled-caden if you can tell me roughly where your noises sit relative to 0.5s -- even
    "nearly all under" versus "a mix" -- that is enough to say which half of this table your calls
    live in.