[Bugfix][Router] Keep avg_latency within the request-stats sliding window - #1110
Open
David-Wu1119 wants to merge 1 commit into
Open
David-Wu1119 wants to merge 1 commit into
David-Wu1119 wants to merge 1 commit into
Conversation
…ndow get_request_stats drops samples older than the sliding window from the QPS and TTFT monitors before reading them, but not from the latency monitor, so avg_latency averaged every request since the router started and stayed frozen at the last value once an engine went idle. Drop old latency samples the same way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: David-Wu1119 <133224895+David-Wu1119@users.noreply.github.com>
David-Wu1119
requested review from
ApostaC,
Shaoting-Feng,
YuhanLiu11 and
ruizhang0101
as code owners
October 1, 2026 06:21
Contributor
There was a problem hiding this comment.
Code Review
This pull request ensures that requests outside the sliding window are correctly dropped when calculating average latency. This is achieved by calling update_no_value on the latency monitors within get_request_stats. Additionally, a new unit test test_avg_latency_drops_requests_outside_the_window has been added to verify this behavior. No review comments were provided, and the changes appear correct and complete.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
RequestStatsMonitor.get_request_statsreports per-engine statistics over the--request-stats-windowsliding window. Before reading the QPS and TTFT monitors, it callsupdate_no_value(current_time)on each one so that samples older than the window are dropped. It never did that for the latency monitor, soavg_latencyonly shed old samples when a new request finished on that engine. Once an engine went idle,avg_latencystayed at the last value forever: the router kept exporting it (vllm:avg_latency) and logging it, while QPS and TTFT for the same engine had already dropped to 0 / -1.For example, with a 10 s window, take one request that finishes 2 s after it started. 100 s later the stats read
qps=0.0, ttft=-1, avg_latency=2.0. This change makes itavg_latency=-1, the same as TTFT, by dropping old latency samples the same way. (Open #1054 adds the same cleanup foravg_decoding_length; this is the latency counterpart.)Tests:
test_avg_latency_drops_requests_outside_the_windowinsrc/tests/test_request_stats.pychecks the example above. It fails onmain(avg_latencyis still 2.0) and passes with this change.pytest src/tests: 241 passed, Python 3.12,pip install -e .plus pytest, pytest-asyncio and httpx.pre-commit runon both files passes.Found while auditing the router's request statistics with AI assistance; the fix and test were written with Claude Code.
-swhen doinggit commit[Bugfix],[Feat], and[CI].🤖 Generated with Claude Code