Skip to content

[Router] Add --timeout-keep-alive to configure uvicorn's idle keep-alive timeout - #1098

Open
mrembalski wants to merge 2 commits into
vllm-project:mainfrom
mrembalski:mrembalski/router-timeout-keep-alive
Open

mrembalski wants to merge 2 commits into
vllm-project:mainfrom
mrembalski:mrembalski/router-timeout-keep-alive

Conversation

@mrembalski

@mrembalski mrembalski commented Sep 22, 2026 •

Copy link
Copy Markdown

The router started uvicorn with the default 5 s keep-alive timeout and exposed no way to change it, so clients whose requests are spaced more than 5 s apart reconnect on every request even when they pool connections. Default unchanged.

FIX #1102


  • Make sure the code changes pass the pre-commit checks.
  • Sign-off your commit by using -s when doing git commit
  • Try to classify PRs for easy understanding of the type of changes, such as [Bugfix], [Feat], and [CI].

…ive timeout

The router started uvicorn with the default 5 s keep-alive timeout and exposed no way to change it, so clients whose requests are spaced more than 5 s apart reconnect on every request even when they pool connections. Default unchanged.

Signed-off-by: Michał Rembalski <michal.rembalski@elevenlabs.io>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new command-line argument --timeout-keep-alive to configure the idle HTTP keep-alive connection timeout, which is then passed to the server configuration in app.py. There are no review comments, and I have no additional feedback to provide.

@mrembalski mrembalski changed the title [Router] Add --timeout-keep-alive to configure uvicorn's idle keep-al… [Router] Add --timeout-keep-alive to configure uvicorn's idle keep-alive timeout Sep 24, 2026
@mrembalski

Copy link
Copy Markdown
Author

@Shaoting-Feng @ruizhang0101 friendly ping per the guideline

@ruizhang0101 ruizhang0101 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feature: expose uvicorn timeout_keep_alive as a router flag

2 participants