Repository navigation
Description
Problem
Python has no supported way to fail over from one model or provider to another when a call fails. Common triggers are a 429, a 5xx or "overloaded" response, a regional outage, or a capacity limit on an Azure OpenAI PTU deployment. .NET already has this: Microsoft.Extensions.AI 10.9+ ships FailoverChatClient and OrderedFailoverChatClient (dotnet/extensions#7662), and the .NET Agent Framework builds on them (#7641, sample Agent_Step24_MultiModelRouting). Python has nothing comparable.
Writing it yourself is not safe in Python. Every shipped client mixes FunctionInvocationLayer into the client class:
class OpenAIChatClient(FunctionInvocationLayer, ChatMiddlewareLayer, ChatTelemetryLayer, RawOpenAIChatClient): ...So the obvious wrapper ("try client A, if it raises try client B") fails over the whole tool loop, not the single model call that failed. If A has already executed tools and then fails on its next model call, B starts again from the original messages and executes the same tools a second time.
Minimal repro against main (scripted clients, no network):
SENT = []
@tool(approval_mode="never_require")
def send_email(to: str) -> str:
SENT.append(to); return f"sent to {to}"
class ScriptedClient(FunctionInvocationLayer, BaseChatClient):
# model call 1 -> asks for send_email; model call 2 -> answers
# primary is configured to raise ChatClientException("429") on model call 2
...
class NaiveFallbackClient:
async def get_response(self, messages, **kw):
for c in self.clients:
try: return await c.get_response(messages, **kw)
except ChatClientException: continue
await NaiveFallbackClient(primary, fallback).get_response(msgs, options={"tools": [send_email]})Output:
failover: primary: 429 Too Many Requests
final: done via fallback
send_email executions: 2 ['cfo@contoso.com', 'cfo@contoso.com']
The caller sees a successful response, but the side effect ran twice. This is the same class of problem as #9031 and #10172, but here the user's own resilience code causes it.
A second gap: there is no provider-neutral way to decide whether an error should trigger failover. Clients raise a plain ChatClientException with no status code and no retryable flag. Callers have to unwrap the provider SDK exception (openai.RateLimitError, Anthropic status_code, and so on) per provider. #9904 showed that even the shipped auto_retry.py sample got this wrong.
Use Case and Evidence
- Azure OpenAI capacity and quota: overflow from a PTU deployment to pay-as-you-go, or to a second region, when the first returns 429/503. This is standard production guidance for Azure OpenAI customers, and agents with tools are exactly the workloads that hit these limits.
- Provider outages: overflow from OpenAI to Anthropic or Bedrock (or the reverse) during an incident, without stopping a long-running harness or loop agent.
- Cost tiers: try a small or local model first and fall back to a larger hosted model on failure.
- Peer frameworks treat this as basic infrastructure: Pydantic AI
FallbackModel(withfallback_on), LangChainModelFallbackMiddleware(which runs per model call inside the agent loop), LiteLLM Router fallbacks, and MEAIFailoverChatClientin .NET. - Existing Python MAF content (for example, community posts on an "SLM-default, LLM-fallback" pattern) builds fallback by hand with workflow edges, at the run level, so the duplication described above still applies.
Desired Outcome
A Python developer can configure an ordered set of chat clients so that:
- A qualifying failure on one model call is retried on the next client for that same model call. The tool loop continues with the tool results already produced, and no tool runs twice.
- Streaming fails over only before the first update reaches the caller. After that, the error surfaces (the same rule MEAI uses).
- Cancellation never triggers failover.
- What counts as a qualifying failure (rate limit, server error or unavailable, timeout, optionally content filter) can be decided without importing provider SDK exception types.
- Telemetry records each attempt and which client finally served the call.
- Using clients that keep service-side conversation state (
conversation_id/previous_response_id) is either rejected or clearly documented, because that history cannot move between providers. .NET documents the same constraint for routing.
Alternatives Considered
auto_retry.py(tenacity + ChatMiddleware): retries the same model. It doesn't help with an outage or quota exhaustion, and it relies on provider-specific exception types (Python: [Bug]: auto_retry.py sample never retries a 429: the client raises ChatClientException, not RateLimitError #9904, Python: Fix the auto_retry sample so it retries 429 errors #10068).- Wrapping full clients: re-executes tools (see the repro in the Problem section).
- Composing
Raw*clients under a hand-builtFunctionInvocationLayer: possible, but undocumented. It also loses per-client middleware and telemetry, and every user has to rediscover the right layering. - Workflow conditional edges between agents: runs at the run level, so the same duplication applies, and it's heavy for a resilience concern.
Proposed Direction
Open to maintainer preference:
- Do failover at the per-model-call layer, below
FunctionInvocationLayer. One possible shape: aFailoverChatClient([client_a, client_b], should_failover=...)that owns a single function-invocation loop and calls only the inner, non-tool-looping path of each candidate. Another: a chat middleware like LangChain's, sinceChatMiddlewareLayeralready runs per model call. - Use semantics close to MEAI's
FailoverChatClient/OrderedFailoverChatClient(attempt reporting, failover only before output is committed), so .NET and Python behave the same. - A possibly separate, independently useful piece: give
ChatClientExceptionprovider-neutral classification, for examplestatus_code,is_transient/retry_after, or subclasses such as a rate-limit or service-unavailable exception, populated by the first-party clients. This would also simplify the retry guidance from Python: Add typical rate limiting handling for model clients #1362 without the framework having to own a retry policy. - Mark it experimental.
Scope and Non-Goals
- Not built-in retry or backoff for a single client. That stays user-owned, following the decision in Python: Add typical rate limiting handling for model clients #1362.
- Not semantic or content-based routing, and not manual mid-conversation route switching. Python parity with
RoutePersistingRoutingChatClient([Feature]: Allow switching from provider A to provider B without losing the chat history to the new provider #6865 / .NET: Add session-persisted chat client routing #7641) could be a follow-up. - No .NET changes. .NET is covered by MEAI.
Related: #1362, #6865, #7641, #9904, #10068, #9031, #10172, dotnet/extensions#7647, dotnet/extensions#7662
Language
Python
Contribution Intent
I am willing to implement this
Acknowledgements
- I searched existing issues and did not find a duplicate.
- I will wait for explicit maintainer agreement before starting implementation of a non-trivial change.
Metadata
Metadata
Labels
Type
Projects
- StatusShow more project fieldsNo status
Activity
[-][Feature]: Model failover across chat clients that never re-runs tools (parity with .NET FailoverChatClient)[/-][+]Python: [Feature]: Model failover across chat clients that never re-runs tools (parity with .NET FailoverChatClient)[/+]