Skip to content

Python: [Feature]: Model failover across chat clients that never re-runs tools (parity with .NET FailoverChatClient) #10332

Description

Problem

Python has no supported way to fail over from one model or provider to another when a call fails. Common triggers are a 429, a 5xx or "overloaded" response, a regional outage, or a capacity limit on an Azure OpenAI PTU deployment. .NET already has this: Microsoft.Extensions.AI 10.9+ ships FailoverChatClient and OrderedFailoverChatClient (dotnet/extensions#7662), and the .NET Agent Framework builds on them (#7641, sample Agent_Step24_MultiModelRouting). Python has nothing comparable.

Writing it yourself is not safe in Python. Every shipped client mixes FunctionInvocationLayer into the client class:

class OpenAIChatClient(FunctionInvocationLayer, ChatMiddlewareLayer, ChatTelemetryLayer, RawOpenAIChatClient): ...

So the obvious wrapper ("try client A, if it raises try client B") fails over the whole tool loop, not the single model call that failed. If A has already executed tools and then fails on its next model call, B starts again from the original messages and executes the same tools a second time.

Minimal repro against main (scripted clients, no network):

SENT = []

@tool(approval_mode="never_require")
def send_email(to: str) -> str:
    SENT.append(to); return f"sent to {to}"

class ScriptedClient(FunctionInvocationLayer, BaseChatClient):
    # model call 1 -> asks for send_email; model call 2 -> answers
    # primary is configured to raise ChatClientException("429") on model call 2
    ...

class NaiveFallbackClient:
    async def get_response(self, messages, **kw):
        for c in self.clients:
            try: return await c.get_response(messages, **kw)
            except ChatClientException: continue

await NaiveFallbackClient(primary, fallback).get_response(msgs, options={"tools": [send_email]})

Output:

  failover: primary: 429 Too Many Requests
final: done via fallback
send_email executions: 2 ['cfo@contoso.com', 'cfo@contoso.com']

The caller sees a successful response, but the side effect ran twice. This is the same class of problem as #9031 and #10172, but here the user's own resilience code causes it.

A second gap: there is no provider-neutral way to decide whether an error should trigger failover. Clients raise a plain ChatClientException with no status code and no retryable flag. Callers have to unwrap the provider SDK exception (openai.RateLimitError, Anthropic status_code, and so on) per provider. #9904 showed that even the shipped auto_retry.py sample got this wrong.

Use Case and Evidence

  • Azure OpenAI capacity and quota: overflow from a PTU deployment to pay-as-you-go, or to a second region, when the first returns 429/503. This is standard production guidance for Azure OpenAI customers, and agents with tools are exactly the workloads that hit these limits.
  • Provider outages: overflow from OpenAI to Anthropic or Bedrock (or the reverse) during an incident, without stopping a long-running harness or loop agent.
  • Cost tiers: try a small or local model first and fall back to a larger hosted model on failure.
  • Peer frameworks treat this as basic infrastructure: Pydantic AI FallbackModel (with fallback_on), LangChain ModelFallbackMiddleware (which runs per model call inside the agent loop), LiteLLM Router fallbacks, and MEAI FailoverChatClient in .NET.
  • Existing Python MAF content (for example, community posts on an "SLM-default, LLM-fallback" pattern) builds fallback by hand with workflow edges, at the run level, so the duplication described above still applies.

Desired Outcome

A Python developer can configure an ordered set of chat clients so that:

  1. A qualifying failure on one model call is retried on the next client for that same model call. The tool loop continues with the tool results already produced, and no tool runs twice.
  2. Streaming fails over only before the first update reaches the caller. After that, the error surfaces (the same rule MEAI uses).
  3. Cancellation never triggers failover.
  4. What counts as a qualifying failure (rate limit, server error or unavailable, timeout, optionally content filter) can be decided without importing provider SDK exception types.
  5. Telemetry records each attempt and which client finally served the call.
  6. Using clients that keep service-side conversation state (conversation_id / previous_response_id) is either rejected or clearly documented, because that history cannot move between providers. .NET documents the same constraint for routing.

Alternatives Considered

Proposed Direction

Open to maintainer preference:

  • Do failover at the per-model-call layer, below FunctionInvocationLayer. One possible shape: a FailoverChatClient([client_a, client_b], should_failover=...) that owns a single function-invocation loop and calls only the inner, non-tool-looping path of each candidate. Another: a chat middleware like LangChain's, since ChatMiddlewareLayer already runs per model call.
  • Use semantics close to MEAI's FailoverChatClient / OrderedFailoverChatClient (attempt reporting, failover only before output is committed), so .NET and Python behave the same.
  • A possibly separate, independently useful piece: give ChatClientException provider-neutral classification, for example status_code, is_transient/retry_after, or subclasses such as a rate-limit or service-unavailable exception, populated by the first-party clients. This would also simplify the retry guidance from Python: Add typical rate limiting handling for model clients #1362 without the framework having to own a retry policy.
  • Mark it experimental.

Scope and Non-Goals

Related: #1362, #6865, #7641, #9904, #10068, #9031, #10172, dotnet/extensions#7647, dotnet/extensions#7662

Language

Python

Contribution Intent

I am willing to implement this

Acknowledgements

  • I searched existing issues and did not find a duplicate.
  • I will wait for explicit maintainer agreement before starting implementation of a non-trivial change.

Activity

  1. added
    pythonUsage: [Issues, PRs], Target: Python
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Nov 13, 2026
  2. changed the title [-][Feature]: Model failover across chat clients that never re-runs tools (parity with .NET FailoverChatClient)[/-] [+]Python: [Feature]: Model failover across chat clients that never re-runs tools (parity with .NET FailoverChatClient)[/+] on Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

agentsUsage: [Issues, PRs], Target: Single agentpythonUsage: [Issues, PRs], Target: PythontriageUsage: [Issues], Target: All issues that still need to be triaged

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions