Microsoft.Extensions.AI Failover: Streaming Boundaries

Microsoft.Extensions.AI 10.9.0 gives .NET applications first-party routing and failover primitives, but the useful production guarantee is narrower than “try another model when anything goes wrong.” A failed non-streaming call can move to another client. A streaming call can move only before the first update reaches the caller. After that point, output is committed and the failure is terminal. A reliable design starts from that boundary, then adds bounded attempts, route ownership, telemetry, and tests that prove exactly where recovery stops.

Choose the guarantee before the client

Begin with the behavior your caller is allowed to observe. For a normal response, the application either receives one complete answer or an exception. Retrying a failed attempt is invisible because no answer escaped. For streaming, the first ChatResponseUpdate changes the contract: the UI may already have rendered text, a consumer may have persisted it, or a tool coordinator may have acted on it.

That gives the API two honest promises. Before output commitment, failover may be transparent. After commitment, the stream may end with a partial answer and an error, but it must not silently restart on a second model. Restarting could duplicate tokens, contradict visible text, repeat a tool request, and create two billable generations that look like one operation.

Make the partial-stream state explicit in the product contract. Give each logical response an operation ID, mark incomplete output, and let the user start a new attempt. Do not append a second provider’s answer to the first provider’s partial stream.

Understand what failover actually retries

Version 10.9.0 ships RoutingChatClient, SemanticRoutingChatClient, FailoverChatClient, and OrderedFailoverChatClient. They all implement IChatClient, so they fit into the normal Microsoft.Extensions.AI pipeline. RoutingChatClient chooses one client before invocation. FailoverChatClient adds another selection after an uncanceled failure that occurred before streaming output was exposed. OrderedFailoverChatClient supplies the simplest concrete policy: walk a fixed list.

The base failover loop does not infer business meaning from an exception. A pre-output provider exception can advance the chain; cancellation does not. Once streaming output has been committed, a later exception is terminal. Selection failures and exceptions thrown by your own routing-update hook also end the request, so custom routing code must release request state before it throws.

Non-streaming success: return the response and stop.
Non-streaming failure: select again unless canceled or the attempt limit is reached.
Streaming failure before the first update: another selection is possible.
Streaming failure after the first update: propagate the failure; no failover follows.

Caller cancellation: honor cancellation rather than trying another provider.

Build a bounded ordered failover chain

Start with an ordered chain because its worst case is visible in configuration. If the clients come from dependency injection and remain shared elsewhere, keep ownership outside the router with leaveOpen: true. Otherwise, the default router ownership disposes its inner clients when it is disposed.

#pragma warning disable MEAI001

IChatClient failover = new OrderedFailoverChatClient(
    [primaryClient, regionalBackupClient, lastResortClient],
    leaveOpen: true)
{
    MaximumAttemptsPerRequest = 2
};

#pragma warning restore MEAI001

The limit counts invocations, not retries, so a value of two means the primary plus one backup. This is both a latency budget and a cost budget. If an upstream timeout is 30 seconds, allowing three sequential 15-second attempts cannot meet it. Give the AI operation its own cancellation deadline, pass that token through every call, and reserve time for response serialization and the network hop back to the caller.

A fixed list also exposes a policy question: should every error reach the backup? Authentication failures, malformed requests, content-policy rejections, and unsupported capabilities usually need correction, not another provider. The ready-made ordered client retries pre-output failures generically. If your application must distinguish transient from terminal errors, put that classification in an adapter around the provider or implement a custom FailoverChatClient policy that stops after cleaning its request state. Do not assume the built-in list is a circuit breaker.

Keep request policy separate from route policy

Each routed request gets a RoutingContext with the messages and a clone of ChatOptions. Changes to context.ChatOptions survive into a later attempt. Route-specific settings belong on the selected client, usually through a configured wrapper that clones the request options and applies its own values.

IChatClient lowEffort = baseClient.AsBuilder()
    .ConfigureOptions(options =>
        options.Reasoning = new ReasoningOptions
        {
            Effort = ReasoningEffort.Low
        })
    .Build();

IChatClient highEffort = baseClient.AsBuilder()
    .ConfigureOptions(options =>
        options.Reasoning = new ReasoningOptions
        {
            Effort = ReasoningEffort.High
        })
    .Build();

IChatClient router = RoutingChatClient.Create((context, cancellationToken) =>
    new(isComplexRequest(context) ? highEffort : lowEffort));

Keep correlation properties, the application session ID, safety requirements, and the response contract at request level. Put deployment names, reasoning effort, token ceilings, and provider-only features on route wrappers. Before combining unrelated providers, validate that every route can satisfy required capabilities such as tools, vision, structured output, context length, data residency, and content policy.

For a multi-turn conversation, store an application-owned route name against an application-owned session ID. A provider conversation ID or opaque reasoning artifact may not transfer to another provider, and switching routes on every turn can discard prompt-cache value. Sticky selection is a state-management decision, not a property the router can safely guess.

Place the router deliberately around tool calling

A function-invocation client can call the model several times for one user request. If the router wraps the whole function-invocation layer, selection occurs once and the tool loop stays on one route. If the function-invocation layer wraps the router, each model turn can be selected independently. The second design may save money, but it also asks another route to continue from tool results and provider-specific state it did not create.

Use one route for the entire tool loop unless you have verified that schemas, tool-call identifiers, structured responses, reasoning artifacts, and safety behavior are compatible across routes. Failover should never repeat a side-effecting tool merely because a model response failed. Give every tool operation an idempotency key and persist completion independently from the chat stream.

Observe every attempt without logging prompts

OnRoutingUpdateAsync receives a FailoverChatClientAttempt after every invocation, including success, failure, and abandoned streaming enumeration. The record exposes the client, duration, exception, completion state, output-commitment state, and time to first streaming update. Those fields are enough for route-level service indicators without storing prompts or generated content.

#pragma warning disable MEAI001

sealed class ObservedOrderedFailover : FailoverChatClient
{
    private readonly (string Name, IChatClient Client)[] _routes;
    private readonly ConcurrentDictionary<RoutingContext, int> _next = new();
    private readonly ILogger<ObservedOrderedFailover> _logger;

    public ObservedOrderedFailover(
        IEnumerable<(string Name, IChatClient Client)> routes,
        ILogger<ObservedOrderedFailover> logger)
    {
        _routes = routes.ToArray();
        if (_routes.Length == 0) throw new ArgumentException("At least one route is required.");

        _logger = logger;
        MaximumAttemptsPerRequest = _routes.Length;
    }

    protected override ValueTask<IChatClient> SelectClientAsync(
        RoutingContext context,
        CancellationToken cancellationToken)
    {
        int index = _next.TryGetValue(context, out int value) ? value : 0;
        return new(_routes[index].Client);
    }

    protected override ValueTask OnRoutingUpdateAsync(
        RoutingContext context,
        FailoverChatClientAttempt attempt,
        bool isTerminal,
        CancellationToken cancellationToken)
    {
        int index = _next.TryGetValue(context, out int value) ? value : 0;
        string outcome = attempt.ResponseCompleted ? "completed" :
            attempt.OutputCommitted ? "failed-after-output" : "failed-before-output";

        _logger.LogInformation(
            "AI route {Route} ended as {Outcome} in {DurationMs} ms; terminal={Terminal}; error={ErrorType}",
            _routes[index].Name,
            outcome,
            attempt.Duration.TotalMilliseconds,
            isTerminal,
            attempt.Exception?.GetType().Name);

        if (isTerminal) _next.TryRemove(context, out _);
        else _next[context] = index + 1;

        return default;
    }
}

#pragma warning restore MEAI001

This skeleton deliberately leaves client lifetime with the caller. Keep the hook fast, bounded, and unable to throw: an exception from a nonterminal update stops routing, and there is no later callback to clean state. Export route names, attempt counts, duration, time to first update, completion, commitment, and sanitized exception categories. Keep message bodies, model output, credentials, and tenant data out of routine telemetry.

Prove the five failure boundaries

A happy-path test proves almost nothing about failover. Put each provider behind a deterministic fake IChatClient or a controlled HTTP handler, then run the same acceptance suite for non-streaming and streaming calls.

Primary succeeds: one invocation, no backup call, one completed terminal update.

Primary fails before output: backup is invoked once and the caller receives only the backup response.
Primary streams one update and fails: backup is not invoked; the caller sees the partial update followed by the original failure.
Caller cancels: no reselection occurs and cancellation reaches the caller.

Attempt cap is reached: later routes are untouched and the final attempted failure is rethrown.

dotnet test --configuration Release \
  --filter "FullyQualifiedName~AiRoutingFailoverTests"

# Acceptance evidence to assert in the test sink:
# ai.route.attempts{case=pre_output_failure} = 2
# ai.route.backup_calls{case=post_output_failure} = 0
# ai.route.backup_calls{case=cancellation} = 0
# ai.route.output_committed{case=post_output_failure} = true

Add a deadline test in which the first attempt consumes most of the request budget, and a disposal test when the router owns clients. If tools are enabled, inject a failure after a side effect and prove that the tool’s idempotency record prevents duplication. These tests turn “we have a backup” into an observable contract.

Roll out an experimental API without hiding risk

The routing types in 10.9.0 are marked experimental with diagnostic ID MEAI001. Pin the package version, contain the types behind one application-owned registration or adapter, and suppress the diagnostic only at that boundary. Record the reason and the version being evaluated; do not disable the diagnostic across the whole solution.

Start with one non-critical workload, the primary plus one compatible backup, and a small traffic slice. Watch success rate, pre-output failover rate, post-output failure rate, latency percentiles, time to first update, cancellation, token usage, and cost per successful logical response. A higher completion rate can still be a bad rollout if retries double cost or push p95 latency beyond the product deadline.

Routing is not hedging, ensemble voting, or quality-based cascading. The built-in router chooses before invocation, and failover reacts to failure rather than a low-quality successful response. Treat those as separate architectures with separate cost and consistency risks.

Production decision checklist

The caller contract distinguishes pre-output recovery from a post-output terminal failure.

Every route satisfies required tools, formats, safety, context, and residency constraints.
The attempt limit fits both the end-to-end deadline and the cost budget.
Client ownership and disposal are explicit.
Multi-turn affinity uses an application session key, not a provider conversation ID.

Tool calls are idempotent and are not silently repeated after partial progress.
Telemetry records attempts and commitment without prompt or response bodies.
Failure injection proves success, pre-output retry, post-output stop, cancellation, and attempt limits.

The experimental API is pinned, isolated, and can be disabled without rewriting callers.

The backup client is the smallest part of the design. The real production feature is a tested boundary that knows when another attempt is still safe—and when the only honest result is to stop.