Calling a flaky payment gateway: timeouts, retries and circuit breakers in .NET

Short answer: give every gateway call a timeout shorter than your user's patience, retry only operations that are safe to repeat (status inquiry, verify, refund with a reference), put a circuit breaker in front of the gateway so an outage fails fast, and treat a timeout on a charge as an unknown outcome that you resolve with an inquiry, never with a blind retry.

A real example: the double charge

A user taps "Pay". The server calls the gateway's charge endpoint. The gateway takes the money but its response is slow, the default HttpClient timeout is 100 seconds, the reverse proxy gives up at 30, and a well-meaning retry policy sends the charge again. The user is charged twice, and the app shows "payment failed".

Three mistakes stack up here: no deliberate timeout, a retry on a non-idempotent operation, and treating "I didn't get an answer" as "it failed".

Rules for payment calls

1. Timeouts are mandatory. Set a per-attempt timeout (2 to 5 seconds) and an overall budget (10 to 15 seconds).
2. Classify every endpoint. Inquiry and verify are safe to retry. Charge is not, unless you send an idempotency reference the gateway honours.
3. Retry with backoff and jitter, and only on transient failures: network errors, timeouts, 408, 429 and 5xx.
4. Break the circuit when most calls fail, so you stop hammering a gateway that is down and fail fast with a clear message.
5. Unknown is a state. A timed-out charge becomes PendingVerification, and a job asks the gateway what happened.

Runnable code

With the Microsoft.Extensions.Http.Resilience package (Polly v8 under the hood):

using Microsoft.Extensions.Http.Resilience;
using Polly;

builder.Services.AddHttpClient<GatewayClient>(c =>
    {
        c.BaseAddress = new Uri("https://gateway.example.com/");
        c.Timeout = Timeout.InfiniteTimeSpan; // the pipeline owns timeouts
    })
    .AddResilienceHandler("gateway", pipeline =>
    {
        pipeline.AddTimeout(TimeSpan.FromSeconds(12)); // whole operation

        var retry = new HttpRetryStrategyOptions
        {
            MaxRetryAttempts = 3,
            BackoffType = DelayBackoffType.Exponential,
            UseJitter = true,
            Delay = TimeSpan.FromMilliseconds(400),
        };
        retry.DisableForUnsafeHttpMethods(); // never retry POST/PUT/PATCH/DELETE
        pipeline.AddRetry(retry);

        pipeline.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
        {
            FailureRatio = 0.5,
            MinimumThroughput = 10,
            SamplingDuration = TimeSpan.FromSeconds(30),
            BreakDuration = TimeSpan.FromSeconds(20),
        });

        pipeline.AddTimeout(TimeSpan.FromSeconds(3)); // one attempt
    });

The default ShouldHandle of HttpRetryStrategyOptions already treats network errors, timeouts, 408, 429 and 5xx as transient. DisableForUnsafeHttpMethods means the charge (a POST) is attempted once, while inquiry (a GET) gets retries.

Handling the unknown outcome:

public async Task<PaymentResult> ChargeAsync(Payment payment, CancellationToken ct)
{
    try
    {
        var response = await gateway.ChargeAsync(payment.Reference, payment.AmountRials, ct);
        return payment.Apply(response);
    }
    catch (BrokenCircuitException)
    {
        return PaymentResult.Unavailable("The bank gateway is unavailable. Nothing was charged.");
    }
    catch (Exception ex) when (ex is TimeoutRejectedException or HttpRequestException)
    {
        payment.MarkPendingVerification();      // we don't know yet
        await db.SaveChangesAsync(ct);
        return PaymentResult.Pending("We are confirming your payment with the bank.");
    }
}

A background job then calls the gateway's inquiry endpoint for every PendingVerification payment and settles it as succeeded or failed. The same nightly reconciliation catches anything that still slips through.

Pros and cons

Pros

• Users wait a bounded, predictable time instead of 100 seconds.
• A gateway outage turns into fast, honest error messages instead of piled-up threads.
• No double charges from retries, because unsafe calls are never repeated.

Cons

• More states in your payment model (PendingVerification) and a job to resolve them.
• Badly tuned breakers can open on a short blip; watch the metrics and adjust.
• Retries multiply load on a struggling gateway, so keep the count low and always use jitter.

Integrating a bank gateway or a third-party payment API and want it to fail safely? See how I can help.