1Example run (not a real result)
2run_id
3What this run changed.
Comparing Azure Foundry Load Balancing
Azure API Management versus client-side failover
Explanation
Azure
APIM
AI Foundry
Load Testing
Performance
Is it better to load balance server or client side?
For engineers who already retry and fail over between AI model regions in client code, and wonder if a gateway is worth it. No Azure experience needed.
Notetl;dr
A gateway did not beat a well-built client. Nothing in the tests showed API Management (APIM) performing better. Success rates matched. APIM p95 answer time was worse, by about a quarter of a second in a clean test.
What mattered was a sin bin. When a region refuses a call,a circuit breaker makes it sit out for the next minute. Setups with a sin bin answered every request when one region ran short.
When every region is out of quota, nothing helps. Only more quota, or waiting for theresponse’s Retry-After header was better for success rate.
Regions differ in speed. From a UK laptop, UK South answered in about 300 ms, Sweden Central in 365 and West Europe in 480, and West Europe was the least steady.
What to build:
- Add a sin bin. In the client, a 60-second circuit breaker for a region that returns a 429.
- Use a priority list with the fastest region first.
Should you stand up APIM? Do it for operations, not speed: one set of credentials, one place for rules and monitoring. Those benefits were not tested. Goodput did not make the case for APIM.
The problem
AI services ration tokens. Each model deployment has a quota, counted per minute. When it runs out, the service refuses the call with HTTP status 429 (“Too Many Requests”).
The usual fix is to deploy the same model in several regions and fall back between them. Most teams write that logic into the client. The alternative is a gateway: a server between your callers and the AI models that applies rules to every call. Azure API Management (APIM) is Microsoft’s gateway service.
A gateway is one more service to pay for, run and debug. Is it worth it? This article tests that.
How it fits together
Grafana k6
Local load · subscription key
HTTPS
Azure API Management
Inbound policyAdmits the call and hands it to the pool. It can also cap how many calls one subscriber makes each minute.
Weighted backend poolSends each call to a region in proportion to that region's weight.
Circuit breakersAfter repeated failures (5xx, and optionally 429), takes that backend out of the pool for 60 seconds: the gateway's sin bin.
Retry on 429When a region refuses with a 429, asks the pool again, up to three attempts in all, then passes on the last refusal. This sits in the backend section, not inbound.
managed identity
Azure AI Foundry
UK South
West Europe
Sweden Central
Callers send requests to the gateway. The gateway picks a region. If that region is out of quota, it refuses with a 429. The gateway can then try another region, or take the refusing region out of rotation. A client-side setup does the same job without the middle box. The gateway holds the model credentials, so callers only need the gateway’s key.
Terms you need
| Term | Meaning |
|---|---|
| Quota | What a model deployment can handle each minute. Exceed it and calls are refused. There are two limits: tokens per minute (TPM, where a token is a chunk of text) and requests per minute. |
429 and Retry-After |
A 429 means “out of quota, try later”. The reply often carries a Retry-After header: how many seconds to wait. A 5xx error means the service itself failed. |
| Pool, weights, priority | The pool is the three regions behind one name. Weights set each region’s share: 90/5/5 sends 90% to UK South. Priority ranks the regions: a lower-ranked region gets traffic only when the higher ones are out. |
| Retry | Try a failed call again. The client can do it, or the gateway can. |
| Circuit breaker | The gateway’s sin bin. After a set number of failures, a region leaves the pool for 60 seconds. You choose which errors count. All callers share it. |
| Cool-down | The client’s sin bin: a timer that skips a region after it refuses a call. |
What was tested
Three Azure AI Foundry regions each got a small quota, so quota exhaustion could be reached easily: UK South, West Europe and Sweden Central. The K6 load tool sent requests through the gateway, or straight to the regions. One setting changed at a time.
There were two stages of experiments:
- Bursts. A quick look at what each setting does when every region runs out of quota.
- Gateway against client. The main test. Each gateway setup was paired with a client built to do the same job. Both got the same steady stream of requests and the same quota, in alternating five-minute runs.
Three situations were tested:
| Situation | What it models |
|---|---|
| Every region short | Demand beyond all your quota |
| Main region short | One region overloaded, the others have room |
| Nothing wrong | What the gateway costs when all is healthy |
A rule was fixed before the main test. The gateway counts as better if it gets more than 2 points more success, or has a p95 answer time more than 50 ms faster. In these experiments, the rule was never met.
The ceiling
Each deployment allowed 60 requests a minute (rpm). The test calls were tiny, so rpm limited rate before the token limit did. A gateway can move requests between regions but cannot raise that ceiling.
How results are measured
| Measure | Meaning |
|---|---|
| Success rate | Share of requests that got a valid answer |
| Time to answer | The median and p95 latencies in milliseconds |
| Throughput | Requests handled per second |
| Goodput | Answered requests per second (throughput × success rate) |
A gateway that refuses everything instantly has high throughput and zero goodput. Goodput is a better measure for meaningful work rate. The animation below visualises a goodput of 9 requests a second.
Sent0
Time0 s
Success rate-
Throughput-
Goodput-
What happened
Each dot below is one run. Pick a situation to see its runs and what they showed.
Bursts. Short, sharp tests. 200 requests were fired as fast as possible at three small quotas (10,000 tokens a minute each), so every region ran out within seconds. The question was whether any setting helps when demand beats all the quota at once. Gateway setups, direct calls and client retries all took part.
Steady load. The same situation with a realistic load. Requests arrived at a steady 6 a second for five minutes against bigger quotas (60,000 tokens a minute each). In practice each region answered about 72 requests a minute, so together the three could serve 3.6 of the 6 requests a second. That caps any setup at 60% answered. Each gateway setup was paired with a client built to do the same job. The question was how close to that cap each one could get.
Nothing helped. In 200-request bursts, nearly every setup answered about 17%. A circuit breaker on an even split made it worse (4.5%). Only a client that waited as long as Retry-After asked got nearly everything through (99.5%). That took about 30 seconds per request.
Under steady load every setup answered 55-60%, at or close to the 60% cap, so there was little left for any setup to win.
One region overloaded, the others with room to spare. UK South was sent 90% of the traffic, more than its quota allowed, and the other two regions 5% each. Requests arrived at a steady 2 a second (filled dots) or 3 a second (hollow dots), when the fallbacks were busy too, for five minutes. Each gateway setup was paired with a client built to do the same job. The question was which setups steer calls to the regions with room. Dots at 100% are nudged sideways so they do not hide each other.
Here the setup mattered. UK South got 90% of the traffic, more than it could serve. The other two regions sat nearly idle.
| Setup | Success |
|---|---|
| Nothing | 70% |
| Retry only, in the gateway | 77% |
| Retry only, in the client | 79% |
| Sin bin in the gateway (circuit breaker or priority list) | 100% |
| Sin bin in the client (cool-down) | 100% |
| Client tries UK South first, nothing remembered | 100% |
- Retry alone barely helped. Each retry picked a region again by weight, so nine in ten went straight back to the region that had just said no.
- A sin bin fixed it. The region sat out, and every call went where there was room.
- Trip on the first failure. It was safe at every rate tested, and it wasted the fewest calls.
- No sin bin costs calls. Trying UK South first took 1.4 calls per request. A sin bin took about 1.1.
- The gateway matched the client. Same success. The p95 was 880-1,040 ms through the gateway and 640-770 ms through the client.
Raising the load did not change this.
What a gateway costs when nothing is wrong. Requests arrived at a steady 1 a second for five minutes, split evenly across three regions with quota to spare, so nothing was ever refused. There was nothing to recover from, so any difference between the gateway and direct calls is the gateway’s own overhead. Hover over a dot for its answer times.
Both sides answered everything. The median answer took 372 ms through the gateway and 396 ms direct. The p95 was 710 ms against 593 ms. The gateway’s cost is in the slow tail. The next tab looks at that more closely.
A separate set of tests that measured speed alone. Quota was plentiful, each request got one attempt, and there was no retry, sin bin or cool-down. Three kinds of run were compared, shown as shapes. Through the gateway (circles) pinned the gateway to one region at a time. Direct, all regions together (diamonds) sent calls straight to the three regions in one run, alternating between them. Direct, one region at a time (squares) did the same one region per run, so the regions were never tested together. The question was how fast each region is from a UK laptop, and what the gateway adds.
| Region | Direct: median | Direct: p95 | Via gateway: median | Via gateway: p95 |
|---|---|---|---|---|
| UK South | 297 ms | 434 ms | 299 ms | 798 ms |
| Sweden Central | 365 ms | 675 ms | 396 ms | 929 ms |
| West Europe | 481 ms | 982 ms | 473 ms | 1,233 ms |
- The order held. UK South was fastest, Sweden Central about 65 ms behind, West Europe slowest. The same order appeared in all three repeats across the day, and in a one-region-at-a-time check.
- West Europe was the least steady. Its median ranged from 450 to 764 ms between runs. The others moved by about 10 ms.
- The gateway added nothing visible to the median for UK South and West Europe, and about 30 ms for Sweden Central.
- It did raise the p95. By 250-360 ms in every region. This time there was no retry, no breaker and no quota pressure, so those are not the cause. The extra hop is still the best guess. It is a guess.
All of this was measured from one laptop in the UK on one day. A caller elsewhere could see a different order.
What to build
- Add a sin bin. Whether in the client or the gateway, this is the one thing that made a difference. Retrying without it mostly sends calls back to the region that just refused.
- In a gateway, use a priority list for one main region and some fallbacks. Rank them by speed from where your callers are. From the UK that meant UK South, then Sweden Central, then West Europe. Add a circuit breaker that counts 429s and trips on the first failure. A breaker that counts only 5xx never fires on quota errors.
- Handle the 503. If every region is short, a breaker on 429 can empty the pool. Callers then get instant 503s.
- Mind the retry pause. In one test, a one-second pause pushed the p95 to 2.6 seconds. With no pause it was about 1 second (two runs).
- Use
Retry-Afterfor batch work only. Waiting out the minute suits a nightly job. It does not suit a chat. - Do not cap requests below real demand. The cap will refuse work that would have succeeded.
- Buy quota before you tune routing. When every region is short, nothing else helps.
Should you stand up APIM?
If success rate matters more than speed, start with client rules. Take a batch job such as transcription, where a few hundred milliseconds do not matter. The gateway’s one measured cost, a slower p95, stops mattering. But it never won on success either: a client with a sin bin matched the gateway’s success rate in every matched test. A gateway cannot create quota or fix retries on its own.
Client rules
- Pros: nothing new to run, pay for or debug. No extra hop or shared point of failure. The best success result in the tests, including waiting out
Retry-Afterwhen every region was short. Easy to change. - Cons: every caller has to implement it. Each caller learns about a failing region on its own, because the sin bin is per caller. Rules get copied per language or service and drift apart. There is no central view of refusals.
A gateway (APIM)
- Pros: one sin bin shared by every caller. Callers need no regional logic. One place for credentials, routing changes, logging and limits. The same behaviour for every team and language. None of these were tested.
- Cons: another service to run and pay for. A p95 250-360 ms higher. New ways to fail (see Before you go live). A shared dependency for every caller. The tier tested, Developer, has no SLA.
For a use-case where success matters more than speed
- A sin bin in the client: skip a region for 60 seconds after a 429.
- Regions ordered by speed from where you run.
- When every region is refusing, wait out
Retry-Afterand retry. It is slow, but latency is not the goal. - A clean way to handle “every region refusing”, so the job is queued instead of failed.
Before you go live
Checks for anyone running this in production.
| What goes wrong | What to do |
|---|---|
| APIM retry silently does nothing on requests with a body. The retry fails inside the gateway and never reaches the model. | Add buffer-request-body="true" to the forward-request inside the retry. Then confirm the model sees more than one call per refused request. |
| A retry block with no forward-request returns instant fake 200s. | Alert on answers that come back far faster than the model can respond (here, 14 ms against hundreds). Check the body holds a real answer. |
count means retries, not attempts. count="3" is four attempts. |
Match your client’s attempts to the gateway’s. |
| Policy changes take up to two minutes. Breaker state survives a redeploy. | Wait two minutes after a change before you trust it. |
| The request limit can bite before the token limit. Here it was 60 requests a minute alongside 60,000 tokens. | Read your limits first: az cognitiveservices account deployment show, then properties.rateLimits. |
| Turning a breaker off with an empty rules list is rejected. | Set the breaker to null in Terraform or ARM. |
Limits
- Tiny calls. Every request was a short chat prompt with a one-word answer, and the request limit bit first. Transcription sends larger payloads, often limited by tokens or audio length, and every retry sends the audio again. Check where your real ceiling is.
- No queue. A job queue with retry and backoff is probably the most robust design for batch work. It was not tested.
- Retry safety. A refusal (429) is safe to retry. For a 5xx, check that a repeated request cannot be billed or applied twice.
- Developer tier. One gateway unit, no SLA. Production tiers will differ. On multi-unit tiers each instance keeps its own breaker state.
- Gateway and client were not identical. The gateway paused one second between retries and the client none. The gateway shared its sin bin across all callers. The client kept one per simulated caller. The client ran from a laptop. The gateway sat in UK South.
- Few runs for some setups. Ten runs were discarded when the laptop slept. Several setups have two or three valid runs. The cards list every one.
- Region speed came from one laptop in the UK, over one day. West Europe moved a lot between runs. The one-region-at-a-time check has one run per region. For West Europe the gateway’s slow-tail cost overlaps with the spread between direct runs, so it is less clear there than in UK South and Sweden Central.
- The bursts ran with a broken retry rule. They show what weights and breakers do, not what a working retry does in a burst. The steady runs had working retry.
- The measured ceiling sat about 20% above the configured limit. The reason is unknown.
- Success means a valid answer. Answer quality was not judged.
Appendix: all the runs
Every run, as a card. Each tab holds runs that share a quota and load, so compare cards within a tab. Cards run from highest goodput to lowest, with any reference card first (the speed tab runs fastest to slowest). The first four tabs are the gateway-against-client tests: a coloured edge is the gateway and a grey edge is a client calling the regions directly. The fifth is the region speed tests, which measured answer time only. The last five are the 200-request bursts. A reference card is pinned on most tabs. Hover a numbered note to highlight that part of the card.
Test setup
| Gateway | APIM Developer tier, UK South |
| Regions | Azure AI Foundry in UK South, West Europe and Sweden Central |
| Model | gpt-4.1-mini, answers of about 5 tokens, one fixed prompt |
| Quota | 10,000 TPM per region for the bursts, 60,000 TPM for everything else |
| Load tool | Grafana k6 on a Mac laptop |
| Bursts | 200 requests from 20 simulated callers, sent as fast as possible |
| Steady runs | 1, 2, 3 or 6 requests a second for five minutes |
| Gaps | 60 seconds between runs, 2 minutes after a policy change |
| Success | HTTP 200 with a valid answer in the body |
| Discarded | Steady runs longer than 330 seconds (the laptop slept) or with dropped requests |
| Speed tests | Rebuilt regions and gateway, 60,000 TPM per region, one request every 2 seconds per region for about 11 minutes, first 20 per region ignored |
The Terraform, k6 scripts and results are in r-leyshon/azure-load-balance-perf.
Run cards
Primary region short
Fallbacks busy
Every region short (steady)
Nothing wrong
Region speed
No retry
Direct (no gateway)
Client retry
Breaker and weights
Quota to spare
How to read a card
- 1Plain-language name. A grey tag on some cards marks the reference pinned for comparison. The coloured edge matches the tab; on the matched-pair tabs a grey edge means a client with no gateway.
- 2The id used in the repo and on the chart.
- 3What this run changed.
- 4Success, throughput and goodput. Bar length is this run against the best run in the tab, coloured red (low) through yellow to green (best). On the region speed tab the bars are median and p95 answer time instead, and a longer bar means a faster answer.
- 5The plain numbers behind the bars: completions out of requests, run length and quota. Gateway-against-client cards list each run's success rate and answer times instead.
Steady 2 requests a second for five minutes at 60,000 TPM per region, with 90% of traffic aimed at UK South, which can only serve about two thirds of that. Cards with a coloured edge go through the gateway; grey-edged cards are a client calling the regions directly. Where a setup was repeated, the card shows the middle result of its valid runs.
Gateway: 90/5/5, nothing elseReference
car_90_5_5, car_90_5_5_r2, car_90_5_5_r3
No breaker and no retry. UK South is offered about 108 requests a minute against a ceiling near 72, and the other two regions are nearly idle.
Gateway: priority list
p6a_apim_priority (r1, r2, r5)
UK South is first choice and the others are used only when it is out of the pool. Breaker trips after one failure.
Client: 90/5/5, retry, 60 s cool-down
p5a_direct_weighted_mem (r1, r2, r3, r4, r5), p6a_direct_weighted_mem (r1, r3, r4, r5)
As above, but after a refusal that caller skips that region for 60 seconds.
Gateway: retry and breaker (trips after 1 failure)
p6a_apim_retry_cb_trip1 (r1, r2, r3, r4, r5)
The same, but UK South leaves the pool after its first refusal.
Client: UK South first, 60 s cool-down
p5a_direct_seq_mem (r1, r2, r3, r4, r5), p6a_direct_seq_mem (r1, r4, r5)
The client tries UK South, then the next region, and skips a region for 60 seconds after a refusal.
Client: UK South first, no memory
p5a_direct_seq (r4, r5)
The client tries UK South, then the next region, with nothing remembered between requests.
Gateway: retry and breaker (trips after 3 failures)
p6a_apim_retry_cb (r1, r3, r4, r5)
UK South leaves the pool for 60 seconds after three refusals, so retries can only reach the other two regions.
Client: 90/5/5, retry, no memory
p5a_direct_weighted (r1, r2, r3, r4, r5), p6a_direct_weighted (r1, r3)
No gateway. The client picks a region by weight and retries up to twice more, picking again each time.
Gateway: retry only, no waiting
p6a_apim_retry_no_cb_int0 (r2, r3)
The same, with no pause between attempts.
Gateway: retry only, no breaker
p6a_apim_retry_no_cb (r1, r2, r3, r4, r5)
The gateway retries a refused call up to twice more, waiting one second each time. Nothing is remembered, so each retry picks a region by weight again.
Gateway: breaker on server errors only
car_cb5xx_90_5_5
The breaker ignores 429s, so running out of quota never removes UK South. No retry.
Steady 3 requests a second for five minutes at 60,000 TPM per region, with 90% aimed at UK South. The two fallback regions must take about 108 requests a minute between them, three quarters of what they can serve. Grey-edged cards are a client calling the regions directly.
Gateway: retry and breaker (trips after 3 failures)
p6b_apim_retry_cb (r1, r2, r3)
UK South leaves the pool for 60 seconds after three refusals.
Gateway: retry and breaker (trips after 1 failure)
p6b_apim_retry_cb_trip1 (r1, r2, r3)
UK South leaves the pool after its first refusal.
Client: UK South first, 60 s cool-down
p5b_direct_seq_mem (r1, r2, r3)
As above, but a region that refused is skipped for 60 seconds.
Client: 90/5/5, retry, 60 s cool-down
p5b_direct_weighted_mem (r1, r2, r3)
The client picks a region by weight, retries, and skips a region for 60 seconds after a refusal.
Gateway: priority list
p6b_apim_priority (r1, r3)
UK South first, the others only when it is out of the pool. Breaker trips after one failure.
Client: UK South first, no memory
p5b_direct_seq (r1, r2, r3)
No gateway. Every request tries UK South first and moves on after a refusal.
Steady 6 requests a second for five minutes at 60,000 TPM per region, about 1.7 times what the three regions can serve (3.6 a second). The best any setup can do is about 60%. Grey-edged cards are a client calling the regions directly.
Client: even split, retry, 60 s cool-down
p5d_direct_weighted_mem_eq (r1, r2, r3)
The client picks a region at random, retries, and skips a region for 60 seconds after a refusal.
Client: UK South first, 60 s cool-down
p5d_direct_seq_mem (r1, r2, r3)
No gateway. The client tries UK South, then the next region, and skips a region for 60 seconds after a refusal.
Gateway: priority list
p6d_apim_priority (r1, r2, r3)
UK South first, the others only when it is out of the pool. Breaker trips after one failure.
Gateway: retry and breaker (trips after 1 failure), even split
p6d_apim_retry_cb_trip1_eq (r2, r3)
Equal weights, retry, and a breaker that trips after one failure.
Steady 1 request a second for five minutes at 60,000 TPM per region, an even split and no failures. This measures what the gateway costs when nothing is wrong. The grey-edged card is the reference.
Client: even splitReference
p5c_direct_equal (r1, r2, r3)
No gateway and no failures: the client splits calls evenly across the three regions.
Gateway: even split
p5c_apim_baseline (r1, r2, r3)
The same traffic through the gateway with equal weights and no breaker. Nothing is refused, so no recovery is needed.
Speed only, with no quota pressure: one request every 2 seconds per region for about 11 minutes, no retry, breaker or cool-down, and the first 20 answers per region ignored. A longer bar means a faster answer. A coloured edge is the gateway, pinned to one region; a grey edge is a direct call. Every valid run had no refusals or errors. Two runs that dropped a request were repeated, and Azure Monitor confirmed that every gateway call reached its pinned region.
Direct, one region at a time: UK South
rlB_uksouth (r1)
No gateway. Each run called one region only, so the regions were never measured together.
Direct, all regions together: UK South
rlA (r1), rlA_r2_x2, rlA_r3_x2
No gateway. One run called all three regions in turn, so the regions were measured side by side, at 0.5 requests a second each.
Gateway: UK South
rlC_uksouth (r1, r2, r3)
Through the gateway, pinned to this one region. No retry and no breaker.
Direct, one region at a time: Sweden Central
rlB_swedencentral (r1)
No gateway. Each run called one region only, so the regions were never measured together.
Direct, all regions together: Sweden Central
rlA (r1), rlA_r2_x2, rlA_r3_x2
No gateway. One run called all three regions in turn, so the regions were measured side by side, at 0.5 requests a second each.
Gateway: Sweden Central
rlC_swedencentral (r1, r2, r3)
Through the gateway, pinned to this one region. No retry and no breaker.
Gateway: West Europe
rlC_westeurope (r1, r2, r3)
Through the gateway, pinned to this one region. No retry and no breaker.
Direct, all regions together: West Europe
rlA (r1), rlA_r2_x2, rlA_r3_x2
No gateway. One run called all three regions in turn, so the regions were measured side by side, at 0.5 requests a second each.
Direct, one region at a time: West Europe
rlB_westeurope (r1)
No gateway. Each run called one region only, so the regions were never measured together.
200-request bursts, 10,000 TPM per region. With no retry, nothing gets past about 17%. A breaker or a request cap on its own makes it worse, and the busiest run completes nothing.
Control: even split, no policiesReference, 10,000 TPM
baseline
Equal weights across the three regions. No circuit breaker, no retry.
Skewed 90/5/5 and breaker, no retry
asymmetric_cb
90% of traffic lands on UK South, which runs out first and can trip the breaker. No retry.
Breaker on server errors only, no retry
cb_5xx_only
The breaker ignores 429s and only trips on 5xx, so running out of quota never removes a region. No retry.
Weighted 60/25/15, nothing else
weighted_primary
UK South takes 60% of the traffic, West Europe 25%, Sweden Central 15%. Still no breaker and no retry.
Circuit breaker only
circuit_breaker
Equal weights. A region leaves the pool after three failures inside 60 seconds, and a 429 counts as a failure. Nothing retries onto whoever is left.
Gateway cap of 10 requests a minuteFast refusals
gateway_throttle
The gateway refuses calls beyond 10 a minute, before Foundry sees them.
Weights, breaker and a 500-a-minute capFast refusals
full_resilience
Weights, a circuit breaker and a 500 requests-a-minute gateway cap, with no retry. Every call was refused in about 12 ms, too fast to have reached the model.
200-request bursts, 10,000 TPM per region. Calling Foundry directly gives the same result as the gateway with no policies. Walking the regions in order only burns calls.
Control: even split, no policiesReference, 10,000 TPM
baseline
Equal weights across the three regions. No circuit breaker, no retry.
Direct, 60/25/15, no fallback
direct_weighted
No gateway. The client picks a region at random (60/25/15) and does not fall back when that region is exhausted.
Direct, walk the regions in order
direct_sequential
No gateway. The client tries UK South, then West Europe, then Sweden Central, moving on only after a failure. It made 567 HTTP calls to land 33 successes.
Direct to Foundry, even split
direct_equal
No gateway. The client splits calls evenly across the three regions and does not fall back. The matched pair for the control.
200-request bursts, 10,000 TPM per region. Short pauses do nothing useful. Only sleeping for the full Retry-After works, and it costs about 30 seconds a call.
Control: even split, no policiesReference, 10,000 TPM
baseline
Equal weights across the three regions. No circuit breaker, no retry.
Client retry, immediately
retry_immediate
The client retries a 429 straight away, up to three times, against an equal-weight pool with no breaker.
Client retry, exponential backoff
retry_backoff
Client retry with exponential backoff: 500 ms, then 1 s, then 2 s.
Client retry, backoff with jitter
retry_backoff_jitter
Backoff plus jitter, so the virtual users do not all wake up on the same tick.
Client retry, fixed 1 s pause
retry_fixed_1s
The same client retry, with a fixed one second pause between attempts.
Breaker plus client Retry-After
optimal_asym_client
Skewed 90/5/5 weights and a breaker on 429 and 5xx, with the client sleeping for Retry-After.
Client retry, wait for Retry-After (repeat)
optimal_equal_client
A repeat of the same setup, run again under another name.
Client retry, wait for Retry-After
retry_after_header
The client sleeps for the Retry-After value Azure returns, usually the rest of the quota window.
200-request bursts, 10,000 TPM per region. A gateway retry rule was set in these runs but never worked (a bug found later), so read them as breaker and weights only. Nearly all land at the same 16-18%, and two identical runs of one setup differ widely.
Control: even split, no policiesReference, 10,000 TPM
baseline
Equal weights across the three regions. No circuit breaker, no retry.
Breaker trips on first failure, 90/5/5Fast refusals
trip_count_1
90/5/5 weights, with the breaker tripping on the first failure instead of the third.
Breaker, 70/15/15Fast refusals
weight_70_15_15
A gentler skew than 90/5/5, with the breaker on 429 and 5xx.
Breaker, 80/10/10
weight_80_10_10
An in-between skew: the spare regions get 10% each, with the breaker on 429 and 5xx.
Breaker on 429 and 5xx, 90/5/5
apim_retry_asym
Skewed weights and a breaker on 429 and 5xx. UK South is expected to trip first.
Skewed 90/5/5, no breaker
asym_retry_no_cb
Skewed weights with the breaker off.
Even split, no breaker
apim_retry_equal
Equal weights and no breaker. A gateway retry rule was set but never worked, so this is the control again.
Even split, no breaker (repeat)
optimal_equal_apim
A repeat of the same configuration, run again under another name.
Breaker on 5xx only, 90/5/5
asym_retry_cb_5xx
Skewed weights, breaker on 5xx only. Running out of quota never trips the breaker.
Breaker on 429 and 5xx, 90/5/5 (repeat)
optimal_asym_apim
A repeat of the identical configuration, run again under another name. The two results differ.
Breaker on 429 and 5xx, even split (repeat)
apim_retry_cb
Equal weights and a breaker on 429 and 5xx: the same as the breaker-only run, run again. The gateway retry rule set here never worked.
200-request bursts, 60,000 TPM per region: six times the quota, so 200 requests fit inside one minute's allowance. A gateway retry rule was set but never worked, so these are even-split and skewed controls. The control from the 10,000 TPM runs is pinned for contrast.
Control: even split, no policiesReference, 10,000 TPM
baseline
Equal weights across the three regions. No circuit breaker, no retry.
Even split, no breaker
apim_retry_equal_r2
The even-split control with six times the quota. 200 requests fit inside one minute's quota. A gateway retry rule was set but never worked.
Skewed 90/5/5, no breaker
asym_retry_no_cb_r2
Skewed weights with the breaker off, at 60,000 TPM.
Even split, no breaker (repeat)
optimal_equal_apim_r2
The same configuration again, run on its own later.
Further reading
Microsoft documents these pieces in backends, set-backend-service, retry, rate-limit-by-key, AI gateway capabilities, how policies work and APIM key concepts. The setup and the result files are in github.com/r-leyshon/azure-load-balance-perf.
NoteAcknowledgements
Thanks to Harry, Francis and Adela for their thoughts and feedback.
fin!