Case UW-006. The underwriting agent asks for get_roof_assessment on SUB-1006, and the roof-data provider does not answer. Not an error, not "no record": nothing. Somewhere a connection is open, and on the other end a service is slow, overloaded, or gone.
Consider what the application could do. It could wait. Then the run waits, the underwriter who started it waits, and every other run that asks the same provider joins the queue, each holding a connection. It could retry at once, as fast as it can. Then a provider that was struggling receives two, four, eight times its normal traffic from the clients that most need it to recover. Or it could give up on the first silence and tell the model the roof is unknown, which is wrong whenever the silence would have lasted half a second.
None of those is a design. A design says how long one attempt may take, how long the whole call may take, which failures are worth another try, how long to wait between tries, what to do when the provider asks you to slow down, when to stop calling a provider that is clearly down, and what the model is told at the end. The model proposes a call. Your code decides whether it runs, and on this page, how long it may run and how often.
A timeout is a decision about when to stop waiting#
A timeout is the longest the application will wait for one attempt before treating it as failed. In the example, each tool has one: 2000 ms for the roof-data provider, 1000 ms for the guidance index, the decision history and the recommendation store, 500 ms for the rules engine. When get_roof_assessment gets no answer in 2000 ms, the adapter stops waiting and records a timeout.
One timeout per attempt is not enough once retries exist, because three attempts of 2000 ms plus the waits between them can take longer than anyone is willing to wait. So the adapter also holds a deadline: a point in time, fixed when the call starts, after which nothing more is attempted. In the example it is 5000 ms per call. No retry starts if its wait would reach the deadline, and a late attempt's timeout is cut to whatever time is left.
Before AI. Every HTTP client has a connect timeout and a read timeout, and every production incident review eventually finds one that was left at "infinite". gRPC goes further: its deadlines guide, read on 2026-11-18, defines a deadline as a point in time past which a client is unwilling to wait, notes that gRPC sets none by default, and says some implementations (Java and Go among them) propagate it to downstream calls as the time remaining. The adapter's per-attempt timeout is the read timeout; its per-call deadline is the gRPC deadline.
How long should a timeout be? Amazon's Builders' Library article "Timeouts, retries, and backoff with jitter", read on 2026-11-18, describes its practice: choose an acceptable rate of false timeouts, such as 0.1%, and set the timeout at the provider's matching latency percentile, such as p99.9. A timeout set by guess is either too short (healthy calls fail) or too long (a dead provider holds every caller hostage). The example's 2000 ms is illustrative; a real value comes from the provider's measured latency.
Retry only what may pass#
A retry is a second attempt at the same call after the first failed. It helps only when the failure was temporary. The adapter decides which failures those are with two sets, quoted from executor.py:
RETRYABLE = {"timeout", "rate_limited", "provider_unavailable"}
BREAKER_FAILURES = {"timeout", "provider_unavailable"} # rate_limited means busy, not broken; not_found is an answer
Three codes are retryable: the provider was slow (timeout), asked the caller to wait (rate_limited), or answered with a server error (provider_unavailable). Everything else is final. A retry of invalid_arguments sends the same wrong value. A retry of not_authorized asks the same caller again. A retry of not_found asks a provider that already answered "no record", as in UW-003, and gets the same answer. The second set matters later, for the circuit breaker: a rate limit means "busy", not "broken", and does not count against the provider.
There is one more rule, and it comes from Safety: a write is retried only with an idempotency key. A timeout on record_recommendation may mean the row was stored and the reply was lost, which is this site's own case UW-T04. The key, explained in full on that page, is what makes the retry return the stored row instead of writing a second one. The Amazon article states the general rule plainly: APIs with side effects are not safe to retry unless they provide idempotency.
The example retries once by default (max_retries 1, so two attempts in total). That number is a policy, set per adapter, and the labs let you change it.
Backoff and jitter: wait, and not all at once#
If an attempt fails because the provider is overloaded, retrying immediately adds load at the worst moment. So the adapter waits before each retry, and waits longer each time. That growing wait is called backoff; the usual form is exponential: 200 ms after the first failure, 400 ms after the second, 800 ms after the third.
Backoff alone has a flaw. A hundred runs that failed at the same moment, because the provider stalled for all of them, back off by the same amounts and retry at the same moment again. The fix is jitter: a random extra wait per caller that spreads the retries out. The Amazon article says jitter "adds some amount of randomness to the backoff to spread the retries around in time".
The example uses both, quoted from executor.py:
def fnv1a32(text: str) -> int:
"""FNV-1a, 32 bit. Small enough to port to JavaScript line for line (the labs do)."""
h = 0x811C9DC5
for byte in text.encode("utf-8"):
h ^= byte
h = (h * 0x01000193) & 0xFFFFFFFF
return h
def backoff_ms(base_ms: int, failed_attempt: int, seed: str) -> int:
"""Wait after attempt `failed_attempt` failed: exponential delay plus jitter in [0, delay/2]."""
delay = base_ms * 2 ** (failed_attempt - 1)
return delay + fnv1a32(f"{seed}:{failed_attempt}") % (delay // 2 + 1)
The jitter here is not random. It is a hash of the run id, the call id and the attempt number, so two different calls get different waits, and the same call gets the same wait on every machine. That makes the example's output reproducible byte for byte and lets the labs' JavaScript match the Python exactly. A production adapter would use a random number; the spreading effect is the same.
Each try of one call has a number. The attempt number counts them: 1 for the first try, 2 for the first retry. It appears in meta.attempt_number of every result envelope, and as an attribute on each attempt's span in the trace, so anyone reading either can tell a first-try answer from one that needed a retry.
The timeline, run#
The script below was written for this page and run in a scratch copy of examples/uw-tools; it is not one of the example's files. It prints the backoff waits for one seed, then sends get_roof_assessment for SUB-1004 through the real adapter under five scripted provider faults and prints every attempt on the simulated clock: start, end, outcome, and the wait after it. A successful run proves four things: a retry turns a single timeout into a success, a provider that never answers ends within the 5000 ms deadline however many retries are allowed, a rate-limited provider is given the wait it asked for, and the provider is never called more often than the attempts shown. The model here is a scripted stand-in, so these results show how the tool layer works, not how good any model is.
from executor import AdapterOptions, Executor, backoff_ms
from functions import World
from run_cases import load_cases
from tracing import Tracer
UW004 = load_cases("UW-004")[0]
print("waits after failed attempt n, seed run-7f3a:call_01:",
[backoff_ms(200, n, "run-7f3a:call_01") for n in (1, 2, 3, 4)])
def roof(label, script, retries=1):
world = World(UW004, provider_script={"get_roof_assessment": script})
record = Executor(options=AdapterOptions(max_retries=retries)).execute_detailed(
world, "underwriting-agent", "get_roof_assessment", {"submission_id": "SUB-1004"}, "call_01",
Tracer("run-7f3a", world.clock), None, "run-7f3a")
steps = "; ".join(f"#{s['attempt_number']} {s['start_ms']}-{s['end_ms']} {s['outcome']}"
+ (f", wait {s['wait_ms_after']}" if s["wait_ms_after"] else "") for s in record.timeline)
env = record.envelope
print(f"{label:<26} {steps}")
print(f"{'':<26} -> {'ok' if env['ok'] else env['error']['code']}, attempt {env['meta']['attempt_number']}, "
f"{env['meta']['duration_ms']} ms, provider called {record.provider_calls}x")
roof("timeout once, 0 retries", ["timeout"], retries=0)
roof("timeout once, 1 retry", ["timeout"])
roof("timeout always, 1 retry", ["timeout"] * 9)
roof("timeout always, 3 retries", ["timeout"] * 9, retries=3)
roof("rate limited (UW-T05)", ["rate_limited"])
cd examples/uw-tools
/opt/homebrew/bin/python3 faults_timeline.py
waits after failed attempt n, seed run-7f3a:call_01: [207, 534, 901, 1614]
timeout once, 0 retries #1 0-2000 timeout
-> timeout, attempt 1, 2000 ms, provider called 1x
timeout once, 1 retry #1 0-2000 timeout, wait 207; #2 2207-2387 ok
-> ok, attempt 2, 2387 ms, provider called 2x
timeout always, 1 retry #1 0-2000 timeout, wait 207; #2 2207-4207 timeout
-> timeout, attempt 2, 4207 ms, provider called 2x
timeout always, 3 retries #1 0-2000 timeout, wait 207; #2 2207-4207 timeout, wait 534; #3 4741-5000 timeout
-> timeout, attempt 3, 5000 ms, provider called 3x
rate limited (UW-T05) #1 0-30 rate_limited, wait 2000; #2 2030-2210 ok
-> ok, attempt 2, 2210 ms, provider called 2x
The first line shows the waits growing (207, 534, 901, 1614 ms): each is the base delay, 200 × 2 to the power n−1, plus a jitter of up to half of it. The first two timelines are the reason retries exist: the same single slow answer is a failure with no retry and a success after 2387 ms with one.
The fourth timeline is the reason deadlines exist. Three retries were allowed, so up to four attempts. The third attempt started at 4741 ms with only 259 ms left before the 5000 ms deadline, so its timeout was cut from 2000 ms to 259 ms, it failed at exactly 5000 ms, and no fourth attempt started. The deadline, not the retry count, decided how long the call took.
Diagram: One call, two attempts: timeout, jittered wait, success
Time runs down the page, on the example's simulated clock. The first attempt to the roof-data provider gets no answer and is cut off by its 2000 ms timeout. The adapter waits 207 ms (200 ms of backoff plus jitter) and tries again. The second attempt succeeds at 2387 ms, well inside the 5000 ms deadline of the call.
- Execution adapter to Roof-data provider: attempt 1 at 0 ms. get_roof_assessment for SUB-1004, timeout 2000 ms.
- Roof-data provider to Execution adapter: no answer: timeout at 2000 ms. timeout is retryable, one retry is allowed, and the wait fits before the deadline.
- Execution adapter: wait 207 ms (backoff + jitter). 200 ms times 2 to the power 0, plus jitter from a hash of the run and call ids.
- Execution adapter to Roof-data provider: attempt 2 at 2207 ms. attempt_number 2; the same arguments.
- Roof-data provider to Execution adapter: roof age 22 years at 2387 ms. ok on attempt 2; meta says attempt_number 2, duration_ms 2387.
The diagram draws the "timeout once, 1 retry" line as a sequence: the adapter, the provider, the failed first attempt, the jittered wait, and the second attempt that succeeds.
The lab below starts with no retries: a single timeout, and the call fails. Raise the retries to 1 and run it again; then pick "timeout always" and raise them to 3, and watch the deadline cut the last attempt. The table after it holds the same results.
| Fault | Expected | Result in this example (validation on, 1 retry, key on) |
|---|---|---|
| timeout once | ok after one retry | timeout at 2000 ms, wait 207 ms, ok on attempt 2 at 2387 ms; with 0 retries: timeout at 2000 ms |
| timeout always | a controlled timeout within the 5 s deadline | timeout after attempt 2 at 4207 ms; with 2 or 3 retries the third attempt is cut to the deadline, 5000 ms |
| rate limited | wait what the provider asks, then ok | rate_limited at 30 ms, wait 2000 ms, ok on attempt 2 at 2210 ms |
| reply lost after a write, key on | exactly one row | timeout at 1000 ms (the row was stored), wait 207 ms, attempt 2 returns the stored result (idempotent_replay), 1 row |
Rate limits: the provider asks you to wait (UW-T05)#
A rate limit is a cap a provider puts on how many requests a caller may send in a period. When a caller exceeds it, the provider refuses the request and usually says how long to wait. On HTTP this has a standard form. RFC 6585, read on 2026-11-18, defines status 429 Too Many Requests for a user who "has sent too many requests in a given amount of time", and says the response MAY include a Retry-After header indicating how long to wait. RFC 9110 defines that header's value as either an HTTP date or a number of seconds.
This site's own case UW-T05 is the roof-data provider answering "rate limited, retry after 2 s". The last timeline above shows the adapter's response: the refusal came back after 30 ms, the adapter waited the full 2000 ms the provider asked for (its own backoff would have been only 207 ms), and the second attempt succeeded at 2210 ms. The rule is to wait the longer of the two: max(backoff, retry_after_ms). A provider that tells you when to come back knows more about its own load than your backoff formula does.
Two consequences follow. A rate limit is not a failure of the provider, so the circuit breaker below does not count it. And a Retry-After can be longer than the deadline allows; the adapter then does not retry at all and returns rate_limited with retry_after_ms in the envelope, so the model, or a person, knows when the provider will answer again.
Circuit breakers: stop calling a provider that is down (UW-T06)#
Retries handle a provider that fails now and then. They make things worse when the provider is down for minutes: every call spends its full timeout, retries, spends it again, and the provider, if it is trying to restart, receives a steady stream of traffic it cannot serve.
A circuit breaker is a small state machine, kept per provider, that stops calls to a provider after it has failed several times in a row, and lets one trial call through after a cool-down. Martin Fowler's article "CircuitBreaker" (2014), read on 2026-11-18, describes the idea: wrap a protected call in an object that monitors failures; once failures reach a threshold, the breaker trips and calls return an error without the protected call being made; after a timeout it lets a trial call through. The example's breaker is that idea in about thirty lines of executor.py. Its three methods that decide everything, quoted verbatim without the class indentation:
def allow(self, now_ms: int) -> bool:
"""May a call go to the provider now? Moves open -> half_open when the cool-down has ended."""
if self.state == "open" and self.opened_at_ms is not None and now_ms >= self.opened_at_ms + self.cooldown_ms:
self._move("half_open", now_ms)
return self.state != "open"
def record_success(self, now_ms: int) -> None:
self.failures = 0
self._move("closed", now_ms)
def record_failure(self, now_ms: int) -> None:
self.failures += 1
if self.state == "half_open" or self.failures >= self.threshold:
self.opened_at_ms = now_ms
self._move("open", now_ms)
Three states. Closed: calls go through, and failures in a row are counted; a success resets the count. Open: calls fail at once with provider_unavailable, without calling the provider, until the cool-down ends. Half-open: one trial call goes through; success closes the breaker, failure opens it again. In the example the threshold is 5 failures in a row and the cool-down is 30 s.
This site's own case UW-T06 needs time to pass across runs, so the runner gives five runs one shared clock and one shared set of breakers. Runs 1 to 4 start 10 s apart, with a provider that answers every request with a server error; run 5 starts at 60 s, when the provider has recovered. The script below was written for this page and run in a scratch copy of the example; it calls the runner's own run_t06 and prints, per run, the roof lookup's result, its attempts, the real provider calls, the recommendation and the breaker's state afterwards, then the breaker's transitions and the error the model read in run 4. A successful run proves that the breaker opens on the fifth failure, that run 4 does not call the provider at all, and that one trial call closes it again.
import run_cases
case = run_cases.load_cases("UW-T06")[0]
runs, rows, transitions = run_cases.run_t06(case)
for r in runs:
roof = next(c for c in r["calls"] if c["name"] == "get_roof_assessment")
print(f"{r['label']:<52} roof: {roof['error_code'] or 'ok':<20} attempts {roof['attempts']} "
f"provider {r['provider_calls'].get('get_roof_assessment', 0)} -> {r['final']['recommendation']:<12} "
f"breaker {r['breaker_after']}")
print("transitions:", [(t["at_ms"], t["from"], t["to"]) for t in transitions])
fast = [m for m in runs[3]["messages"] if m["role"] == "tool"][0]["content"]
print("run 4, the model reads:", fast["error"])
/opt/homebrew/bin/python3 breaker_runs.py
run 1: provider down roof: provider_unavailable attempts 2 provider 2 -> request_info breaker closed
run 2: provider down roof: provider_unavailable attempts 2 provider 2 -> request_info breaker closed
run 3: the fifth failure opens the breaker roof: provider_unavailable attempts 2 provider 1 -> request_info breaker open
run 4: breaker open, the call fails fast roof: provider_unavailable attempts 1 provider 0 -> request_info breaker open
run 5: after the cool-down, one trial call succeeds roof: ok attempts 1 provider 1 -> escalate breaker closed
transitions: [(20040, 'closed', 'open'), (60000, 'open', 'half_open'), (60180, 'half_open', 'closed')]
run 4, the model reads: {'code': 'provider_unavailable', 'message': 'The roof-data provider failed repeatedly; calls to it are paused for 20.0 s more. Continue without it.', 'retryable': False}
Count the provider calls: 2, 2, 1, 0, 1. Run 3's first attempt was the fifth failure in a row, which opened the breaker at 20040 ms; its retry then found the breaker open and failed fast, so the provider saw only one request from that run. Run 4, at 30 s, never reached the provider. Its result says calls are paused for 20 more seconds and, unlike an ordinary provider_unavailable, sets retryable to false, because asking again now will fail the same way. At 60 s the cool-down is over, the breaker lets a trial call through, the provider answers, and run 5 recommends escalate under R-ROOF-03, as UW-004's facts require.
Diagram: The circuit breaker: closed, open, half-open (UW-T06)
Read left to right. Closed lets calls through and counts failures in a row. The fifth failure opens it, at 20040 ms in UW-T06. Open fails every call fast without calling the provider. When the 30 s cool-down ends it becomes half-open and lets one trial call through; success closes it, failure opens it again.
- 1 · Closed
- Closed: Counts timeout and provider_unavailable in a row; a success resets the count.
- 2 · Open
- Open: Every call returns provider_unavailable, retryable false, without calling the provider.
- 3 · Half-open
- Half-open: After the cool-down, one call is allowed through to test the provider.
The diagram shows the three states and what moves the breaker between them, with the times from UW-T06.
The lab below lets you set the threshold and the cool-down and feed the breaker your own sequence of outcomes. It starts on UW-T06's attempts. The table holds the same steps for a reader without JavaScript.
| Time | Provider would answer | State before | Allowed | State after |
|---|---|---|---|---|
| 0 ms | fail | closed | yes | closed (1 failure) |
| 256 ms | fail | closed | yes | closed (2) |
| 10 s | fail | closed | yes | closed (3) |
| 10.251 s | fail | closed | yes | closed (4) |
| 20 s | fail | closed | yes | open (5 in a row) |
| 20.323 s | fail | open | no: fails fast, provider_unavailable, retryable false | open |
| 30 s | fail | open | no: fails fast | open |
| 60 s | ok | open, then half-open (cool-down over) | yes: the trial call | closed |
Backpressure: slow the intake, not only the calls#
Timeouts, retries and breakers protect one call. Backpressure protects the whole system: when a downstream provider cannot keep up, the pressure is pushed back to whoever is sending work, so that work waits at the entrance or is refused there, instead of piling up in the middle. A tiny example: the roof-data provider can serve about ten requests at a time (an illustrative number). If fifty runs start at once, an adapter with a limit of ten concurrent calls per provider lets ten through, makes the rest wait a bounded time for a free slot, and answers the ones that wait too long with a retryable error. Without the limit, all fifty hit the provider, all fifty time out, and all fifty retry.
The example does not implement a concurrency limit; it runs one call at a time on a simulated clock. Its breaker and deadline are backpressure's cousins: the breaker refuses work for a provider that is down, and the deadline caps how long any call may hold resources. A production adapter adds the limit per provider, sized from the provider's capacity, and reports a full queue with a code the model can act on.
Agents add one source of pressure that ordinary services do not have. A model may request several tools in one turn, as Tool calling showed with UW-004's three reads, and several runs may do so at once. Each parallel call is load. MCP puts the duty on the server: the tools specification, read on 2026-11-18, says servers MUST rate limit tool invocations. A tool server that does so turns an overloaded provider into a clear rate_limited result instead of a slow timeout.
What the model is told (UW-006)#
Back to UW-006, where every attempt timed out. After two attempts, the model reads this result, quoted from the run's messages.json in the scratch copy:
{"ok": false, "error": {"code": "timeout", "message": "The roof-data provider did not answer in 2.0 s.", "retryable": true}, "meta": {"tool": "get_roof_assessment", "attempt_number": 2, "duration_ms": 4221, "evidence_ids": []}}
Every field carries a decision the adapter already made. attempt_number 2 says a retry has already happened. retryable: true says the failure is of a kind that may pass, but the call's retry budget is spent; Results explains that flag. The message names the dependency and the time, not a host or a stack trace. The stand-in model reads it, does not ask again in this run, continues with the other tools, and ends with a controlled fallback instead of a guess:
{"recommendation": "request_info", "reasons": ["The roof-data provider did not answer (timeout) after the adapter's attempts; the stated roof age (12 years) is unverified. Request the roof assessment."], "citations": [], "missing": ["roof_assessment"], "stated_confidence": 0.5, "requires_human": true, "policy_version": "2026.2", "evidence_ids": []}
The model recommends; a person decides, and the underwriter sees a request for the missing roof assessment, not a recommendation built on a roof age nobody verified. That is the goal of this whole page: a provider failure becomes an explicit gap in the recommendation, never a silent assumption.
Why should the model not retry too? Because retries multiply across layers. The Amazon article gives the arithmetic: in a stack five layers deep where each layer retries three times, load on the bottom layer could rise 243-fold, and its advice is to retry at a single point in the stack. In the example that point is the adapter. A model that also retries, inside a harness that also restarts failed runs, multiplies the adapter's attempts by its own. Telling the model "already retried" (attempt_number 2) and "paused" (retryable: false from an open breaker) is how the adapter keeps the model from becoming a second retry layer.
How MCP and the libraries express the same ideas#
The MCP specification, revision 2026-07-28, as documented on 2026-11-17 and checked again on 2026-11-18 for the tools page, has three relevant rules. The cancellation page says implementations should set timeouts on all sent requests, may reset them when progress notifications arrive, but should enforce a maximum: that maximum is a deadline. A client cancels a request with notifications/cancelled on stdio, and by closing the response stream on Streamable HTTP; both sides must tolerate the race in which the response was already sent. And the tools page asks clients to implement timeouts for tool calls and servers to rate limit invocations. MCP transports covers how those messages travel.
On the Java side, Resilience4j's circuit breaker documentation, read on 2026-11-18, describes the same three states (CLOSED, OPEN, HALF_OPEN, plus special states), a sliding window that is either count-based or time-based, CallNotPermittedException for calls rejected while open, and defaults that differ from the example's: open at a 50% failure rate over a window of 100 calls, wait 60 s in the open state, then permit 10 trial calls. The example opens on 5 failures in a row and permits one trial call. Neither is the right number; both are policies to tune from the provider's real behavior. Documented shape; not run here.
Anthropic's handle-tool-calls page, read on 2026-11-18, advises instructive error messages and gives "Rate limit exceeded. Retry after 60 seconds." as an example of one, which is the envelope's rate_limited message in other words.
What this is not. This page makes one tool call reliable. Recovering a whole workflow after failures, with queues, compensating steps, sagas, service level objectives and incident response, is system-wide reliability and belongs to https://reliable.sammuthu.com. Measuring latency and failure rates as a discipline, with dashboards and alerts, belongs to https://observe.sammuthu.com.
In a coding agent. A coding agent'srun_testsneeds a deadline more than anything: a test suite that hangs holds the whole session. The harness gives it one, kills the process when it passes, and returnstimeoutwith the name of the test that was running. Agit_pushthat times out is theUW-T04problem again: the push may have reached the server. Before retrying, the agent checks whether the remote branch already has the commit, which is the same move as looking up an idempotency key. Harness Engineering covers the runtime that enforces those limits.
Retry budgets and the multiplication across layersOne place retries, everyone else reports; a budget caps retries when everything fails at once
Count the layers that might retry one roof lookup in a real deployment: the HTTP client library inside the wrapper, the adapter, an MCP client that reconnects, the model that asks again, the harness that restarts a failed run, and a queue that redelivers a failed job. If each tries twice, one slow answer becomes 64 requests. That is how a brief slowdown turns into an outage.
The defense has two parts. First, choose one layer to retry and make every other layer report. In the example the adapter retries; the wrapper raises a failure and does not retry inside; the model is told the attempt number and does not retry within the run. A harness that restarts runs should check whether the run failed on a provider whose breaker is open, and wait for the cool-down instead.
Second, cap retries globally, not only per call. The Amazon article describes a token bucket: each retry spends a token, tokens refill at a fixed rate, and when the bucket is empty calls are not retried at all. In normal operation every call may retry; when everything fails at once, retries fall to the refill rate, so the retry traffic cannot exceed a fixed share of the real traffic. A per-provider budget fits naturally next to the per-provider breaker.
A timeout on a write is an unknown, not a failureCancellation does not undo; the idempotency key turns the unknown back into a known
A timeout on a read means "no answer"; the next attempt asks again and nothing has changed. A timeout on a write means "unknown": the store may have committed it, may be committing it now, or may never have received it. The example models the first possibility directly: ReplyLost is a subclass of ProviderTimeout, so the adapter sees an ordinary timeout while the row exists.
Cancelling does not help. MCP's cancellation notification, as documented, tells the receiver the sender is no longer interested; both sides must tolerate a response that was already sent, and nothing in it rolls back work. An HTTP client that closes its connection gives the server no promise either. So after a write times out, there are only two honest options: retry with the same idempotency key, which returns the stored result if the first attempt landed (UW-T04), or ask the store whether a row with that key exists before deciding. Retrying without a key is the third option, and it produces REC-0002.
The deadline adds a twist. A write whose last attempt was cut by the deadline may still complete after the adapter has given up and told the model timeout. The next run, with a new run id and so a new key, may then record a second recommendation. If that matters, choose a key that spans runs for the same submission and outcome, or have the cockpit show the latest recommendation per submission rather than all of them. Safety discusses the scope of a key.
Designing the breakerPer provider or per tool, count or rate, one trial or ten, and where its state lives
Granularity first. The example keeps one breaker per provider, not per tool, because a provider that is down is down for every tool that uses it. Two tools on one provider share its fate; one tool on two providers needs two breakers. Going finer, per endpoint or per region, helps when a provider fails partially.
Then what counts. The example counts timeout and provider_unavailable, five in a row. A count in a row is easy to reason about and easy to fool: a provider that fails four times out of five never trips it. A failure rate over a sliding window, Resilience4j's default, catches that pattern but needs enough calls to be meaningful, which is why that library also has a minimum number of calls. Slow calls can count as failures too, which protects callers from a provider that answers, eventually.
Half-open is where breakers misbehave. One trial call, as in the example, is gentle on a recovering provider but slow to close if the first trial is unlucky. Several trials close faster and risk a burst. Whatever you choose, the trial calls should be ones that are safe to fail: reads, not writes.
Finally, where the state lives. The example's breakers live in one process, and UW-T06 shares them across runs by sharing the executor. Several application instances each keep their own breakers, so each must discover the outage separately, which is usually acceptable and much simpler than a shared store. What matters more is that the breaker's state is visible: an open breaker should be an event in the trace and a line on a dashboard, because "every run is recommending request_info" is otherwise a mystery.
Degrade, fail, or wait: what the run should do without a toolThe model's fallback must be visible in the recommendation and testable in the trace
When a tool cannot answer, a run has three options. It can degrade: continue without the tool and record the gap, as UW-006 does with missing: ["roof_assessment"] and request_info. It can fail: stop the run and report that it could not complete, which is right when the missing evidence makes any recommendation misleading. Or it can wait: park the run until the provider recovers, which needs durable state and a scheduler, and belongs to the workflow layer, not the tool.
The choice is a product decision, per tool and per case. For the roof lookup, degrading is right, because the policy fixture has a rule for an unverified roof and the underwriter can request an inspection. For check_eligibility, which is deterministic code, a failure is a bug, and degrading would hide it. Say which tools may be degraded in the instructions the model gets, and check it: a trace in which get_roof_assessment failed and the recommendation still cites R-ROOF-01 is a fallback that invented evidence. Testing builds such checks.
Whatever the run does, the deadline of the whole run matters as much as the deadline of each call. A run that spends 4.2 s on a dead provider for each of ten submissions makes the cockpit feel broken even when every individual call behaved. An open breaker that fails fast is a feature for the person waiting.
Checklist for Monday#
| Check | Why |
|---|---|
| Every attempt has a timeout set from the provider's measured latency | A dead provider cannot hold the run hostage |
| Every call has a deadline that bounds all attempts and waits | Retries cannot stretch a call past what anyone will wait (5000 ms here) |
Only timeout, rate_limited and provider_unavailable are retried | invalid_arguments and not_found give the same answer again |
| A write is retried only with an idempotency key | A lost reply stays one row (UW-T04) |
| Waits grow exponentially and carry jitter | Callers that failed together do not retry together |
| A rate limit's retry-after is honored, and does not count as a breaker failure | UW-T05: wait 2000 ms, then succeed |
| A circuit breaker per provider opens after repeated failures and fails fast | UW-T06: run 4 never reaches the provider |
| Exactly one layer retries; the result tells the model what was already tried | Retries across layers multiply |
| The fallback is visible in the recommendation | UW-006: request_info, missing roof_assessment |
In one breath#
A tool's provider will be slow, busy or down, so every attempt gets a timeout and every call a deadline. Only failures that may pass are retried, with exponential backoff and jitter, a provider's retry-after is honored, and a write is retried only behind an idempotency key. A circuit breaker per provider stops calling one that is down, and the result tells the model what was already tried, so the run degrades into a visible gap instead of a guess.
You can now keep a tool call bounded and honest when its provider misbehaves. What you cannot yet do is show that all of this still works next month, after someone changes a schema, a message or a retry count: the adapter has two callers, code and a model, and both need tests. That is Testing.