A broker's email reaches the insurer's submissions inbox:
From: broker@brokerage.example
To: submissions@insurer.example
Subject: Quote request SUB-1004
Please quote property cover for a masonry office, built 2001, 20,000 sq ft, sprinklered. The owner says the roof was replaced 9 years ago. Wind zone C. No losses.
Less than a second later on the example's simulated clock (the row is stored at 586 ms), an underwriter has a recommendation in the cockpit: escalate to a senior underwriter, because the roof-data provider says the roof is 22 years old, not 9. Fifteen pages of this site each looked at one step of what happened in between. Seen one step at a time, every step is reasonable. The questions that matter in production cross the steps. If the provider had timed out, would the row still be stored once? If a second agent with fewer permissions had asked to store it, who would have noticed? When the underwriter disagrees, which code writes that, and with whose authority?
This page answers those questions on one run, case UW-004, and then on all 18 cases of the example. Every output on it came from running examples/uw-tools in a scratch copy on 2026-11-18. The model here is a scripted stand-in, so these results show how the tool layer works, not how good any model is.
Honest boundary. This is an illustrative reconstruction for teaching. It follows the shape of a real project by the author, but the cases, the rules, the fields, the provider and every number are invented. Nothing here reports the quality of a real system or model. Example roof thresholds are synthetic, not insurance practice.
The whole system in one picture#
Start with the parts and who owns each one. An intake step turns the email into a submission: one JSON record with ten fields in the example. The underwriting agent is a loop: the application sends the submission and the tool definitions to the model; the model writes tool requests; the execution adapter checks and runs them; the results go back; the loop repeats until the model writes a final answer. The answer is a recommendation, stored as one row in recommendations with status awaiting_underwriter. The underwriter reads it in the underwriter cockpit, the web app where people review recommendations and record decisions, and the cockpit writes the decision into a second table, decisions, through its own code.
Diagram: From a broker's email to an underwriter's decision
Read left to right. Intake turns the email into a submission. The model proposes tool calls. The execution adapter checks and runs each one and stores the single recommendation row. The underwriter reads it in the cockpit, and the cockpit's own code writes the decision. The model's lane touches no table.
- 1 · Intake
- Submission: An intake step turns the broker's email into one JSON record; later steps read only this.
- 2 · Model (stand-in)
- Proposes calls: Writes tool requests with a name, arguments and a call id, then the final recommendation.
- 3 · Execution adapter
- Checks and runs: Refuses what it must, retries what may pass, shapes every outcome into an envelope.
- recommendations row: The one write, stored once under the application's idempotency key.
- 4 · Underwriter cockpit
- Cockpit: The web app where an underwriter reads the recommendation, its reasons and its evidence.
- 5 · Person
- decisions row: The person decides; the cockpit's own code stores it with the person's role. No tool can.
The diagram has five lanes, and the rule they encode is the first primitive of this site. The model proposes a call. Your code decides whether it runs. The model's lane holds no box that touches a table. The adapter's lane holds every check. And the last lane belongs to a person: the model recommends; a person decides.
Two pieces of the world sit outside the agent. The roof-data provider is the outside service that assesses a building's roof from aerial images and returns an age; in the example it is a stand-in inside functions.World, and on SUB-1004 it says 22 years. The policy fixture is the insurer's written underwriting guidance as data: rules with ids, grouped in versions. The example ships UW-GUIDE version 2026.2 (current, effective 2026-07-01) and the older 2025.4 in policy_fixture.json; both are illustrative.
| Rule id (policy fixture 2026.2) | Rule |
|---|---|
R-ROOF-01 | Roof older than 25 years in wind zone C: not eligible |
R-ROOF-02 | Roof age unknown: request the roof age or an inspection |
R-ROOF-03 | Application and provider roof ages differ by more than 5 years: escalate to a senior underwriter |
R-SPRK-01 | Warehouse over 50,000 sq ft without sprinklers: refer to a senior underwriter |
R-HUMAN-01 | Every recommendation needs a decision by an authorized underwriter |
In version 2025.4, R-ROOF-01 drew the line at 30 years instead of 25. Case UW-007 depends on that difference.
The catalog: five tools, one write#
The agent is offered five tools. Four read and one writes. Each needs a scope, a named permission the caller must hold. This is the output of tools.py, which prints the registry as a table:
cd examples/uw-tools
/opt/homebrew/bin/python3 tools.py
name kind scope provider hints
get_roof_assessment read uw.read roof-data-provider readOnly,idempotent,openWorld
search_guidance read uw.read guidance-index readOnly,idempotent
find_similar_decisions read uw.read decision-history readOnly,idempotent
check_eligibility read uw.read rules-engine readOnly,idempotent
record_recommendation write uw.recommend recommendation-store idempotent
The hints are annotations: statements about behavior that a client may show or use, never permissions. Discovery explains them. Three callers exist. The underwriting-agent holds uw.read and uw.recommend; the assessment-agent, which only assesses roofs, holds uw.read; an unknown caller holds nothing. Pick a tool and a caller in the lab to see the definition the model reads and which tools each caller is offered. The table holds the same information.
| Tool | Kind | Scope | Provider | Timeout | Annotations (hints) |
|---|---|---|---|---|---|
get_roof_assessment | read | uw.read | roof-data-provider | 2000 ms | readOnly, idempotent, openWorld |
search_guidance | read | uw.read | guidance-index | 1000 ms | readOnly, idempotent |
find_similar_decisions | read | uw.read | decision-history | 1000 ms | readOnly, idempotent |
check_eligibility | read | uw.read | rules-engine | 500 ms | readOnly, idempotent |
record_recommendation | write | uw.recommend | recommendation-store | 1000 ms | idempotent (destructive false) |
underwriting-agent is offered all five; assessment-agent the four reads; unknown none.
What the catalog leaves out matters as much. No tool approves, declines, binds a policy, quotes a price or changes a submission. A test, test_no_tool_decides_or_changes_a_submission, fails if a tool name ever contains one of those words; Testing quotes it. Every input schema is strict: submission_id must match ^SUB-[0-9]{4}$, the recommendation is one of four values, requires_human must be true, and no object accepts a field it does not name. Schemas builds those schemas from zero.
One run, turn by turn#
Now the run of UW-004, with run id run-7f3a. A run id names one execution of the workflow for one submission; every span of the trace, every audit event and the idempotency key of the write carry it, so anything recorded during the run can be traced back to it. The application writes the first message: the broker's email, the submission, the run id, and the idempotency key it chose for the one write of this run, SUB-1004:run-7f3a:recommendation.
The stand-in model answers with three read requests in one turn: call_01 for the roof, call_02 for guidance, call_03 for similar past decisions. The adapter runs each one and the application appends each result under its call id. The roof result, as the model read it:
{"ok": true, "data": {"submission_id": "SUB-1004", "status": "ok", "roof_age_years": 22, "source": "roof-data provider (stand-in)", "method": "aerial image, synthetic", "evidence_id": "EV-ROOF-0412"}, "resource": {"uri": "uw://roof-assessments/SUB-1004", "mime_type": "application/json", "bytes": 351}, "meta": {"tool": "get_roof_assessment", "attempt_number": 1, "duration_ms": 180, "evidence_ids": ["EV-ROOF-0412"]}}
EV-ROOF-0412 is an evidence id: the identifier of one piece of evidence a tool returned, which the recommendation later cites so that a person can open exactly what the agent relied on. Then the model asks for check_eligibility (call_04), which applies the policy fixture in code and returns escalate under R-ROOF-03. Then it asks for record_recommendation (call_05) with the key from the first message. One row is stored. Then the model writes its final answer without a tool call, and the loop stops.
{"recommendation": "escalate", "reasons": ["The application states a roof age of 9 years; the roof-data provider assessed 22 years (EV-ROOF-0412).", "The two sources differ by 13 years, more than R-ROOF-03 allows: a senior underwriter should review."], "citations": ["R-ROOF-03"], "missing": [], "stated_confidence": 0.6, "requires_human": true, "policy_version": "2026.2", "evidence_ids": ["EV-ROOF-0412"]}
The lab below steps through that loop for this site's own case UW-T04, which is UW-004 with one fault on the write: the row is stored and the reply is lost. Use Back and Next, or the arrow keys; the bar marks who acts at each step. The picker also offers UW-004 and the other main cases. The table holds the UW-T04 run.
| Step | Who acts | What happens in UW-T04 |
|---|---|---|
| 1 | application | Sends the request (broker email, submission, run id and idempotency key) with the tool definitions |
| 2 | model | Proposes three calls in one turn: call_01 get_roof_assessment, call_02 search_guidance, call_03 find_similar_decisions |
| 3 to 8 | adapter, then application | Executes each call and appends each result under its call id; the roof result says 22 years, EV-ROOF-0412 |
| 9 to 11 | model, adapter, application | call_04 check_eligibility: outcome escalate, rule R-ROOF-03 |
| 12 to 14 | model, adapter, application | call_05 record_recommendation: attempt 1 stores the row and its reply is lost (timeout at 1000 ms); the adapter waits 251 ms; attempt 2 finds the key and returns the stored result, idempotent_replay true; one row |
| 15 | model | Final answer: recommend escalate, cite R-ROOF-03; no tool call, so the loop ends |
| 16 | application | Stops the loop; the underwriter decides in the cockpit |
Tool calling explains the protocol behind those steps, and how three providers name its messages.
The adapter: where "your code decides" lives#
Every request in that run went through the same function, Executor.execute_detailed in executor.py. Its docstring at the top of the file lists the order, quoted verbatim:
1. lookup the requested name must be in the registry -> unknown_tool
2. validate the arguments must match the input schema -> invalid_arguments (with paths)
3. authorize the caller must hold the tool's scope -> not_authorized (+ audit event)
(and, if switched on, a write needs a person's approval)
4. per attempt:
idempotency a write whose key is already stored returns the stored result (or conflict)
breaker if the provider's circuit breaker is open, fail fast -> provider_unavailable
execute run the function with a timeout per attempt
5. retry retryable errors are retried with exponential backoff and deterministic jitter,
honoring retry_after for rate_limited, never past the call's deadline
6. shape every outcome becomes a result envelope (see `envelope_ok` / `envelope_error`)
Each step has a case that exercises it. This site's own UW-T02 asks for get_roof_report, which does not exist, and stops at step 1. UW-T01 sends {"submission_id": 1004} and stops at step 2 with the path /submission_id; the model corrects it and the second request succeeds. UW-T03 stops at step 3. UW-T04 is decided at step 4. UW-006, UW-T05 and UW-T06 live in steps 4 and 5. Every outcome, success or refusal, leaves step 6 as a result envelope with ok, then data or error, then meta. Implementation walks through the adapter's code gate by gate, and Results designed the envelope.
The authorized write#
The write is where permission matters. record_recommendation needs uw.recommend, which only the underwriting-agent holds. In this site's own case UW-T03 the assessment-agent runs the same submission and, at the end, asks to store its recommendation. The adapter refuses at step 3, writes nothing, and records an audit event for people. This is out/UW-T03/audit.json from the scratch run:
[
{
"event": "tool_refused",
"run_id": "run-9d25",
"caller": "assessment-agent",
"tool": "record_recommendation",
"required_scope": "uw.recommend",
"tool_call_id": "call_05",
"reason": "missing_scope",
"at_ms": 574
}
]
The model reads not_authorized with the message "This caller may not use record_recommendation: it needs the scope uw.recommend. Nothing was written." It still writes its recommendation as the final answer, and adds a reason: the recommendation could not be stored, so a person must record it. The runner's summary shows rows 0 for this case. The adapter can also require a person's approval for every write, a switch the lab below lets you turn on. Pick a caller and a tool and see the decision, the envelope, the rows and the audit event.
| Caller | Tool | Approval required | Result | Audit event |
|---|---|---|---|---|
| underwriting-agent | record_recommendation | no | allowed, 1 row | recommendation_recorded |
| underwriting-agent | record_recommendation | yes, not given | refused, not_authorized: needs a person's approval; nothing written | tool_refused, reason approval_missing |
| assessment-agent | record_recommendation | either | refused, not_authorized: needs the scope uw.recommend; nothing written (UW-T03) | tool_refused, reason missing_scope |
| assessment-agent | any read | either | allowed | none |
| unknown | any tool | either | refused, not_authorized | tool_refused, reason missing_scope |
Safety explains least privilege, scopes and approval in full.
Faults: the run when the world misbehaves#
A run in production meets slow providers, busy providers and lost replies. The example injects each of them through World.provider_script, the stand-in world's list of what each provider does on its first, second and third call. The fault that matters most for a write is the lost reply of UW-T04. This is the write's timeline from the scratch run (a script written for this page that calls the runner's own run_one; the example's file is unchanged):
{'attempt_number': 1, 'start_ms': 574, 'end_ms': 1574, 'outcome': 'timeout', 'provider_called': True, 'wait_ms_after': 251}
{'attempt_number': 2, 'start_ms': 1825, 'end_ms': 1825, 'outcome': 'replay', 'provider_called': False, 'wait_ms_after': 0}
rows: 1 | check rows_without_key: {'field': 'rows_without_key', 'expected': 2, 'actual': 2, 'passed': True}
Attempt 1 reached the store, which saved the row, and then the reply was lost, so the adapter saw a timeout. Attempt 2 did not reach the store at all: the adapter found the idempotency key already stored and returned the stored result. One row. The runner then reran the case with the key switched off and counted two rows, which is what the last line checks. The idempotency key is the application's choice, the same for every attempt of one logical write; Safety designs it.
The lab starts on that fault. Switch the key off to see the second row; pick other faults and change the retries.
| Fault | Expected | Result in this example (validation on, 1 retry, key on) |
|---|---|---|
| none | ok on attempt 1 | ok, attempt 1, 180 ms, provider called once |
invalid arguments (submission_id 1004) | rejected before execution | invalid_arguments, attempt 0, provider never called; with validation off: not_found after 180 ms |
| timeout once | ok after one retry | timeout at 2000 ms, wait 207 ms, ok on attempt 2 at 2387 ms |
| timeout always | a controlled timeout within the 5 s deadline | timeout after attempt 2 at 4207 ms |
| rate limited | wait what the provider asks, then ok | rate_limited at 30 ms, wait 2000 ms, ok on attempt 2 at 2210 ms |
| reply lost after a write, key on | exactly one row | timeout at 1000 ms (the row was stored), wait 207 ms, attempt 2 returns the stored result, 1 row |
| reply lost after a write, key off | exactly one row | attempt 2 writes again: 2 rows (REC-0001, REC-0002) |
(The lab's wait is 207 ms and the run above waited 251 ms: the jitter is a hash of the run id and the call id, and the two runs have different run ids.) Three series cases show what the model is told when a provider fails for good. In UW-006 the roof provider times out twice; the model reads timeout with attempt_number 2, does not ask again, and recommends request_info with missing set to roof_assessment. In UW-T05 the provider asks for a 2 s wait; the adapter honors it and attempt 2 succeeds. In UW-T06 five failures in a row open the circuit breaker, and later calls fail fast without reaching the provider until the 30 s cool-down ends. Reliability has the timelines.
The trace: what really happened#
Every run leaves a trace. On this site there is one trace per run, a span for each tool call, and a child span for each attempt with its attempt_number. The script written for this page prints the tool-call spans of UW-004; a successful run proves that the trace carries the run id, five calls in request order, and one attempt each.
trace_id 0c17c0046d4371b42858072814d7b4ca run_id run-7f3a
call_01 get_roof_assessment ok start 0 dur 180 attempts [1]
call_02 search_guidance ok start 180 dur 240 attempts [1]
call_03 find_similar_decisions ok start 420 dur 150 attempts [1]
call_04 check_eligibility ok start 570 dur 4 attempts [1]
call_05 record_recommendation ok start 574 dur 12 attempts [1]
The trace answers questions the final answer cannot. Case UW-008 has the same facts as UW-001 and the same right answer, decline under R-ROOF-01, but its trace has four tool calls instead of five: search_guidance was never requested. Only a check on the trace sees it; Testing runs that check. The ids are deterministic in the example (derived from the run id with SHA-256) so that two runs produce the same bytes; real tracing uses random ids. Advanced maps these spans to the OpenTelemetry conventions.
The underwriter's decision (UW-009)#
The last step is not a tool. In UW-009 the submission is SUB-1009, a masonry office with a 14-year-old roof in wind zone B. No rule blocks it, and the agent recommends approve. The underwriter reads the broker's loss run, finds an open water damage claim that the application did not mention, and declines. That is an override: a person's decision that differs from the agent's recommendation, recorded as such so that disagreement can be counted and reviewed.
Who writes the decision? Not the agent. The cockpit calls its own function, which is in functions.py and deliberately not in the registry. Quoted verbatim:
def record_underwriter_decision(world: World, submission_id: str, human_decision: dict[str, Any]) -> dict[str, Any]:
"""The cockpit's own code path. A person decided; this stores it. The agent has no tool for this.
`human_decision` comes from the case file (UW-009): who decided (a role), what, and why.
"""
recommended = next(r for r in reversed(world.state["recommendations"]) if r["submission_id"] == submission_id)
row = {"submission_id": submission_id, "recommendation_id": recommended["recommendation_id"],
"human_decision": {"decided_by": human_decision["decided_by"], "decision": human_decision["decision"],
"override": human_decision["decision"] != recommended["recommendation"],
"reason": human_decision["reason"]}}
world.state["decisions"].append(row)
world.audit("decision_recorded", submission_id=submission_id, actor=human_decision["decided_by"], actor_type="human",
override=row["human_decision"]["override"])
return row
The scratch script ran the case, counted the decisions rows after the agent's run, then called the runner's cockpit_step, which calls that function. A successful run proves that no tool wrote a decision and that the cockpit's row carries the person's role and the override.
recommended: approve | decisions rows after the run: 0
cockpit row: {"submission_id": "SUB-1009", "recommendation_id": "REC-0001", "human_decision": {"decided_by": "underwriter", "decision": "decline", "override": true, "reason": "The broker's loss run shows an open water damage claim that is not on the application."}}
last audit event: {"event": "decision_recorded", "submission_id": "SUB-1009", "actor": "underwriter", "actor_type": "human", "override": true, "at_ms": 586}
The audit event says actor_type: "human". If a model ever asks for record_underwriter_decision by name, the adapter answers unknown_tool, and test_cockpit_decision_uw_009 asserts exactly that. The model recommends; a person decides, and the code makes the sentence true rather than hoping it stays true.
Diagram: UW-009: two writes, two different writers
Time runs down the page. The model asks for record_recommendation; the adapter checks the scope and stores the approve recommendation. Later the underwriter declines in the cockpit, and the cockpit's own function writes the decision row with the person's role and override true. A request from the model for that function would be answered unknown_tool.
- Model (stand-in) to Execution adapter: record_recommendation: approve. SUB-1009, roof 14 years, wind zone B: no rule blocks it.
- Execution adapter to Store: insert recommendations row. The caller holds uw.recommend; REC-0001, status awaiting_underwriter.
- Execution adapter to Model (stand-in): ok, recorded. The envelope with recommendation_id and status; the model writes its final answer.
- Underwriter cockpit: underwriter declines. An open water damage claim is not on the application.
- Underwriter cockpit to Store: insert decisions row. decided_by underwriter, decision decline, override true; audit decision_recorded, actor_type human.
The sequence shows the two writes of UW-009 and who makes each one: the adapter stores the recommendation for the agent; the cockpit stores the decision for the person.
All 18 cases#
The example has ten series cases, shared with the other sites of the series, and eight of this site's own, UW-T01 to UW-T08. The runner executes each one, compares it with the tools.expected block in its case file, writes out/, and exits with 0 only if every case matches. This is the run from the scratch copy:
/opt/homebrew/bin/python3 run_cases.py
case result recommendation calls errors rows
UW-001 PASS decline 5 - 1
UW-002 PASS approve 5 - 1
UW-003 PASS request_info 5 not_found 1
UW-004 PASS escalate 5 - 1
UW-005 PASS decline 5 - 1
UW-006 PASS request_info 5 timeout 1
UW-007 PASS decline 5 - 1
UW-008 PASS decline 4 - 1
UW-009 PASS approve 5 - 1
UW-010 PASS escalate 5 - 1
UW-T01 PASS escalate 6 invalid_arguments 1
UW-T02 PASS escalate 6 unknown_tool 1
UW-T03 PASS escalate 5 not_authorized 0
UW-T04 PASS escalate 5 - 1
UW-T05 PASS escalate 5 - 1
UW-T06 PASS escalate 25 provider_unavailable x4 1
UW-T07 PASS escalate 6 - 1
UW-T08 PASS decline 6 unknown_tool 1
18/18 cases match their expected block. (Scripted stand-in model: this shows the tool layer, not model quality.)
calls counts the tool requests the stand-in wrote; errors lists the error codes in results the model read (an error a retry repaired, as in UW-T04 and UW-T05, does not appear); rows counts rows in recommendations afterwards. UW-T06 is five runs on one clock, hence 25 calls. What each case shows, in the words of its case file:
| Case | What it shows about the tool layer | Page |
|---|---|---|
UW-001 | The plain path: four reads, then one write | Retrieval |
UW-002 | A clean submission: no blocker is invented | this page |
UW-003 | The provider has no record: the result says not_found. It is data, not an exception | Results |
UW-004 | Two sources disagree on the roof age. The default case of this site | Tool calling |
UW-005 | A retrieved passage carries an instruction. Tool results are untrusted data | Retrieval |
UW-006 | The provider times out: one retry by the adapter, then a controlled fallback | Reliability |
UW-007 | A past decision carries its policy version. The current rules win | Retrieval |
UW-008 | Right recommendation, but search_guidance was never requested. Only the trace of calls shows it | Testing |
UW-009 | The underwriter declines after an approve recommendation. The cockpit writes the decision; no tool does | this page |
UW-010 | Same input as UW-004. Repeated trials and statistics belong to eval.sammuthu.com; here it runs once | Testing |
UW-T01 | Validation rejects {"submission_id": 1004} before anything runs; the model corrects the argument | Schemas |
UW-T02 | get_roof_report is not in the catalog: unknown_tool, nothing runs, no internals named | Implementation |
UW-T03 | The assessment-agent holds uw.read only: its write is refused, no row, an audit event | Safety |
UW-T04 | The row is stored, the reply is lost, the retry carries the same key: one row; without the key, two | Safety |
UW-T05 | Rate limited, retry after 2 s: the adapter waits, and attempt 2 succeeds inside the deadline | Reliability |
UW-T06 | Five failures in a row open the breaker; later calls fail fast until the cool-down ends | Reliability |
UW-T07 | 14 matches, a page of 5: a partial result with next_cursor; the next page only when needed | Advanced |
UW-T08 | The email asks to update the roof age: no tool can change a submission, so the request is unknown_tool | Discovery |
What this is not. These 18 rows are checks that the tool layer behaves as designed with a scripted model. They are not a score. Measuring how often an agent with a real model reaches the right recommendation, over repeated trials and with graders, is evaluation, and the sister site has its own end-to-end lab for it: the Eval Engineering insurance lab. UW-010 exists here only to say so.
Run everything locally#
You need Python 3 and nothing else; the commands below use /opt/homebrew/bin/python3 because that is where it lives on the machine these outputs came from. Java is optional. The runner rewrites out/, which git ignores; the outputs on this page came from a copy of the folder, as below.
cp -R examples/uw-tools /tmp/uw-tools && cd /tmp/uw-tools
/opt/homebrew/bin/python3 run_cases.py # all 18 cases; writes out/; exit 0 if all match
/opt/homebrew/bin/python3 run_cases.py --case UW-004 --verbose
/opt/homebrew/bin/python3 -m unittest test_tools -v # 62 tests
/opt/homebrew/bin/python3 mcp_client.py # an MCP-style session over stdio, printed as JSON
/opt/homebrew/bin/python3 tools.py # the catalog as a table
cd java && java TypedTool.java # the typed Java handler (JDK needed)
| What you run | What you get | What it proves |
|---|---|---|
run_cases.py | The table above and out/<case>/ with result.json, trace.json, messages.json, rows.json, audit.json | Every case matches its expected block |
run_cases.py --case UW-004 --verbose | Each check with expected and actual, and the final answer | Why a case passes or fails |
unittest test_tools | 62 tests, OK | Validator, adapter, idempotency, retries, breaker, loop, trace, MCP |
mcp_client.py | A JSON transcript of 12 requests with their direction | The same adapter behind a teaching subset of MCP 2026-07-28 over stdio |
java TypedTool.java | Nine payloads and the stage that stopped each | A typed Java handler returning the same envelope |
The output is deterministic: run it twice, compare the files, and they are equal byte for byte. The MCP server is a teaching subset written from the specification, not the official SDK and not conformance tested; MCP architecture reads its transcript message by message.
In a coding agent. The same end-to-end shape fits a coding agent. Its catalog has reads (read_file,search_code,run_tests) and writes (edit_file,git_push,open_pull_request); its adapter is the harness's permission layer; its idempotency question is a push whose reply was lost; its trace shows whetherrun_testsran after the last edit; and its human decision is a reviewer approving the pull request, through the code host's own code path, which no tool of the agent can reach. Harness Engineering covers that runtime.
From the example to a deployment: what is missingReal providers, a real store, remote MCP with authorization, and a model that varies
The example is complete as a tool layer and deliberately incomplete as a system. Five gaps separate it from a deployment, and each has a home in this series.
The providers are stand-ins in one process. A real roof-data provider is an HTTP API with credentials the model must never see, latency you measure to set the 2000 ms timeout, and its own failure modes. Implementation shows where the wrapper and the secret live.
The store is a list in memory. A real recommendations table needs the idempotency key under a unique constraint, so that two application instances retrying the same write cannot both insert; the check-then-insert of the example is safe only because one process runs one call at a time.
The MCP server speaks stdio and takes its caller from an environment variable. A remote server speaks Streamable HTTP and learns its caller from an OAuth access token, with the scope check against the token's scopes. The example does not implement that; MCP transports describes it from the specification.
The model is scripted. A real model varies run to run, may request tools in a different order, and may not correct an argument after one error. Every test on the tool layer still holds; what changes is that the end-to-end behavior becomes a distribution, which is evaluation's subject.
And the cockpit is one function. A real cockpit authenticates the underwriter, checks that the person's role may decide this class of risk (R-HUMAN-01 says "an authorized underwriter"), and writes the decision with the person's identity from the session, never from a field the agent could fill.
Why the decision path must be separate in the database tooA missing tool is a strong control; separate credentials make it a structural one
The example enforces "no tool decides" in three ways: no such tool is in the registry, a test fails if one is added, and the adapter answers unknown_tool to a request for the cockpit's function by name. All three live in the application. A Principal engineer asks what happens when the application has a bug.
The structural answer is separate credentials. The adapter's connection to the database should be able to insert into recommendations and nothing else; it should have no privilege on decisions at all. The cockpit's connection writes decisions and reads recommendations. Then a confused-deputy bug in the adapter, or an injected instruction that somehow reaches a tool with a raw query, cannot write a decision, because the database refuses. That is least privilege applied one layer down, where the code cannot argue with it.
The same thinking applies to the audit log. decision_recorded with actor_type: "human" is only trustworthy if the agent's credentials cannot write that event type. Append-only storage, separate writers per event class, and the person's identity taken from the authenticated session, not from the request body, turn the audit log from a diary into evidence.
Reading the out/ folder after an incidentFive files per run answer five different questions; know which one to open first
The runner writes five files per case, and they map to the questions people ask after something goes wrong.
result.json answers "what did it decide, and did it match?": the final answer, the checks with expected and actual, and per call the stage where it stopped and its attempts. Open it first.
messages.json answers "what did the model see?": every message in order, with every tool result exactly as the model read it. If a recommendation is strange, the cause is usually visible here, in a result the model misread or a message that said too little.
trace.json answers "what happened, when, and how long did it take?": the run span, the model turns, a span per tool call and a child per attempt, with events such as validation_failed, authorization_refused and the waits between attempts. It also holds what the model never sees, such as the exception type of an internal error.
rows.json answers "what changed in the world?": the recommendations and decisions rows. A duplicate shows up here and nowhere else.
audit.json answers "who tried to do what they may not?": refusals with the required scope and reason, the recorded recommendation, and the person's decision. In production this file is the one compliance asks for, and the one that must be kept longest.
Versions everywhere: policy, tool, and protocolEvery stored row should say which rules, which tool and which schema produced it
UW-007 shows the policy side: a similar past case was approved under 2025.4, when the roof limit was 30 years, and the current 2026.2 rule wins. The recommendation carries policy_version: "2026.2", so a reviewer reading it next year knows which rules it applied.
Tools change too. If the roof-data provider changes its assessment method, a 22 recorded today and a 22 recorded next year are different kinds of evidence. Store the tool's version on the span and next to the evidence id in the row. Advanced treats tool versioning.
And the protocol changes. The example's MCP server serves revision 2026-07-28 only and rejects the initialize handshake of 2025-11-25, naming the versions it supports. A deployment usually needs a period where both are served. Whatever the policy, record the protocol version of each session in the trace, so that a behavior change can be tied to the day a client upgraded.
Checklist for Monday#
| Check | In the example |
|---|---|
| Every tool has a scope, and only one caller holds the write scope | uw.recommend: underwriting-agent only |
| The catalog has no tool that decides, and a test keeps it that way | test_no_tool_decides_or_changes_a_submission |
| Every request passes lookup, validation and authorization before anything runs | execute_detailed steps 1 to 3; UW-T01, UW-T02, UW-T03 |
| The write carries an application-chosen idempotency key | SUB-1004:run-7f3a:recommendation; UW-T04 |
| Failures reach the model as envelopes with a code it can act on | UW-006 ends in request_info, missing roof_assessment |
| Every run leaves one trace, and required tools are checked on it | UW-008 |
| The person's decision is written by the cockpit with the person's role | record_underwriter_decision; UW-009 |
| Every case has an expected block, and the runner exits non-zero on a mismatch | run_cases.py: 18/18 |
In one breath#
A broker's email becomes a submission; the model proposes tool calls; the adapter looks each one up, validates it, authorizes it, runs it with a timeout, retries what may pass, keeps the one write to one row with an idempotency key, and shapes every outcome into an envelope the model can act on. Every step lands in one trace per run, every refusal in an audit log, and the recommendation waits in the cockpit. The model recommends; a person decides, and the decision is written by the cockpit's own code with the person's role, through a path no tool can reach.
This is the end of the site. You can now build a tool layer that a model may use and a person can trust: from one function, through schemas, results, discovery and MCP, to safe, reliable and tested execution. What remains unsolved is outside this site's question: how good the agent is, which Eval Engineering measures, and the runtime around it, which Harness Engineering builds. To start again from the first primitive, go back to the Start page; for the sources, the sister sites and how the facts were checked, read About.