What compounds in an agent system

Anyone who sets up AI agents to do real work has to pick a model to run them on, and the price difference between models is large enough that the choice matters. The natural assumption is that the model is the intelligence, the intelligence does the work, and so the system will be about as good as its model, which makes the correct move buying the most capable model available and worrying about cost later. We build tools for directing many agents at once, and we run our own projects on those tools. We held this assumption too: for months we scheduled work around frontier-model quality.

A blind comparison

A comparison we ran in one of our own projects changed our position. The project produces a crafted deliverable, and for weeks its quality had been carried by an accumulating layer of tooling rather than by any particular model: a domain-specific language that encodes the craft decisions, written checks that automatically fail a draft when it violates a rule, and a library of graded past examples that every new run reads before it starts. On top of this layer we staged a controlled comparison. The briefs were identical and the tooling was identical, with a frontier model producing half of the drafts and a mid-tier model at a fraction of the price producing the other half. The drafts were graded blind, so the reviewer never knew which model produced which draft.

The mid-tier model matched the frontier model. On every dimension the tooling had encoded, the drafts were indistinguishable, and drafts from both models passed the automated checks at the same rate. The one place the frontier model still won was a fine-grained interaction sequence the tooling had not yet encoded, and that exception located the craft precisely: the quality lived in the machinery we had written down, and the model only mattered where the machinery stopped.

Rented and owned

The result sorts an agent system into two kinds of parts. The model, together with the context assembled for one call, is rented: it is paid for per call, it is supplied by a handful of vendors at prices that keep falling, and it remembers nothing from one call to the next. Everything else persists between calls and is owned: the accounts and credentials the system holds, the append-only history of everything it has ever done, the graded judgments recorded on its past work, the craft encoded as rules and checks, and the standing it has built with the people it serves.

Models are interchangeable for a structural reason. Everyone can rent the same models at the same prices, so whatever advantage a model confers, it confers on your competitors in the same month. The gap between the best model and the tier below it also keeps narrowing, and it narrows fastest exactly where tooling can carry the difference, which is what the comparison measured.

An advantage that everyone can buy is a cost, not a moat.

The owned parts have the opposite property: nobody else can rent your event history, your recorded judgments, or your standing with customers, because those exist only inside your system and accumulate only through its operation.

Two questions decide which side a given part of your system falls on:

• If the current model disappeared tomorrow, would the system still have its identity, its resources, its history, its obligations, and its reputation?
• Could a different model take over the same state and continue the same work without customers noticing?

When both answers are yes, model choice stops being an identity decision and becomes a pricing decision, and every model release becomes good news, because the improvement arrives into a system that keeps everything it has learned.

Building for the swap

Most of the market is currently building the other way around. Products sell the model's intelligence, treat the money and the records as plumbing, and store their accumulated judgment in prompts and fine-tunes that die with the model that carried them. If the comparison result generalizes, and our operating experience says it does, that inventory is worth less than it looks, because the part being sold is the part that commoditizes.

For anyone building agent systems, the practical consequences are direct:

• Move judgment out of the prompt and into written rules. A prompt lasts one call and evaporates. A rule file that every future run must read persists, and it compounds.
• Record every verdict where the next run can read it. A judgment given once and thrown away has to be paid for again tomorrow. Graded examples with the reasons attached are the cheapest asset an agent system can accumulate.
• Keep the event history append-only and owned. The history is what the next model reads to continue the work, and it is the one record that cannot be reconstructed later.
• Re-run the blind comparison on a schedule. Tooling keeps absorbing craft, so the cheapest model that passes keeps changing. The comparison is how you notice.

When a better model ships, we swap it in, and the system keeps its history, its rules, and its customers. The model is the one part of the system you should plan to replace.