Systems appreciate, prompts depreciate

Anyone who has worked with a language model has told it to think harder. The model complies in the only way it can, by producing text that sounds like harder thinking: the reply gets longer sentences, more qualifications, and a more deliberate tone. The instruction was interpreted as a request for a style, because a style is the only thing an instruction can buy at the prompt layer. There is no dial inside the model that maps “think harder” to more computation, so the words purchase the register of effort rather than the effort.

The prompt layer has a second structural limit: nothing there persists. A prompt is one forward pass through the model, with no revision loop inside the call and no memory across calls, so a correction given in chat improves the output in front of you, and then the context window closes and the correction is gone.

None of this means prompting is useless. A well-written prompt demonstrably improves output, and whole teams tune prompts for a living. But the patterns they find are workarounds for one model’s specific weaknesses, and the next generation does not have those weaknesses, so the value of a prompt pattern goes to zero on a schedule set by someone else’s release calendar. Prompting is a floor you stand on, not an asset you hold. The dial that “think harder” reaches for does exist, one level up, in the structure the model runs through.

What survives a model release

The durable object in an agent setup is the system: the files, checks, examples, and pipeline structure that exist between calls. Right now that object is concretely a codebase: the repository that holds the rules, the examples, the checks, and the pipeline code, and it persists while the model inside it is swapped per call.

Any capability gain at the model layer therefore multiplies through whatever structure it flows into. A weak structure wastes the gain: the smarter model re-derives context it was never given, repeats failures nobody recorded, and produces variance nobody can review. A strong structure converts the gain: the smarter model reads the same rules, passes the same checks, and clears work the previous model could not. A more intelligent model immediately upgrades a more effective system.

The generic code in a system depreciates too, because a smarter model can regenerate scaffolding and glue from scratch, and it will. What a model cannot regenerate is the record of your selections against reality: which outputs you accepted, which you rejected and for what reason, which checks encode constraints of your actual domain. That part of the codebase is data about judgment rather than code, and it accumulates only through operation.

The four parts

An effective system, in our experience, has four parts, and the parts only work when they are versioned together as one unit.

• A skill file is the written rules of the craft, and every run reads it before generating anything.
• A golden set is a library of past outputs that were accepted, stored with the grades and the reasons, and new work is compared against it.
• A verifier is a set of automated checks that fail a draft when it violates a rule, and it runs without a human present.
• A rejection ledger is the record of failed outputs with the reason each one failed, and it is where the next rule comes from.

Around the four sits the pipeline: staged passes, checks between the stages, and a rule for how many parallel attempts get reduced to one accepted result.

The four pieces reference each other, which is why they version as one unit. A golden set without its verifier drifts, because nothing fails the examples that stop deserving their grades. A verifier without a ledger stops growing, because the record of what failed is where new checks come from. A rule without the example that motivated it gets argued with, by humans and by models alike. A change to any piece updates the others.

When outputs are wrong, the instinct is to argue with them one at a time in chat, and the productive move is one level up: change the rule, the example, or the check that allowed the output, and the whole class of failure closes at once. If there is too much randomness at the prompt layer, move up a level and teach the generative process instead.

Written checks sound like evals, and an eval suite is one of the four pieces. The difference is where corrections land. Most eval practice grades the model and leaves the grader’s reasoning in a spreadsheet or a chat thread. When the four pieces are versioned together, every correction lands in an object that every future run must read, so nothing the operator learns evaporates.

A natural experiment

During a client engagement we had temporary access to a frontier model. Partway through, the access was revoked, and some weeks later it came back. The revocation turned out to be a controlled experiment on where the quality of the work lived.

While the access was gone, the work continued on weaker models, and the effort went into the system. The frontier model’s earlier outputs were graded into a golden set. The patterns that made those outputs good were extracted into written rules. The checks that would have caught the weaker models’ failures were built and wired into the pipeline. When the access returned, the same model, no smarter than before, cleared the entire accumulated backlog almost immediately, at roughly thirty percent of the usage it had needed earlier, because it now flowed through the pipeline that had been built in its absence.

The model was identical before and after, so the only variable that changed was the system. Before the revocation, a strong model in a thin system produced good outputs slowly, each one hand-steered by the operator. After it, the same model in a compiled system finished the backlog in one pass, and prompting effort went down rather than up, because the pipeline reads the rules so the operator does not restate them. What survived the revocation was the golden set, the extracted rules, and the checks. The individual prompts and the recipes behind individual outputs did not survive, and nothing was lost with them.

The system could only be built because one real deliverable had already shipped: the graded examples had to come from work that had made contact with a real recipient, because a golden set graded against imagined standards records taste nobody has tested. A separate experiment points the same way: in a blind comparison we ran on the same class of system, a mid-tier model matched a frontier model on every dimension the tooling had encoded, and only on those.

Compiling your own judgment

The corrections a person gives in chat are the most valuable data they produce and the least preserved, because by default they die with the context window. Telling a model in chat to stop using a phrase fixes the draft in front of you and nothing else. Writing the same instruction into the skill file that every run must read, and storing the rejected draft in the ledger with the reason attached, fixes every draft produced after that day. The correction is identical in both cases. What differs is its half-life, which goes from one call to permanent when the correction lands in the versioned unit instead of the conversation.

Run over months, the loop amounts to compiling your own judgment into an executable form. Each rejection enters the ledger with its reason, each correction becomes a rule, each accepted output joins the golden set, and the compiled judgment then runs in every parallel agent at once, on every future model, with nobody present.

The frontier is rented. What you distill from it is owned.

The obvious worry is that compiling your judgment automates you away. What gets compiled is the settled part of the judgment, the rules you no longer want to restate, and compiling them moves the review upward: the person spends their time only on what the system cannot yet check, and each session at that level produces the next rule. The operator’s position in the loop is permanent even as its altitude rises, and the skill practiced there shifts from producing outputs to producing the loop.

Scaling with intelligence

A practical test for what you have built is to drive more compute into it and watch what comes out. An effective system converts more compute into more finished, accepted work, a property we call compute market fit. A prompt-only workflow fails the test: more runs produce more variance to hand-review rather than more accepted output, because nothing converges the candidates. Verifiers and golden sets are what turn parallel generation from a review burden into a search: candidates that fail the checks are discarded mechanically, candidates that pass are compared against the graded examples, and the human sees only the survivors. The property that absorbs a smarter model is the same property that absorbs more compute from the current one, because both are more intelligence flowing in, and the system is what turns intelligence flowing in into work coming out.

Model releases will keep arriving, on a schedule nobody downstream controls. Each release erases the value of the prompt patterns tuned to the previous generation, and each release multiplies through whatever examples, rules, and checks have accumulated by the time it lands. The work that pays is building the codebase the next model will flow through.