Encoding judgment as rules

AI agents produce drafts faster than any person can read them. That is the point of using them, and it creates the obvious bottleneck: the quality of anything crafted depends on judgment, the judgment lives in a person, and the person does not scale. Hiring more reviewers reproduces the problem the agents were supposed to remove, because now the reviewers' judgment has to be aligned too.

The way out starts with an observation anyone who reviews agent output will recognize: the corrections repeat. A person editing agent drafts rejects the same sentence shapes, the same layout mistakes, the same staging errors, week after week, and each rejection costs the same attention as the first one did, because nothing carries the correction forward to the next draft. What we learned building our systems is that the correction can be carried forward, if it is written down in a form every future run must read.

From verdict to rule

We run this as an explicit pipeline. A verdict is a binary judgment on one piece of work, with the reason stated: this draft fails, because the opening buries the strongest material. When the same verdict recurs, it is generalized into a rule, meaning a written instruction in a file that every future run must read before it starts. A rule carries annotated examples of passing and failing work, because agents follow examples more reliably than they follow abstract instructions; in practice the example is the specification, and a rule without one gets interpreted differently by every run that reads it. Where a rule is machine-checkable it additionally becomes a check, an automated audit that fails a draft on violation with no human present, so the rule is enforced at the moment of production instead of at review.

The strongest form is a rule that becomes a structural fact: if a mistake was possible because a value was set by hand, the fix is to make the value derived, and then the mistake cannot be expressed at all, no matter how careless the run. The final property is inheritance. A run starting today reads every rule derived before it, and is checked by every audit written before it, so nothing has to be retaught.

A rule derived in three rounds

The clearest worked example from our own operation is a writing register. One of our projects needed its public pages written in plain documentation prose, and the agents drafting them kept producing something else: dramatic fragments, metaphors, sentences arranged for rhythm. The rule took three rounds to derive. In the first round a person read the drafted copy and killed sentences one at a time, stating the reason each one died: this sentence narrates the page instead of describing the product, this one is a metaphor doing no work, this one exists for its sound. In the second round those verdicts were compiled into a rule file, built around a short corpus of prose in the target register for the runs to study, followed by a banned-pattern list with the verbatim verdicts attached to each entry. Drafts written under that rule were better and still failed in a subtler way, because the sentences were now compressed and manicured, performing plainness instead of being plain, and the verdict on that round became the rule's second revision.

In the third round, drafts began passing on first review. The rule has cost nothing to apply since, and every page the project publishes inherits it.

Small judgments are recorded the same way as large ones. A reviewer once rejected a narration line because it counted the product's features, and the whole correction became one written sentence: never count features in a narration. The rule has held through every narration since, and nobody has had to give that verdict twice. Writing it down took under a minute, which is the general economics of the pipeline: a verdict is expensive because it needs a person's attention on real work, and everything downstream of it, the rule, the examples, the check, is clerical. The judgment is paid for once, and applying it afterward costs close to nothing.

Where the human ends up

As rules accumulate, the person's role changes shape, and the change follows a fixed gradient. At the start the person approves each artifact before it ships, which is the correct posture for any new category of work, because the rules for it do not exist yet. When drafts start passing first review reliably, approval moves to batches: a whole wave reviewed at once. When batches pass, the person approves policies, meaning they approve the rules and the rules approve the work. The end state is review by exception, where only the low-confidence, unusual, or expensive cases reach a person at all. Each graduation is earned by the recorded evidence of the stage before it, and a category that starts failing again is demoted back down the same gradient.

What remains for the human concentrates into two motions: approving, which is cheap, and correcting, which is the expensive judgment work. The discipline that makes the whole system compound is that every correction lands in a rule file in the same session it is given, because a correction that stays in the person's head applies to one draft, and a correction that lands in a rule applies to every draft after it.

Judgment is usually described as the scarce input to AI systems, and usually as a property of gifted individuals. Our experience building these systems is that most of a person's judgment can be written down, and that the written form is what does the governing. The verdicts a person gives on real work, generalized into rules and enforced by checks, carry that judgment into thousands of runs the person never sees, and every review that produces a new correction makes the rules more complete.