2026-08-19
Designing evaluator loops that improve output instead of chasing their own tails (2026-08-19)
An evaluator-optimizer loop is only as good as its evaluator. This guide covers criteria as checklists, independent judges, calibration against real outputs, round caps, and the failure modes — score inflation, judge drift, and infinite polish — that turn critique loops into token furnaces.
← Back to BlogThe Evaluator-Optimizer pattern is the one agent teams reach for whenever quality is subjective: a generator produces a draft, an evaluator scores it against criteria, and the loop repeats until the bar is cleared. The shape is simple, which is exactly why teams are surprised when it fails. An evaluator loop fails in predictable ways, and every one of those failures is a design decision that was skipped. Here are the decisions that matter.
Criteria first, prompts second. The most common mistake is writing the evaluator prompt as prose — 'evaluate whether the answer is accurate, complete, and helpful' — and wondering why scores drift between rounds. The fix is to turn criteria into a checklist: numbered items, each with a binary or narrowly scored answer that a reader could verify by looking at the output alone. 'Every claim cites a source from the retrieved context' is a check. 'Helpful' is a vibe. A checklist evaluator is cheap to audit — you can read the trace and see exactly which item failed — and it stays stable across rounds because it is judging facts, not forming opinions.
Separate the judge from the generator. When the same model call both writes the draft and scores it, the score is a self-report, and self-reports are optimistic. Use a second node for evaluation, and whenever possible a different model for the judge than for the generator — the two fail differently, which is precisely what you want from a critic. In LoopCraft this is two nodes on the canvas; the cost of the extra call is almost always smaller than the cost of one wasted polish round.
Calibrate before you trust. Pick five real outputs whose quality you already know — two good, two bad, one borderline — and run the evaluator over them. If it cannot rank them correctly, it cannot steer your loop, and no amount of threshold tuning fixes a miscalibrated judge. Recalibrate whenever you change the generator prompt, the model, or the criteria itself; evaluators go stale quietly.
Cap the rounds, and watch what the cap catches. Three to five rounds is enough for most writing and code-generation tasks; if the score has not crossed the bar by then, another round rarely closes the gap and usually starts to overfit the draft to the judge. Treat every run that ends on the round cap as data: either the bar is too high for this generator, the criteria contradict each other, or the task should have been split before the loop started.
Finally, know the three failure modes by name. Score inflation: the evaluator nods along, passes everything, and the loop exits on round one — usually a sign the criteria are too vague to fail on. Judge drift: the evaluator's standards shift between rounds, so a draft that passed on round two fails on round three without changing — usually a sign the criteria live in prose instead of a checklist. Infinite polish: the score climbs forever without crossing, because each fix introduces a new nitpick — cap the rounds and log the last two score vectors side by side, and the oscillation becomes visible immediately.
The recipe that keeps an evaluator loop honest: checklist criteria, an independent judge, calibration on known outputs, a hard round cap, and a trace you can read. In LoopCraft, wire a Generator node to an Evaluator node, attach the checklist as the evaluator's rubric, and set the round cap and cost budget in the loop settings — the trace then shows every score, every failed item, and every round's cost. Full toolset at https://getloopcraft.com — sapsap@qq.com.
Reading time: 7 min