Criteria-Based Grading for Subjective Agent Output

The problem

“Is this design beautiful?” is hard to answer consistently. LLMs default to safe, predictable outputs that are technically functional but unremarkable. Without intervention, Claude gravitates toward template layouts, library defaults, and generic patterns.

The solution: decomposed criteria

Replace the vague question with concrete dimensions. For frontend design, Anthropic’s harness used four criteria:

The key insight: weight dimensions where the model is weakest. Design quality and originality were weighted more heavily than craft and functionality because the model already scored well on the technical dimensions. The criteria explicitly penalized generic “AI slop” patterns (purple gradients over white cards, unmodified stock components), pushing the model toward aesthetic risk-taking.

Source: Harness design for long-running application development (Prithvi Rajasekaran, Anthropic, March 2026)

Calibration through few-shot examples

The evaluator was calibrated using few-shot examples with detailed score breakdowns. This served two purposes:

1. Aligned the evaluator’s judgment with the designer’s preferences
2. Reduced score drift across iterations

Without calibration, the evaluator’s grades wander. With it, the scoring becomes consistent enough to drive meaningful iteration.

Criteria as steering

The wording of criteria directly shaped output character in ways that were not fully anticipated. Including phrases like “the best designs are museum quality” pushed designs toward a particular visual convergence. This suggests criteria are not just measurement instruments — they are prompts that steer the generator’s creative direction.

Even on the first iteration, outputs with criteria were noticeably better than a no-prompting baseline. The criteria steered the model away from generic defaults before any evaluator feedback occurred.

Adapting criteria across domains

For full-stack development, the same pattern adapted to different dimensions:

• Product depth — feature richness and completeness
• Functionality — does it actually work when tested?
• Visual design — coherent design language
• Code quality — maintainability and correctness

Each criterion had a hard threshold. If any one fell below it, the sprint failed and the generator received detailed feedback on the specific failure. The evaluator used browser automation (Playwright) to interact with the live application, testing UI features, API endpoints, and database states.

Tuning the evaluator

Getting an evaluator to perform well required significant iteration. Out of the box, Claude as a QA agent:

• Identified legitimate issues then talked itself into deciding they weren’t a big deal
• Tested superficially rather than probing edge cases
• Approved work that a human would flag

The tuning loop: read evaluator logs, find examples where its judgment diverged from human judgment, update the evaluator’s prompt to address those divergences. Several rounds of this were needed before the evaluator graded reasonably.

Counter-argument: keep each judge binary

Teresa Torres arrives at the opposite output shape from the same goal of keeping an LLM grader reliable (Product Talk). Her rule is to define criteria so the judge returns true or false, nothing else. The reasoning is that a judge is trustworthy only when its task is simpler than the task given to the model it grades. A weighted multi-dimensional rubric is not a simpler task; it is a harder one, and it asks the judge for a calibrated magnitude. Whether that is the wrong thing to ask depends on what the grader is for. Torres is measuring how often a specific error occurs in a shipped product, where a rate is the answer and a magnitude is noise. The evaluator described above is steering a generator inside an iteration loop, where a graded magnitude is the signal the generator needs. The two designs are optimizing different things, which is part of why they diverge.

Torres names four conditions for a working judge:

1. Give the judge a much simpler task than the original model received.
2. Define criteria so the response is binary.
3. Align the judge with human judgment.
4. Accept that the judge makes its own mistakes, and account for that error rate when reporting results.

Condition 3 is the tuning loop described above. Condition 2 is partly satisfied already: each criterion carries a hard threshold, so at the criterion level the verdict is pass or fail, and what stays continuous is the score feeding that threshold. Conditions 1 and 4 are genuinely unaddressed. Condition 4 is the sharpest omission: a rubric score reported without the grader’s own error rate reads as more precise than it is.

The two positions are reconcilable. A rubric of N weighted criteria decomposes into N binary judges, each seeing only its own criterion, with weighting applied afterward in code rather than inside the model. That preserves the design lever of weighting while giving each judge the simplest possible task. Torres runs a separate judge per error type for exactly this reason — one for double-barreled questions, one for leading questions, one for whether a suggested question was already answered.

What would settle it: run the same outputs through a single weighted-rubric evaluator and through N binary judges, and compare each against human labels. If the decomposed judges agree with humans more often, the rubric’s convenience is costing accuracy. The failure modes already logged here — talking itself out of legitimate issues, approving work a human would flag — are the ones a single evaluator holding a whole rubric would be expected to show.

A harder bound sits underneath both approaches. Microsoft’s DELEGATE-52 work found that generic similarity measures, LLM-as-a-judge included, capture at most 25% of the variance that domain-specific parsers detect (see Backtranslation as Benchmark Design). That is one benchmark on invertible document-editing tasks, not a general law, but it is a reason to ask of any criterion whether a parser could carry it before either judge design is asked to.

Related Notes

• GAN-Inspired Agent Architecture - Generator Evaluator Loops — the architectural pattern these criteria serve
• Agent Harnesses — harness design is where evaluation criteria live
• AXI Principles - Agent-Ergonomic Interface Design — another approach to principled agent interface design, focused on tool interfaces rather than output quality
• Evaluation Design Lifecycle - From Purpose to Metrics — the systematic process (Phase 4) that produces criteria like these from stakeholder needs
• Cost Per Right Answer as the Metric for Agentic Data Pipelines — supplies the missing denominator; cost per right answer requires deciding what counts as right when no lookup can verify it
• Verifiable Domains Do Not Prove General Capability — this is the attempt to manufacture the verifiability that domain lacks
• Let the Error Shape Choose the Eval Type — the prior question: whether this error needs a judge at all, or a cheaper deterministic check
• AI Evals - A Hands-On Guide for Product Teams — the source of the binary-judge counter-argument above
• Code Is Verifiable in the Small and Unverifiable in the Large — bounds rubric-based grading where the outcome appears only through future changes