Notes

Planted errors are the easy case

November 2026

I wanted to know whether a frontier model pushes back or stops when the plan in front of it contains an error. To test this, I took a sound plan, reversed a prediction or a sign, and measured whether the assistant pushed back or continued. I did this for 40 plans.

Without any special instruction, gpt-6.1-sol named the planted error in 39 of the 40 cases, often just mitigating the problem by using careful language (hedging), and then went on to the next step anyway, every time. So I added a rule to its instructions: do not build on a statement you believe to be false. With the rule, it stopped on all 40 plans. Sounds good.

Then I tried the same rule on 148 real plans, each with a problem that the judges had found and that nobody had planted.

As we can see, it only stopped on 7 of the 148 plans with a real problem!

The planted errors were textbook-level. Take something, reverse the prediction or the sign. Obvious problems. A model can show them to be false from the text alone. The real ones needed knowledge of the field, or an estimate of whether a calculation is feasible at all.

I still plant errors. They are cheap, and the ground truth is certain. But a test suite built only from them will overstate any fix. So plant with care.

What you ask a judge decides what it finds

October 2026

I constructed a panel of three model judges and asked them to read research plans that gpt-6.1-sol had written for computational studies in physics. Asked to list every false statement, the panel found a serious error in at most 3% of the plans, whichever way I combined the votes. Sounds good.

Then I changed the question. Each judge also had to say whether every planned step can deliver what the plan needs from it. Now the panel found a serious problem in 30% of the plans. Sounds bad.

Why such a discrepancy? The answer is simple. It is easier to mitigate the problem by changing the language to be more careful than to fix the underlying problem (which might be nontrivial).

Almost all of the problems were related to feasibility. Here is what a typical case looks like. The request asks for a simulation that is known to be far too expensive at the requested temperature. The model does not repeat the request blindly. It writes a careful plan, adds a sentence such as “the sign problem may prevent a conclusion,” and keeps the infeasible run in the plan. Nothing in the plan is false. The run still cannot work.

I want to be fair to the model. A hedged plan is better than a plainly wrong one. But the hedge changes how the plan reads more than what it will do, and a judge that only looks for false statements rewards exactly that. In my work, I also observed that when the same model judged its own plans, it defended the hedge every time (it upheld none of 21 objections), while two judges from another family upheld 11 of the 21.

Now, before I trust a low error rate, I read the question the judge was asked. And I always use cross-family judges (or a panel of them). The cost of verification is usually lower than the cost of generation (and each problem caught can save a lot in the cost of a prevented run).

For an LLM judge, size may matter more than family

September 2026

When I use a language model to judge the output of another model, I add a second judge from a different vendor. The usual argument is that models from different families make different mistakes. However, I wanted to see how much of the disagreement between judges comes from the family versus, for example, the size of the model.

I took a small and a large model from each of two families: gpt-6-luna and gpt-6.1-sol from OpenAI, Claude Haiku 5.5 and Claude Opus 5.5 from Anthropic. All four judged the same 972 pairs of texts, with the same question for each pair: which of the two uses more jargon, as seen by a scientist from another subfield? A judge could also answer that it cannot tell.

You can see that the two large judges agreed with each other most often, although they come from different vendors. A large and a small judge from the same family agreed no more often than two judges that had nothing in common. The two small judges agreed on 76% of the pairs, no higher than a small and a large judge from different families. It looks as if each small judge is noisy in its own way.

So my working hypothesis is that two large models from different families are now more alike than a large and a small model from the same family. It is a hypothesis, not a claim: this is one task and four judges. An alternative explanation is that the two large judges can simply find some problems that the smaller ones cannot, and this makes them seemingly more similar. As always in science, each experiment creates more questions. I hope to run a follow-up study soon.

The family still matters for other things. I also observed that judges from different families are calibrated differently (what one calls a serious error, the other lets pass), that some say much more than others, and that one judge was more lenient with text written by a model of its own family, while the other was not. This creates a challenge. If your application can generate text with models from different families, you need to make sure that the judges are calibrated to the same standard. Otherwise, in one of the configurations, you might get a much higher error rate than in the other, not reflecting the actual quality of the output.