Independent evals after Anthropic let METR in

On 15 October Anthropic gave a METR team employee-like access to how it judges its own models. Six people, company hardware, office space, and tools close to what Anthropic’s risk staff already use. METR will publish. Anthropic cannot veto the write-up. It can only redact a short, named list of things: customer data, legal privilege, some architecture detail.

That is a serious step for a frontier lab. It is also a useful picture of what “independent eval” actually is, stripped of the press language.

We already run offline evals for Cabin and for Mo inside QuesMo. We are not METR, and we do not sit inside Anthropic. The useful part for a small team is cheaper than a resident examiner. It is still stricter than a founder reading their own traces.

Key takeaways

• Independence means the reviewer is not paid to pass you, and can publish without your veto.
• Access means prompts, failures and how the model is used, not a polished demo.
• A small team cannot copy METR’s setup. It can copy the three tests: independence, access, a written result.
• If the same person who wrote the prompt also scores the eval, you do not have an outside check.
• Start with real conversations that already failed, not with a benchmark you found online.

What Anthropic actually did

The announcement lists four things an evaluator needs: expertise, independence (no equity, no dependence on the company), enough access to see training and safeguards, and freedom to publish.

METR is at the deepest of three tiers Anthropic describes. Lower tiers get narrower access and, at the first tier, publication only with Anthropic’s approval. SecureBio is reviewing a chapter of a Risk Report. Accenture is in an earlier partnership. Hugging Face is in talks.

The bank-examiner analogy is Anthropic’s, not ours. Banks know more about their own books than outsiders do, so someone sits inside. The claim is that frontier labs now need the same kind of eyes.

Fair criticism still applies. METR is small. The rest of METR is siloed from this team. Redactions exist. Other partnerships are early. The test is whether the published reviews change what ships, not whether the office badges were issued.

What this is not, for a product team

You do not need six people in your office with company laptops. You do not need a Risk Report. You do not need to evaluate chemical-weapons chapters.

You do need a check that is not you grading your own homework.

We wrote the practical version in LLM evals for small teams: a fixed set of real conversations, pass or fail checks, and a rule that nothing ships until it passes. That post is about how to build the suite. This one is about who is allowed to say it passed.

A small-team version of the same three tests

1. Independence. The person who scores the eval did not write the prompt and does not get paid more if you ship. A colleague on another product is enough. A contractor with a written brief is enough. The founder replaying their favourite chats is not.

2. Access. They see the prompts, the routing, the failure logs, and a sample of real or realistic conversations. A recorded demo is not access. If you strip the awkward turns before they look, you have already graded the test.

3. A written result they own. Findings go down in a note you cannot quietly edit into a pass. You can disagree in public. You cannot make the original disappear. That is the small version of “cannot veto publication.”

For Cabin, that means someone who did not write the reply prompt reading whether answers stay short, take a side, match the user’s language, and avoid inventing memory. For a client LLM product, it is the same shape: replay, score, write it down.

Where teams fake this

Three habits look like evals and are not:

• The same engineer who changed the prompt also marks the suite. They already know which cases they fixed.
• The suite is only the cases that already pass. Failures got moved to a “later” list.
• An outside vendor runs a generic benchmark, then you rewrite their slide until it says “ready.”

Anthropic’s first tier, where they choose the access and approve publication, is close to that last habit. They are honest that it is a weaker tier. If your “audit” cannot survive a sentence you dislike, it is marketing.

What to do this week

You do not need a lab visit. You need one afternoon:

1. Pick 20 conversations that already went wrong. Not invented edge cases. Real ones.
2. Write a pass or fail for each. Length, invented facts, language, a missed safety line. Binary.
3. Hand the set to someone who did not write the prompts. They run it. They write what failed.
4. Nothing ships until those failures are gone or explicitly accepted.

If you want that read done from outside, that is the LLM audit we sell: cost, quality, evals and reliability, in writing, ranked by what to fix first.

The bottom line

Anthropic let a nonprofit sit inside and promised it could publish. The part that travels is not the office space. It is this: the people who build the model should not be the only people who get to say it is fine.

A small team can do a cheaper version tomorrow. Someone else, real access, a note you cannot rewrite. That is the whole standard.

FAQ

What did Anthropic announce on 15 October 2026?

It gave a small METR team employee-like access to internal tools and risk work, with the right to publish findings. Anthropic cannot veto publication, only redact a short list of sensitive categories.

Who is METR?

A nonprofit that evaluates frontier AI models. Anthropic had already used METR to review public Risk Reports. This step is deeper: people sitting inside, with hardware and access close to Anthropic's own risk staff.

Does a small product team need something like METR?

Not a resident lab. You do need someone who did not write the prompts to read the evals, replay real conversations, and say in writing what still fails. If they cannot publish the result to you without your editing it first, it is not independent.

What should I look for in an outside eval?

Three things: they are not paid to say you passed, they can see prompts and failure logs rather than a demo, and they can write the findings before you rewrite them.