Claude Code now scores plugins against a no-plugin run

The command needs Claude Code v2.1.269 or later. Running claude plugin eval init from the plugin root opens an interactive session in which Claude reads the plugin and asks the author what a good result looks like. It then proposes prompts that should trigger the plugin and prompts that should not, designs graders for each, pilots them once, and writes one case directory per prompt under an evals folder. Authors who would rather write a case by hand can get a blank template with init --bare.

A case is one request a person might realistically type, plus one or more graders. For each run, Claude Code starts a fresh isolated non-interactive session with only that plugin loaded and lets Claude work until it finishes or hits the case's turn or time limit. Each grader then checks the final reply, the transcript, or a file Claude created. According to the documentation, a case runs three times by default; a run's score is the fraction of graders that passed, and the case's score is the mean across its runs. The pass threshold defaults to 1.0, so a single failing grader sinks the case.

None of that is unusual for an evaluation harness. The split comes next. The same case is run again the same number of times with no plugin loaded at all, and the two scores are set side by side. The summary table prints WITH and W/OUT, and the difference between them appears as a delta.

The documentation states that "a high score on its own doesn't tell you the plugin helped." If a case scores 1.0 with the plugin and 1.0 without it, the plugin is not what made it pass. It is the same structure as always keeping a placebo group when testing a drug.

In the example table in the documentation, one case scores 1.00 with the plugin, 0.33 without, a delta of +0.67, across six runs. That is six runs for a single case, and sixty for ten cases. The first constraint on anyone using this tool is that every one of those runs is a real model call billed to the author's account. Anthropic states that plainly at the top of the page, and pairs it with an option to cap the run's cost in dollars and an option to switch the baseline off. Switching the baseline off halves the cost, and removes the delta with it.

There are six grader types. Four of them cost nothing extra because they judge from the transcript and files alone: matching a regular expression against a reply or a file's contents, counting how many times a particular tool was called, checking the order of two tool calls, and confirming a file was created. The other two call a judge model. Of those, the llm grader passes only when at least two of three votes come back as a pass. There are no custom-code graders.

Anthropic's prediction of the first result authors will see is the interesting part. On a first run, the most common outcome is a delta near zero with the grader that checks whether the skill fired coming back as a failure. That means Claude is not choosing that skill from naturally phrased requests, and the fix is not the body of the skill but its description. METAL has reported on research finding that procedure rather than knowledge drove agent skill performance, and this tool measures whether that procedure gets called at all.

To keep the baseline honest, Anthropic deliberately excludes some graders from the score. A grader that checks whether a skill fired can never pass without the plugin, so counting it would push the no-plugin score toward zero and inflate the delta. Those graders are therefore dropped from the score in both arms and kept in the with-plugin arm as pass or fail indicators only. Graders an author marks as with-only are handled the same way. The design ties the harness's hands so it cannot manufacture numbers in its own favor.

The run environment is far emptier than authors might expect. Each run gets a throwaway home directory, working directory, and Claude Code configuration, and starts with user settings, hooks, CLAUDE.md files, MCP servers, other plugins, memory, and skills all absent. Most shell environment variables are withheld too, with only an allowlist and variables prefixed EVAL_ getting through. The Artifact tool is off, and the case definitions themselves cannot be read by the agent. It is like clearing out a kitchen entirely and setting out only the ingredients for this one dish.

The permission design is the next thing an engineer should look at. Runs never stop to ask. Heavy-handed tools such as Bash, Write, Edit, WebFetch, and WebSearch are removed from the session outright unless granted, so Claude cannot call them at all. A tool a case asked for but the author did not grant is printed to standard error as not granted. MCP servers are not actually started by default; instead one markdown file per tool sits in a per-server folder holding a canned response. Writing the expected input into that file means the run aborts with a score of zero the moment the plugin sends something else, and the reason is recorded.

For continuous integration the documentation recommends lowering the threshold below 1.0 and pinning both the model under test and the judge model, so a model rollout is not mistaken for a plugin regression. The exit codes are split out as well: 0 means everything passed, 1 means a case fell below the threshold or a case file failed to load, 2 means a partial run that hit the cost ceiling or had its credential rejected, 130 means interrupted, and 143 means terminated, such as by a CI timeout. Failures while writing or publishing the HTML report never change the exit code.

The result is left behind as a single self-contained HTML file. It makes no external requests, so it can be attached to a CI job or opened straight off disk, and the top of it states in one line how far the plugin's effect sits above the baseline and how many cases improved, stayed flat, or regressed. A case with a negative delta gets a red left edge, so regressions catch the eye on a scroll. Authors signed in with a claude.ai subscription also get the same report published as a private artifact, with the address printed alongside.

Anthropic also wrote down what the tool cannot measure. The isolation limits what the agent under test can reach rather than fencing in the plugin's own code, and a suite that passes says nothing about whether the plugin is safe. There is advice to suspect the judge first, too: when a skill fires but the delta is negative, look before anything else at whether a small judge model marked a correct answer wrong for being formatted differently. That the measuring instrument and the thing being measured come from the same model family is a cost this approach carries.

The troubleshooting section spells out where control sits as well. A message saying the command is unavailable means Anthropic has switched the feature off server-side, and the documentation states that nothing on the author's own machine brings it back. Teams gating CI on a threshold need to factor in that the instrument itself sits behind a switch the company holds.

METAL confirmed an item in the documentation noting that concurrency can be raised from 1 to 8, with the caveat, spelled out on the page, that runs share the account's rate limit, so this shortens wall-clock time rather than raising throughput. Putting a single plugin behind a CI gate means working out the cost and the time in advance. METAL has covered the flow for bundling a team's setup into one plugin and shipping it, and until now there was no way to tell whether such a bundle was earning its place beyond a hunch.

The plugin ecosystem grows up a notch with this command. Once the score from a run with the plugin switched off sits beside the score authors used to advertise, skills with impressive descriptions separate from skills that change outcomes. The result Anthropic chose to forecast in its own documentation, a delta of zero, points that way. A good share of the plugins shipped today work as well without them as with them, and now their authors will be the first to know it.