Claude Code Plugin Evals: 6 Grader Types and a CI Gate for Skills

Anthropic has published a plugin evals workflow for Claude Code, available in version 2.1.269 and later. The new "claude plugin eval" command runs a plugin against realistic prompts, grades Claude's output using one of six grader types, and compares the result against a baseline run where the plugin is not loaded. The score difference between the two arms, called delta, is the key metric: it tells developers whether the plugin is actually doing the work or whether Claude would have passed the test anyway.

What happened

Detail	Value
Minimum version	Claude Code v2.1.269
Grader types	6 total (4 free, 2 billed)
Default turns per case	10
Default timeout per case	300 seconds
Example case result	WITH 1.00, W/OUT 0.33, delta +0.67
Example run cost	$0.41 across 6 runs, 74 seconds
CI max cost flag	–max-cost-usd 20

Anthropic has released a structured evaluation system for Claude Code plugins. It targets three questions that had no formal answer before: does the skill trigger on real prompts, does it survive a model update or an edit, and does it outperform Claude with no plugin at all?

The command works against any directory containing a plugin.json or .claude-plugin/plugin.json manifest, or a skills-directory plugin. Every eval run and every judge grader is a real model call billed to your plan or API account.

How the eval suite is structured

Eval cases live in an evals/ directory inside the plugin. Each case is a subfolder containing a prompt.md file and a graders/ folder. The prompt is sent to Claude exactly as written. Frontmatter on the prompt file can set the model, tags, allowed tools, turn limit, and timeout.

Graders are markdown files. Their frontmatter sets the type, an optional weight, and an optional arm. The six types split into two groups:

• Free graders (computed from the transcript or files on disk): regex, tool_used, tool_order, file_exists
• Billed graders (call a judge model): llm, which scores against prose criteria you write; baseline, which compares against a reference answer

Running claude plugin eval init reads the plugin, asks what a good result looks like, proposes cases and graders, tests them, and writes the files. In CI, adding --bare <name> writes a blank template instead of running the interactive flow.

Why delta is the only number that matters

By default every case runs twice: a “with” arm (plugin loaded) and a “without” arm (plugin not loaded). The difference between the two scores is delta. If a case scores 1.0 in both arms, the plugin is not responsible for the pass. A score of WITH 1.00 and W/OUT 0.33 producing a delta of +0.67 is the documented example of a plugin that is genuinely pulling its weight.

A grader marked with-only, typically a tool_used: Skill check, is reported as an indicator but excluded from the delta calculation, because the without-arm has no skill to fire.

What the most common failure looks like

Anthropic flags one finding as especially frequent: a delta near zero combined with a failing tool_used: Skill grader. That pattern means Claude is not choosing the skill when users phrase things naturally. This is a behavioral defect that claude plugin validate cannot catch, because that command checks manifest syntax and schema, not runtime behavior.

Results are written to evals/results/<timestamp>/report.html with per-grader verdicts and judge votes. Where the account supports it, the report is also published to claude.ai unless --no-publish is set. This is worth knowing if you are evaluating proprietary prompts.

How to wire it into CI

The documented CI invocation uses a faster, cheaper judge model alongside a stronger agent model, plus a hard cost ceiling and a pass threshold:

1. Install Claude Code and set ANTHROPIC_API_KEY in your runner environment.
2. Add the eval command to your pipeline: claude plugin eval . --trust-plugin --json results.json --threshold 0.8 --model claude-sonnet-5 --judge-model claude-haiku-4-5 --no-publish --max-cost-usd 20
3. Gate merges on exit code: the command exits 1 if the threshold is not met or if the plugin is untrusted and no terminal is available.
4. Watch for usage-limit errors. According to Anthropic, hitting a usage ceiling can mimic a regression in the results, so monitor run costs separately.

Note that --json suppresses progress output, and report-level problems do not change the exit code. Only threshold failures and trust errors do.

Why it matters

Plugin developers have had syntax validation but no behavioral signal. An eval suite that ships with the plugin gives teams a repeatable check on whether a new Claude model version breaks existing skills, something that matters more as Anthropic accelerates model releases. The delta framing is also honest: it forces you to prove the plugin earns its place rather than taking credit for what Claude would do regardless.

For teams running AI integration projects that layer Claude plugins on top of business workflows, this fills a real gap. You can now catch skill-trigger regressions before they reach production users rather than after a support ticket surfaces the problem.

Our take

The delta concept is the right instinct. It is easy to write a plugin that passes tests because Claude is already good, not because the plugin helps. Forcing every case to run both arms puts the burden of proof where it belongs.

The cost structure deserves attention. A moderately sized suite with several llm or baseline graders and multiple runs per case can add up quickly, especially on a shared CI budget. The --max-cost-usd flag is useful, but teams should profile a single run manually before committing to automated pipelines. Using a cheaper judge model like claude-haiku-4-5 while running the agent on claude-sonnet-5 is the sensible split the docs already suggest.

We also note that the natural-phrasing trigger problem Anthropic highlights is not unique to plugins. We have seen similar issues in other tool-use training work: models trained to use tools in structured prompts often skip them when users write casually. Having a grader that specifically tests for this is more useful than most teams might expect on first read.

If you are building or maintaining Claude Code plugins, run claude plugin eval init on your next plugin before you ship it. The first delta you see will almost certainly tell you something you did not know.

Source: Marktechpost