Run claude plugin eval command workflows from your plugin's root directory — the exact call is claude plugin eval . — and Claude Code will run every test case in your plugin's evals folder, score the results, and show you precisely what your plugin contributes compared with running no plugin at all. The command arrived in Claude Code v2.1.269, released on 11 September 2026 per the official Claude Code changelog, and it fills a gap plugin builders have complained about since plugins launched: until now, the only way to know whether your plugin actually worked was to load it and eyeball the output. Now you get scored, reproducible results with a JSON and HTML report at the end.
📺 Watch: Claude Code Just Got a HUGE Customization Upgrade
🔥 Get the Agent OS as a free bonus: AI Profit Boardroom members get the full Agent OS zip, prompt libraries, daily tutorials and weekly live coaching calls. → Get inside · Want AI SEO help 1-on-1? Book a free SEO strategy session →
Everything below comes from the official Claude Code changelog and the plugin evals documentation at code.claude.com, both published alongside the 11 September 2026 release. If you build or sell Claude Code plugins — or you use them to automate client work — this is one of the more consequential updates of the year, because it turns plugin quality from a vibe into a number.
What the Claude Plugin Eval Command Actually Does
The claude plugin eval command runs your plugin against a suite of test cases and scores the results. Each case is a realistic prompt — something a user of your plugin would genuinely type — plus one or more graders. A grader is a pass or fail check on what Claude produced: a regex over the reply, a check that a particular tool was called, confirmation that a file was created, or a rubric that a second model judges the reply against.
According to the official documentation, each case runs three times by default, because a single run of a non-deterministic agent tells you very little. A run's score is the fraction of its graders that passed, and the case's score is the mean across its runs. A case passes when its score meets the threshold, which defaults to 1.0 — every grader passing on every run.
The clever part is the no-plugin baseline. By default, every case also runs the same number of times with no plugin loaded, giving you two scores: WITH and W/OUT. The difference between them — reported as a delta — is what your plugin actually contributed. If a case scores 1.0 both with and without your plugin, your plugin is not what made it pass, and you have just learned something most plugin authors never find out. This matters if you are building the kind of automation stacks covered in the best Hermes agent plugins guide — the same testing discipline applies whichever agent platform you build on.
If you want plugin and agent workflows that are already tested and proven — over 1,000 done-for-you AI workflows, daily tutorials and weekly live coaching — check out the AI Profit Boardroom → get the full automation stack here. Prefer 1-on-1 help applying this to your own business? Book a free SEO strategy session and map it out with Julian directly.
Getting Started: claude plugin eval init
You do not have to write the eval suite by hand. From your plugin root, run claude plugin eval init and an interactive session opens in which Claude reads your plugin, asks what a good result looks like, proposes prompts that should and should not trigger the plugin, designs graders for each, pilots them once, and writes one case directory per prompt under evals/. When it tells you the suite is ready, exit the session and run claude plugin eval . to execute everything.
If you would rather see the file format yourself, claude plugin eval init --bare first-case writes a blank template: a prompt.md holding the message Claude receives plus run limits in its frontmatter, and a graders folder with one placeholder grader. The documentation recommends giving each case one grader on the result — the final message or a produced file — and one on how Claude got there, such as a tool-use check, so you learn both whether the answer was right and whether your plugin produced it.
The requirements are worth noting before you start: Claude Code v2.1.269 or later, a plugin directory with a manifest, and the same authentication your normal sessions use. Every eval run and every judge grader is a real model call on your account, counted against your plan's usage or API bill. If usage limits are a concern, the guide to reducing Claude Code token usage pairs well with this — and the docs suggest keeping quick every-change suites to graders that do not call a judge model at all.
Grader Types and Scoring
Six grader types are available, per the official documentation. Four of them — regex, tool_used, tool_order and file_exists — are computed from the transcript and files and cost nothing. Two of them — llm and baseline — call a judge model and add to the run's cost. The judge defaults to a small fast model; you can pass a stronger one with the judge-model flag for nuanced rubrics.
The documentation is refreshingly honest about stability: a judge grader can differ between runs, and it differs more the longer the text it reads. For long output such as a generated file, grade with a regex over the file's contents instead. Keep judge graders for short outputs, with rubrics written as concrete PASS and FAIL conditions. That advice mirrors how the Goldie Bench write-up approaches comparing model brains — define what good looks like before you run anything, or the numbers mean nothing.
One subtlety: in a two-arm run, a check like "the skill was invoked" can never pass without the plugin, so counting it would inflate the delta. Claude Code therefore excludes such graders from the score in both arms and reports them as pass or fail indicators only.
📺 Watch: Free Claude Code Is CRAZY GOOD!
Running Evals in CI
This is where the command earns its keep for teams. In a CI job you run the suite with a json flag to archive the result document, pass trust-plugin so the job never waits at the first-run trust prompt, pin both the agent model and the judge model so a model rollout is not mistaken for a plugin regression, and set a cost ceiling with max-cost-usd as an upper limit.
The exit codes are designed for build gates: 0 means every case scored at or above the threshold, 1 means a case fell below it or a case file failed to load, and 2 means a partial run — the cost ceiling was hit or credentials were rejected — with the results file still written and marked partial. You fail the build on the exit code and you have automated quality gating for plugin changes, exactly the pattern serious teams use for the agent systems covered in Claude Code training.
Security is handled sensibly. Runs never stop to ask permission; tools that need a grant you did not give — Bash, Write, Edit, web access — are simply removed from the session. When you do grant Bash, every command runs under Claude Code's OS-level sandbox: writes confined to the run's workspace, your home directory unreadable, network limited to domains you explicitly allow. On a machine with no sandbox backend, Claude Code refuses the run rather than running it unconfined.
The Report: WITH, W/OUT and Delta
When a suite finishes you get a summary table showing each case's WITH score, W/OUT score, delta, run count and estimated cost, followed by a report.html — a single self-contained file with no external requests, so you can attach it to a CI job or open it from disk. If you are signed in with a claude.ai subscription, Claude Code also publishes the report as a private artifact and prints the link.
The report's verdict line answers the only question that matters: did the plugin help? It shows the suite score, the ablation delta against baseline, how many cases met the threshold, and the share of perfect runs. Inside each case, failed graders come pre-expanded with explanations, and judge graders show the votes and the evidence judged. The documentation notes the most common first finding is a delta near zero with the skill-invocation grader failing — meaning Claude is not choosing your skill on natural phrasing, and the fix is usually the skill's description field, not the skill itself.
For plugins that talk to MCP servers, there is a full mocking system: one Markdown file per tool under evals/mocks answers calls without starting the real service, keeping runs repeatable and free of side effects. That is a genuinely mature testing story for something released this week.
Why This Matters for Automation Builders
If you are wiring Claude Code into business automations — the kind of setups shown in the Claude Code SEO agent guide or when you connect Hermes agent to Claude Code — plugin reliability is revenue reliability. A plugin that fires on 70 percent of natural phrasings quietly costs you 30 percent of your automation's value, and until 11 September 2026 there was no honest way to measure that. Now there is: cases times runs, with and without, delta on the table.
The workflow the documentation recommends is simple. Build the plugin, run claude plugin eval init, run the suite, read the delta. Adjust descriptions and rubrics, re-run at the default three runs before trusting any change, then gate CI on the threshold so regressions cannot ship. The same measure-then-improve loop sits at the heart of Agent OS — instrument the system first, then optimise what the numbers tell you to optimise.
If you want working, tested Claude Code automations handed to you instead of building the eval suites yourself, check out the AI Profit Boardroom → join 3,000+ AI operators inside. And if you want a personal roadmap for AI SEO and automation in your business, book a free SEO strategy session — no pitch, just a plan.
