Run claude plugin eval from your terminal and Claude Code executes your plugin's eval suite and hands back scored, reproducible results as a JSON file plus an HTML report — that is how to use claude code plugin eval in one sentence, and the command arrived in Claude Code v2.1.269 on 11 September 2026, per the official Claude Code changelog. Run claude plugin eval --help first: the changelog points to it as the reference for the command's options, and it is the definitive list of what the tool accepts on your installed version.
📺 Watch: Free Claude Code is Absolutely INSANE!
🔥 Get the Agent OS as a free bonus: AI Profit Boardroom members get the full Agent OS zip, prompt libraries, daily tutorials and weekly live coaching calls. → Get inside · Want AI SEO help 1-on-1? Book a free SEO strategy session →
Why this is a bigger deal than one more CLI subcommand: plugins have quietly become the way serious Claude Code setups get packaged. Slash commands, skills, hooks and agents all travel inside them, and the ecosystem has been building for months — the official what's-new digest for early May 2026 records plugins loading from .zip archives and URLs, and the June digest records a /plugin list command for printing what you have installed. What the ecosystem lacked was a standard way to answer the question every plugin author and every plugin user eventually asks: does this thing actually work, and did my last change make it better or worse? Scored evals are that answer.
How to Use Claude Code Plugin Eval, Step by Step
The workflow is short, and everything here sticks to what the official changelog documents rather than guessing at flags:
- Update Claude Code to v2.1.269 or later. The eval command does not exist on older versions — v2.1.269 shipped on 11 September 2026, so anything current has it.
- Read the built-in help. claude plugin eval --help prints the options your version supports. Because the feature is brand new, the help output is the source of truth — lean on it rather than on second-hand blog posts, this one included.
- Point the command at a plugin with an eval suite. The changelog describes the command as running "a plugin's eval suite against Claude Code", so the suite lives with the plugin — if you are evaluating your own plugin, that means writing eval cases for it first.
- Read the two outputs. You get machine-readable JSON — ideal for tracking scores over time or wiring into CI — and a human-readable HTML report for actually understanding what passed, what failed and why.
If you are newer to the tool itself, the learn Claude Code guide covers the foundations that make the plugin system make sense, and the free Claude Code write-up covers what you can do without paying a penny.
If you want to skip the trial and error and copy working Claude Code systems — plugins, prompts and automations included — check out the AI Profit Boardroom → get the working setups inside. Want a 1-on-1 plan for your specific business? Book a free SEO strategy session and get direct answers.
Why Scored Plugin Evals Change the Game for Automation Builders
Anyone who has built automation on top of a language model knows the core problem: the model is probabilistic, so "it worked when I tried it" is not evidence of much. A plugin that nailed your test prompt on Tuesday can fumble a slightly different one on Thursday. Until now, most people validated Claude Code plugins by vibes — run it a few times, eyeball the output, ship it. That approach collapses the moment a plugin matters to your business, because you have no baseline, no regression signal and no way to compare two versions objectively.
The eval command replaces vibes with numbers. Reproducible, scored runs mean you can make a change to your plugin's prompts or logic, re-run the suite, and see the score move. That is the same discipline software engineers apply with test suites, finally applied to the fuzzy layer. For automation builders the practical wins look like this:
- Regression protection. Before you ship a plugin update to your team or clients, the suite tells you whether the change broke behaviour that used to work.
- Objective comparisons. Two competing approaches to the same workflow stop being a matter of opinion — run both suites, compare scores.
- CI integration. JSON output slots into a pipeline, so a failing eval can block a release automatically instead of relying on someone remembering to check.
- Buyer due diligence. When you evaluate someone else's plugin before adopting it, a runnable eval suite is worth more than any sales page.
A note on writing the eval cases themselves, because the suite is only as honest as its cases: the temptation is to test the happy path you demoed. Resist it. The cases that earn their keep are the awkward ones — ambiguous instructions, inputs at the edge of what the plugin claims to handle, the request phrased the way a distracted colleague would actually phrase it. Write a handful of those before you write the flattering ones, score your plugin against them, and you will learn more in one run than a week of casual use would teach you. Then keep the suite growing: every real-world failure you hit becomes a new case, so the same mistake can never ship twice.
This matters even more given how deep plugins now reach into working setups. A plugin can carry an entire SEO workflow — the kind of system covered in the Claude Code SEO agent guide — or a video-rendering pipeline like the one in the Claude Code Remotion write-up. When a plugin is effectively an employee, you want a performance review process for it.
📺 Watch: How to Use Codex + Claude for FREE!
What Else Shipped Alongside Plugin Eval in v2.1.269
The same 11 September 2026 release bundled several changes that reinforce the theme of Claude Code becoming measurable, controllable infrastructure, per the official changelog:
- Output style switching from the prompt. A new /output-style command lists and switches output styles, and works over Remote Control and in cloud and other headless sessions.
- Bash edit diffs. With the bashEditDiffEnabled setting, Bash commands that edit files attach a diff of what changed to the tool result — more visibility into what your automations actually did.
- Repository-tagged telemetry. OTEL_METRICS_INCLUDE_REPOSITORY tags OpenTelemetry metrics and events with repository attributes, so organisations can slice usage and outcomes by repo.
- A faster diff panel. The fullscreen /diff panel, added on 3 September in v2.1.260, now opens fully rendered in one step instead of showing a loading state first.
Put together, the pattern is unmistakable: evals for plugins, diffs for edits, telemetry for organisations. Claude Code is being built out as something you can run a business on and audit afterwards, not just a clever terminal toy. If you are choosing between agentic coding stacks, that operational maturity belongs in the comparison alongside raw model quality — the DeepSeek harness vs Claude Code comparison looks at how the alternatives stack up.
📺 Watch: Building AI Agent Operating Systems with Claude...
How Plugin Evals Fit a Full Agent Operating System
A single evaluated plugin is useful; an evaluated system is a business asset. The structure Julian teaches — covered in the Agent OS resource — treats agents, skills and plugins as components of one operating system, and the agentic OS with Claude Code guide shows how that shape maps onto Claude Code specifically. Evals slot into that architecture as the quality gate: every component that matters gets a suite, and every change to the system has to keep the scores green before it ships.
If part of your system's job is choosing which model to run a task on, benchmark data belongs next to your eval data — the Goldie Bench write-up covers how the main model brains compare in hands-on tests, which is the complement to evaluating your own plugin layer on top.
The honest caveat to end on: this feature is days old, and the changelog entry plus the built-in help are currently the complete official documentation. Expect the details — suite formats, scoring options, CI patterns — to evolve over the coming weeks. What will not change is the direction: how to use claude code plugin eval is about to become a standard skill for anyone shipping serious automation on Claude Code, the same way writing tests became a standard skill for anyone shipping serious software. Learning it now, while the ecosystem is young, is a genuine edge.
If you want to build automations that hold up — with the full Agent OS zip, prompt libraries, daily tutorials and weekly live coaching calls — check out the AI Profit Boardroom → get inside and start building. Prefer 1-on-1 guidance first? Book a free SEO strategy session and get a plan tailored to you.
