Claude Plugin Eval: How the New Command Works (2026)

Run claude plugin eval in your terminal and Claude Code executes a plugin's eval suite against the live harness, handing you scored, reproducible results as both a JSON file and an HTML report — that is the new command Anthropic added in Claude Code version 2.1.269, per the official changelog on GitHub, and it is the first time plugin behaviour in Claude Code has had a built-in, repeatable test path. If you build plugins, install other people's, or run automations on top of Claude Code, this quietly fixes the most annoying gap in the workflow: until now, "does this plugin actually work?" was answered by vibes.

📺 Watch: Claude Code Just Got a HUGE Customization Upgrade

🔥 Get the Agent OS as a free bonus: AI Profit Boardroom members get the full Agent OS zip, prompt libraries, daily tutorials and weekly live coaching calls. → Get inside · Want AI SEO help 1-on-1? Book a free SEO strategy session →

What claude plugin eval actually does

The changelog entry for version 2.1.269 describes it plainly: claude plugin eval runs a plugin's eval suite against Claude Code and returns scored, reproducible results, exported as JSON alongside an HTML report you can open in a browser. The word doing the heavy lifting there is reproducible. Anyone who has built on top of an AI coding agent knows the failure pattern — a plugin behaves beautifully in the demo, then drifts as models update, prompts change, or an edge case appears, and nobody notices until something breaks in real work. A scored eval suite turns that from an anecdote problem into a numbers problem: run the suite, read the score, compare it with the last run. The full option set lives behind claude plugin eval --help, which the changelog points to as the reference for the command.

Why scored plugin testing matters for automation builders

If you only use Claude Code interactively, an eval command might sound like tooling for tooling's sake. It is not — it is infrastructure for trust. The moment you wire Claude Code into unattended work, every plugin in the chain becomes a dependency whose failure you will not see happen live. Scored evals give you three things you cannot get from manual spot-checks: a regression baseline before you update anything, an objective way to compare two plugins that claim to do the same job, and an artefact — the JSON output — that your own automation can read and act on. That last one is the sleeper feature. A scheduled job can run an eval suite, parse the score, and refuse to deploy a workflow whose plugin just got worse. That is the standard software discipline of CI testing arriving in the plugin layer of AI agents, and it lands at exactly the moment people are building real businesses on these stacks — the approach covered in agentic OS Claude Code setups, where one flaky plugin can quietly poison a whole pipeline.

If you want to build automation systems on Claude Code with people who ship them every week, check out the AI Profit Boardroom → join the community here. Want help applying this to your own business first? Book a free SEO strategy session — no cost, real plan.

How to try claude plugin eval today

Update Claude Code to version 2.1.269 or later, then run claude plugin eval --help to see the command's options — that help output is the canonical reference, and starting there beats guessing at flags. From the help text you can point the command at the plugin you want to test and let it produce the JSON and HTML outputs. If you are not yet on Claude Code at all, the learn Claude Code guide is the on-ramp, and it is worth knowing you do not need a big budget to experiment: the Claude Code free write-up covers the no-cost ways in, and Claude Code local covers running it against local setups. The sensible first experiment is small: take one plugin you already rely on, run its eval suite once, and save the report. That single baseline file is what makes every future run meaningful.

Claude plugin eval vs testing plugins by hand

None of this means manual testing disappears — you still want human eyes on a plugin's actual output the first time you adopt it, because a score never tells the whole story. What changes is the ongoing burden: the claude plugin eval command takes over the boring, repetitive verification that nobody was honestly doing anyway, and leaves human judgement for the decisions it is actually good at.

What else landed in Claude Code 2.1.269 and 2.1.270

The plugin eval command headlines the release, but the same changelog entry lists a second quality-of-life change worth knowing: an output-style command that switches styles across all session types, so a formatting preference follows you between sessions instead of being re-declared. The follow-up release, 2.1.270, fixed an irritation with long sessions — read-only git commands in Bash had started asking for permission unexpectedly after a session had been running for a while, and that prompt-fatigue bug is now gone. Small fixes, but this is the cadence story with Claude Code generally: the tool changes weekly, and the changelog on GitHub is the primary source worth actually reading rather than hearing about second-hand.

📺 Watch: Building AI Agent Operating Systems with Claude

Using claude plugin eval in a real workflow

Here is the practical pattern for automation builders. First, inventory the plugins your workflows actually depend on — most people are surprised how many they have accumulated, and if you are still choosing, community shortlists like the best agent plugins round-up show how much quality varies across ecosystems. Second, run evals before and after every meaningful change: a Claude Code update, a plugin version bump, a prompt rewrite in your own system. Third, keep the JSON outputs in one folder so trends are visible over weeks, not just single runs. If your Claude Code workflows earn money — the position every reader of a Claude Code SEO agent setup is in — then plugin regressions are revenue regressions, and ten minutes of eval discipline is the cheapest insurance available. The Agent OS guide covers where testing like this sits inside a full agent operating system, and the Goldie Bench write-up covers how the underlying model brains compare in hands-on tests — evals for your plugins, benchmarks for your models, and suddenly the whole stack is measured instead of assumed.

Who actually needs claude plugin eval?

Three groups, in order of urgency. Plugin authors first: if you publish a plugin, shipping it with an eval suite is about to become the difference between looking professional and looking hobbyist, because users can now score your work in one command and compare it with alternatives. Automation operators second: anyone running Claude Code on schedules, in pipelines, or against client work inherits every plugin's failure modes silently, and a scored check before each update is the cheapest protection on offer. Teams third: when several people share a Claude Code setup, an eval report is a far better argument than one person's anecdote about a plugin behaving oddly last Tuesday. If you are a casual interactive user with no plugins beyond the defaults, you can safely file this release under good-to-know — until the first time a favourite plugin misbehaves and you want the receipt that proves it.

Claude plugin eval: what we still do not know

Honest caveats, because the changelog is terse. Anthropic's entry confirms what the command does — scored, reproducible eval runs with JSON and HTML output — but the changelog itself does not spell out the full eval-suite format or every option, which is exactly why the --help output is the reference to trust. Expect the details to evolve too: 2.1.269 is the introduction of the feature, and Claude Code features typically sharpen quickly over the following releases. The way to stay current is unglamorous — check the changelog, re-run your suites after updates, and let the scores tell you what changed. What is already clear is the direction: plugin quality in Claude Code is now measurable, and measurable beats memorable every time you are building systems meant to run without you.

If you want the full Agent OS, prompt libraries, daily tutorials and weekly live coaching calls for building measured, money-making Claude Code systems, check out the AI Profit Boardroom → get inside here. And if a 1-on-1 plan would help first, book a free SEO strategy session.

Real wins from inside the AI Profit Boardroom

See all 3,000+ members →

← Back to all posts