Hermes agent video analysis works two ways: the built-in video_analyze tool, which sends a clip to a video-capable model and answers your question about it, and the new video-xray plugin, a full analysis workbench that samples frames, transcribes the speech and writes a timestamped report. The first ships with Hermes but is switched off by default. The second is a community plugin that joined the official plugin catalog in the 15 November 2026 canary build.
📺 Watch: Hermes Agent Makes Animated Videos (One Sentence → Cartoon)
🔥 Get the Agent OS as a free bonus: AI Profit Boardroom members get the full Agent OS zip, prompt libraries, daily tutorials and weekly live coaching calls. → Get inside
This guide covers both routes, what each one sends where, and which one fits the job in front of you. Every command and setting below comes from the Hermes docs, the Hermes source code or the plugin's own README.
Julian Goldie shares his working agent setups, step-by-step trainings and weekly coaching calls inside the AI Profit Boardroom. Want a plan for your own videos first? Book a free AI strategy session.
Two ways to do Hermes agent video analysis
The routes solve different problems, so pick by the question you want answered.
| Question | Built-in video_analyze | video-xray plugin |
|---|---|---|
| Who maintains it | Nous Research, inside Hermes | LuuMoss, community tier of the catalog |
| How you turn it on | Add the opt-in video toolset | Install the plugin, enable it, reload Desktop |
| Which model looks at the video | An auxiliary video-capable model, for example Gemini | The chat model Hermes Desktop is already using |
| Speech | Part of the model's own description | Separate transcript from Whisper, R2T2 or a cloud service |
| Output | One written answer to your question | Markdown report, SRT and JSON cues, frame gallery, generation prompt |
| Platforms | Wherever Hermes runs | Verified on Windows only |
| File limits | mp4, webm, mov, avi, mkv, mpeg; about 50 MB max | Set by ffmpeg and your chosen time window |
A quick question about a short clip suits video_analyze. A long recording you want broken into scenes, quotes and timestamps suits video-xray.
Route 1: the built-in video_analyze tool
The Hermes tools reference describes video_analyze as a tool that analyses video from a URL or a file path, covering captions, scene breakdowns, key timestamps and visual descriptions.
It lives in its own toolset called video. The docs mark that toolset as opt-in, so it is not loaded in the default CLI set. You add it with the toolsets flag set to video, or by listing video under toolsets in your config.
Under the hood the tool takes two things: the video location and your question. The Hermes source tells the model to describe everything in the clip, including motion, audio cues, text overlays and scene changes, and then answer what you asked.
Which model does the watching
This is the part most people miss. Images can go straight to a vision-capable main model, but video_analyze has no native path. The vision tools source says it always runs through an auxiliary model, and its description names Gemini as an example of a video-capable model.
Hermes picks that model from your auxiliary video or vision settings, with the environment variables AUXILIARY_VIDEO_MODEL and then AUXILIARY_VISION_MODEL. If no auxiliary vision client can be found, the tool does not appear at all.
So if video_analyze is missing after you enable the toolset, check the auxiliary vision provider in the configuration docs before anything else.
Formats and size
- Supported formats: mp4, webm, mov, avi, mkv and mpeg.
- Clips over 20 MB may be slow.
- The hard cap is about 50 MB, so trim or compress longer recordings first.
- The source also states that video processing is not available to a hosted agent.
📺 Watch: Hermes: New AI Video Game Agent is WILD!
Route 2: the video-xray workbench plugin
The canary release on 15 November 2026 lists "Catalog: add video-xray (video analysis workbench)" from @LuuMoss in its release notes. The catalog entry is version 0.2.2, needs Hermes 0.22.1 or newer and sits in the community tier, which means a third-party maintainer, not Nous Research.
Its README calls it a video and audio analysis workbench for Hermes Desktop. One run moves through five stages:
- Frame sampling over a time window you pick, or the whole video.
- Visual observation in segments of 8, 16, 32 or 64 frames, sent to your vision model. Cost grows in line with the frames you sample.
- Timestamped transcript from local Whisper, local R2T2, a cloud OpenAI-compatible speech service, or none.
- Structured report with plot flow, camera language, characters and settings, quote references and an opening-versus-ending contrast, saved as Markdown.
- Reverse prompt that turns the analysis into a shot-by-shot video generation prompt, split at real shot cuts.
The plugin also registers two agent tools, video_xray and video_xray_jobs, so you can type "analyze this video" in chat instead of using the panel.
Going deeper than one clip? The AI Profit Boardroom has Julian's step-by-step Hermes trainings and four live coaching calls a week.
📺 Watch: NEW Hermes AI Video Agent is INSANE!
What video-xray sends, and where
The plugin catalog requires a security note on every entry. The video-xray note says:
- It runs ffmpeg, ffprobe and local speech-worker processes on your computer.
- By default it starts an analysis job for every video you attach when you press send. Turn off "Analyze attached videos on send" in the panel if you only want manual runs.
- Video frames go to your configured Hermes chat model.
- Audio goes to the cloud speech endpoint you set, if you set one.
- The faster-whisper engine downloads its model weights from Hugging Face on first use.
- A cloud speech key you save is stored in plain text in an owner-only file in the Hermes data folder.
That default-on behaviour matters. If your chat model is a cloud model, each attached video means frames sent to that provider. For client footage, use a local model or switch the auto setting off.
Setting up video-xray on Windows
The catalog marks the plugin as verified on Windows only. The README lists Hermes Desktop 0.22 or newer, ffmpeg on your PATH, and any working chat model.
- Copy the plugin files from the repo root into a video-xray folder inside the Hermes plugins directory under your local app data.
- Run hermes plugins enable video-xray, then fully quit and reopen Hermes Desktop so the gateway reloads.
- Copy the repo's desktop plugin.js into the matching desktop-plugins folder and press Ctrl+Shift+R.
A gold Video Analysis Workbench strip should appear above the composer. The /video command opens it too.
For speech, the README suggests Whisper as the starting point because it handles many languages. R2T2 is the strongest local option for Chinese, and the cloud option needs no local install.
Stuck between the two routes for your business? A free AI strategy session maps out which one fits.
Practical uses for Hermes agent video analysis
- Webinar notes. Run video-xray over the recording and export the SRT and the Markdown report for a written recap.
- Ad reviews. Ask video_analyze what text appears on screen and when the product first shows up.
- Tutorial chapters. Use the timestamped report to draft chapter markers for a long screen recording.
- Group chat clips. The Telegram docs say a video posted in an observed group is cached, and the agent opens it with video_analyze later when someone asks, as long as the video toolset is on.
If you want to make videos instead of reading them, that is a different toolset. Read the guide to Hermes Agent AI video generation.
My verdict on Hermes agent video analysis
Start with video_analyze. It is already in Hermes, it needs one toolset change and one auxiliary model, and it answers most quick questions about a short clip.
Move to video-xray when you need structure: a transcript with timestamps, a scene-by-scene report and files you can reuse. Treat it as a community plugin, keep it on Windows where it is verified, and switch off auto-analysis if footage is private.
To see how plugins, profiles and the gateway sit together on one machine, read the Agent OS guide. To shortlist the model doing the watching, compare real task scores on Goldie Bench. Members of the AI Profit Boardroom get Julian's agent setups, and you can book a free AI strategy session for a plan built around your content.
FAQ
Can Hermes Agent watch a video file?
Yes. Turn on the opt-in video toolset and the video_analyze tool takes a URL or local path plus a question, and returns a written answer from a video-capable model.
Why is video_analyze missing from my tools?
Either the video toolset is not enabled, or Hermes cannot find an auxiliary vision client. The tool only registers when one of those providers resolves.
Is video-xray an official Nous Research plugin?
No. It is listed in the community tier of the Hermes plugin catalog and maintained by LuuMoss. Nous reviews catalog entries and pins a commit, but the code is third-party.
Does video-xray work on a Mac?
The catalog entry lists Windows as the only verified platform, and its install steps use Windows paths. Plan on a Windows machine running Hermes Desktop.
How big can a video be for video_analyze?
The source caps the payload at about 50 MB and warns that anything over 20 MB may be slow. Trim long recordings or use video-xray's time windows.
Can video-xray create a prompt for an AI video tool?
Yes. It writes a shot-by-shot MiniMax-H3 prompt, and the README says removing a few formatting lines makes it usable in other generators such as Kling or Runway.
Enable the video toolset today, ask one clip one question, and add video-xray when you need full reports from your hermes agent video analysis.
