A skill extracted from a successful conversation can carry over instructions that only work for that particular task. To evaluate agent skills, prepare new cases and compare the same agent with and without the skill. You’ll need a completed session, access to its instructions, and an environment where you can run each test with a clean context.
The goal is to check which parts of the procedure transfer: how it interprets the input, what decisions it makes, and how it verifies the output. Repeating the original task helps you review the extraction, but held-out tasks are what test its reusability.
What you need to evaluate a skill
An Agent Skill is a folder containing a SKILL.md file, which combines YAML metadata and Markdown instructions. It can also include scripts, references, and resources. According to the official specification, the required fields are name and description; the name must match the folder name.
You’ll also need access to the model and tools you’ll use to run the tests. You need to be able to keep sessions separate and save outputs, because a correct answer without its execution context tells you little when making comparisons.
Be clear from the start about what you want to improve. A skill that teaches a capability is evaluated on what it can solve; one that captures your team’s process is evaluated on how faithfully it follows that process. Anthropic covers this distinction in its guide to evaluating and improving skills.
Separating memory from procedure also helps you decide what to save. The article on Hermes Agent and its memory and skill layers provides more context on that distinction.
1. Extract the procedure from the completed conversation
Review the session and write down what the agent received, which tools it needed, which steps it followed, and which conditions the output had to meet. Anthropic’s skill-creator supports this kind of extraction from a conversation, including the corrections that led to the result.
Source: skills/skills/skill-creator/SKILL.md at main
Suppose the task was to transform a table into JSON for another system. Keep the rules for interpreting columns, handling incomplete values, and checking the output. Review the filenames, paths, and exceptions that came up during that session separately.
A fixed path may be part of your team’s actual process. In that case, keep it and explain its purpose. If it only pointed to the file for that particular task, turn it into an input the agent has to identify.
Write the skill using these steps:
- Describe the problem it solves and when it should activate.
- Specify the input it needs and what it should check before acting.
- Put the steps needed to solve the task in order.
- Explain the reasons behind the constraints and checks.
- Define the output it should deliver and how to verify it.
By the end, you should be able to connect every instruction to a procedural need. Pay particular attention to instructions added in response to a one-off error: they may provide a useful rule or introduce an overly narrow exception.
2. Prepare new cases and hold out their inputs
The official evaluation guide suggests saving cases in evals/evals.json, with prompt, expected_output, and optional files. It recommends starting with two or three cases, varying the wording, and including malformed inputs or edge cases.
Separate the cases you use to write the skill from the held-out set you’ll open after finalizing its instructions. If you adjust the procedure every time you see an input, that input has already played a part in its development.
For table transformations, prepare cases with differences you can interpret:
| Case | What you change | What you check |
|---|---|---|
| Different columns | Rename headers and change their order | The agent interprets the fields without relying on their original positions |
| Malformed input | Introduce incomplete records | It applies the intended handling without inventing values |
| New constraint | Change the requested output format | It follows the current requirements without carrying over the previous format |
Define the expected result before running the test. For fields, specify the correct mapping; for incomplete records, state how the task requires them to be handled. If you accept several valid answers, write down the criteria they share.
Don’t change every detail at once. A case that changes paths, structure, and constraints may reveal a failure, but you’ll then need to isolate which change caused it.
3. Freeze the skill and compare both configurations
Save an identifiable copy of the skill before opening the held-out cases. The specification supports recording a version through metadata, although that field is optional. This is the copy you’ll evaluate throughout the comparison.
Run each case with the same prompt and files, first in one configuration and then in the other. Keep the model, tools, and permissions the same; the difference you want to study is whether the skill is present.
The most important condition is a clean context. Use separate sessions without the development conversation or previous test results. If the agent has already seen the solution, you’re introducing help that contaminates the comparison.
Here’s the procedure:
- Open a new session without the skill and run the case.
- Save the output, available evidence, and usage data.
- Open another new session with the frozen copy of the skill.
- Run the same case and save the same data.
- Note whether the skill activated before scoring the result.
You should end up with two comparable runs per case. Keep tool errors and permission blocks too: they help distinguish a problem with the procedure from a problem with the environment.
4. Check results, time, and tokens
Evaluate observable conditions. “The answer looks better” leaves too much room for personal preference; “the JSON is valid and preserves the required records” identifies something you can check and document.
For mechanical checks, the official guide recommends verification scripts. You can check the format, required fields, or counts. For decisions that require judgment, use human review and, where possible, blind comparison without knowing which configuration produced each answer.
Record successes, time, and tokens alongside the evidence. A skill may improve adherence to the procedure while consuming more resources; that difference helps you decide whether it fits your work. If you’re already reviewing agent usage, the article on Claude Code limits provides context for that decision.
If you’re working in the skill-creator environment, its documentation includes this command for aggregating evaluations from the tool’s directory:
python -m scripts.aggregate_benchmark <workspace>/iteration-N --skill-name <name>Replace the placeholders with your evaluation directory and skill name. The command produces benchmark.json and benchmark.md with results for each configuration.
Repeat the cases if you want to study variability. The guide warns that standard deviation only provides meaningful information with multiple runs per case; a single answer lets you inspect a failure, but doesn’t tell you how often it happens.
5. Remove instructions to find out what they contribute
Identify instructions related to failures or to steps the agent could already handle without help. Remove one at a time and rerun the affected cases with a clean context. Keep a copy of each variant so you can attribute the change.
Start with instructions tied to the original case: specific paths, column names, fixed formats, or mandatory steps with no explanation of their purpose. If an instruction doesn’t change the results, flag it for review; if removing it causes a failure, keep the evidence that justifies retaining it.
Review the order of actions too. Vercel documented a case where instructing the agent to consult a skill first produced a correct page.tsx, but left changes to next.config.ts unfinished. Exploring the project first and then consulting the skill resolved both parts in its agent evaluation.
When you add a failure to your development examples, prepare another held-out case for the next evaluation. This prevents you from continuing to measure only tasks whose solutions have already influenced the instructions.
Common failures: activation, rigidity, and inherited context
Separate activation failures from execution failures. If the agent never consults the skill, review its description: one that’s too narrow may prevent its use, while one that’s too broad may activate it where it doesn’t belong.
This issue mattered in Vercel’s Next.js 16 experiment: the skill wasn’t invoked in 56% of cases and achieved the same 53% success rate as the configuration without documentation. With explicit instructions, it reached 79%. These are results from that particular experiment, useful for explaining why you should record activation as a separate check.
If the skill activates but fails when the inputs change, review the inherited assumptions. Replace fixed positions with criteria for identifying fields, explain how to handle incomplete data, and remove imposed formats that contradict the current request. Then repeat the comparison.
Be careful about reusing the chat where you wrote the skill: the agent may rely on corrections that never made it into SKILL.md. Run the next held-out case in new sessions, save both outputs, and first check whether the skill activated; then score whether it solved the task.
What do you think?
Leave your opinion, question or suggestion. Comments are synced with GitHub Discussions .