How to Write an Agent Skill: Build and Test a SKILL.md

To write an Agent Skill, create a folder named after the skill, add a SKILL.md file with a name and a description in YAML frontmatter, and write the instructions below it. Put any fragile or repetitive steps into a script in scripts/. Then drop the folder into your agent's skills directory, test that it triggers on the right prompts, and push it to GitHub.

This tutorial builds a real, useful skill that turns git history into release notes. It covers the folder, the script, the description that makes it trigger, testing, and publishing. For background on how skills load and where they run, see the complete guide to Agent Skills.

The steps at a glance

Step	What you do	What you end up with
1. Pick a job	Run the task without a skill and note what goes wrong	A short list of failures to fix
2. Create the folder	Make a folder whose name matches the skill's name	writing-release-notes/
3. Write the script	Move deterministic steps into code	scripts/collect_commits.py
4. Write SKILL.md	Add frontmatter and short rules	A valid SKILL.md
5. Write the description	Say what it does and when to use it	A description that triggers on the right prompts
6. Test it	Validate, run a query set, compare with and without the skill	Evidence the skill helps
7. Publish	Push a clean repo layout to GitHub	An installable repo
8. Get listed	Submit the repo for a free scan	A public listing with a scan report

Step 1: Pick a job the agent gets wrong

If you are new to skills, read what are agent skills first. If the job needs live data from another system, you may want an MCP server instead; see skills vs MCP servers.

Anthropic's authoring best practices recommend starting from real failures, not imagined ones. Run the task without a skill and note what goes wrong.

Ask an agent to "write release notes since v1.4.0" and you will typically get something reasonable but inconsistent. Sections change between runs. Merge commits sneak in. Breaking changes get buried. Those are exactly the gaps a skill can close:

• A fixed set of sections in a fixed order.
• Merge commits excluded.
• Breaking changes always listed first.
• A script that collects commits the same way every time.

Step 2: Create the folder

The Agent Skills spec requires the folder name to match the skill's name. Anthropic suggests gerund names, so call it writing-release-notes:

writing-release-notes/
├── SKILL.md
├── scripts/
│   └── collect_commits.py
└── references/
    └── STYLE.md

For Claude Code, create it inside .claude/skills/ in a test repo, or in ~/.claude/skills/ to make it available everywhere. Codex, VS Code and Gemini CLI all read .agents/skills/, so that path works if you want to test in several tools.

Step 3: Write the script

Collecting and grouping commits is deterministic work. A script does it the same way every time, and when the agent runs it, only the output enters the context window, not the source code.

Save this as scripts/collect_commits.py:

#!/usr/bin/env python3
"""Group commit subjects between two git refs by Conventional Commit type."""
import re
import subprocess
import sys

PATTERN = re.compile(r"^(\w+)(\([^)]*\))?(!)?:\s*(.+)$")
SECTIONS = {"feat": "Features", "fix": "Bug fixes", "perf": "Performance"}
ORDER = ["Breaking changes", "Features", "Bug fixes", "Performance", "Other"]


def main() -> None:
    if len(sys.argv) < 2:
        print("Usage: collect_commits.py <from-ref> [to-ref]", file=sys.stderr)
        sys.exit(1)
    start = sys.argv[1]
    end = sys.argv[2] if len(sys.argv) > 2 else "HEAD"
    try:
        log = subprocess.run(
            ["git", "log", "--no-merges", "--pretty=format:%s", f"{start}..{end}"],
            capture_output=True, text=True, check=True,
        ).stdout
    except subprocess.CalledProcessError as err:
        print(f"git log failed: {err.stderr.strip()}", file=sys.stderr)
        print("Check that both refs exist: git tag --list", file=sys.stderr)
        sys.exit(1)

    groups = {name: [] for name in ORDER}
    for subject in log.splitlines():
        match = PATTERN.match(subject)
        if not match:
            groups["Other"].append(subject)
            continue
        kind, _scope, bang, text = match.groups()
        if bang:
            groups["Breaking changes"].append(text)
        else:
            groups[SECTIONS.get(kind, "Other")].append(text)

    if not any(groups.values()):
        print(f"No commits found between {start} and {end}.")
        return
    for name in ORDER:
        if groups[name]:
            print(f"## {name}")
            print("\n".join(f"- {item}" for item in groups[name]))
            print()


if __name__ == "__main__":
    main()

The script follows two pieces of Anthropic advice. It handles errors itself and prints a useful hint instead of failing silently. It also uses only the Python standard library and a local git command, with no network calls and no secrets, so it works in restricted environments and is easy for users to trust.

Step 4: Write SKILL.md

Keep the body short. The model already knows what release notes are. It needs your rules.

---
name: writing-release-notes
description: Writes release notes and changelog entries from git history between two tags or refs, grouped into breaking changes, features, fixes and performance. Use when the user asks for release notes, a changelog, what changed since a version, or a summary of commits for a release.
license: MIT
compatibility: Requires git and Python 3.8+. No network access needed.
---

# Writing release notes

## Workflow

1. Find the previous release tag with `git describe --tags --abbrev=0` unless the user gave one.
2. From the repository root, run the collector bundled in this skill's `scripts/` folder:
   `python scripts/collect_commits.py <previous-tag> HEAD`
3. Rewrite each line for users, not developers. Keep the meaning. Drop internal-only changes such as test or CI tweaks.
4. Follow the rules in [references/STYLE.md](references/STYLE.md).
5. Show the draft to the user. Do not create tags or commit files unless asked.

## Rules

- Breaking changes always come first, each with one line on how to migrate.
- Never invent changes that are not in the script output.
- If the script prints no commits, say so and ask which refs to compare.

Then add references/STYLE.md with your tone, heading format and an example of good output. It only loads when the agent follows the link, so it costs nothing when unused.

The license and compatibility fields are optional in the spec. They help people who find the skill later. For the full list of fields and limits, see the SKILL.md format explained.

Step 5: Write a description that triggers

The description is the only part the agent sees before it decides to load your skill. Claude's docs are explicit that it must say both what the skill does and when to use it. Practical rules:

• Write in the third person. "Writes release notes," not "I can help you write release notes."
• Lead with the main use case. Claude Code truncates the combined description and when_to_use text at 1,536 characters in the skill listing.
• Include the words users actually type. "Changelog," "what changed since," "release notes." Synonyms matter.
• Name the inputs. Here, git history, tags and refs.
• Stay specific. "Helps with git" will trigger on the wrong tasks or not at all.

Compare these two:

# Too vague
description: Helps with releases.

# Specific, with triggers
description: Writes release notes and changelog entries from git history between two tags or refs. Use when the user asks for release notes, a changelog, or what changed since a version.

If a skill has side effects, decide whether the agent should be allowed to load it on its own. In Claude Code, disable-model-invocation: true makes a skill manual-only, run as a slash command (Claude Code docs). Our skill only reads git history, so automatic loading is fine.

Step 6: Test it

Validate the format

The spec project's skills-ref tool checks frontmatter and naming:

skills-ref validate ./writing-release-notes

Check the agent can see it

In Claude Code, run /skills to list available skills, or type / to see it in the command menu. Claude Code watches skill folders, so edits apply in the current session without a restart.

Test triggering with a query set

Write prompts that should trigger the skill and prompts that should not. Atlan's guide to writing an agent skill suggests 5 to 7 should-trigger queries and 3 to 5 should-not-trigger queries, and holding out roughly 40% of them to check that a description fix generalizes instead of fitting the prompts you tuned on. A starting set for this skill:

Should trigger:
- Write release notes for everything since v1.4.0
- What changed between v2.0.0 and v2.1.0? Format it as a changelog
- Draft the changelog for this release

Should not trigger:
- Explain what this commit does
- Write a commit message for my staged changes
- Create a v1.5.0 tag

Run each in a fresh session. If it misses a should-trigger prompt, add that phrasing to the description. If it fires on a should-not prompt, make the description narrower. Then rerun the held-out prompts you did not tune on.

Test the output

Anthropic suggests building at least three evaluation scenarios and testing with each model you plan to use, since a skill that works for a large model may need more detail for a small one. For this skill, try a repo with breaking changes, one with no Conventional Commit prefixes, and a bad tag name. Check that the script's error message leads the agent to recover.

With and without the skill: measure it

A skill should beat the same task done without it. Anthropic's best practices call this establishing a baseline: measure Claude on your scenarios without the skill, then compare after each change. Flavio Copes's agent skills write-up shows a simple way to do it: run the same prompt with the same permissions twice, once with the skill and once without, and record tool calls, calls refused by permissions, files read outside the repo, wall-clock time, cost and the final answer. Fewer tool calls and a consistent answer are good signs. A skill that adds time and changes nothing is a candidate for deletion.

Use Anthropic's skill-creator

Anthropic publishes a skill-creator skill that automates much of Step 6. According to its SKILL.md and the Claude Code skills docs, it helps you draft a skill, saves test prompts as evals, runs each test with the skill and against a baseline, grades the outputs, and tunes the description with should-trigger and should-not-trigger prompts. The eval file format is documented on agentskills.io under evaluating skill output quality. It is a quick way to get the query set and the with-and-without comparison above without writing the harness yourself.

Why isn't my skill triggering?

Claude Code's troubleshooting section lists the usual causes. Work through them in order:

1. The description lacks the words users say. Add the phrases from your missed should-trigger prompts.
2. The skill is not loaded. Ask Claude "What skills are available?" or run /skills, and check the folder sits in a path your agent scans.
3. The frontmatter does not parse. If the YAML is malformed, Claude Code loads the body with empty metadata, so you can still run the skill by name but Claude cannot match your description. Run Claude Code with --debug to see the parse error.
4. The frontmatter is not on line 1. Claude Code reads frontmatter only when the opening --- is the file's first line.
5. A field name is misspelled. Claude Code ignores unknown fields without an error, so disable_model_invocation does nothing.
6. Too many skills are installed. Claude Code drops some descriptions when the skill listing exceeds its budget. Run /doctor to see the listing's context cost.

If the skill fires too often instead, make the description more specific, or add disable-model-invocation: true so it only runs when you call it.

Keep data separate from instructions

A skill's instructions are trusted. The data it reads is not. Anthropic's skills overview warns that skills which fetch data from external URLs "pose particular risk", because fetched content might contain malicious instructions. In practice:

• Keep rules in SKILL.md and references/, and treat file contents, web pages and command output as data to process, not instructions to follow.
• Have scripts return structured output, such as grouped commit subjects, rather than raw text the agent might obey.
• Avoid network calls unless the job needs them, and say so in compatibility. This skill reads only local git history.
• Do not bundle secrets. Anyone who installs the skill can read every file in it.

Step 7: Publish to GitHub

A clean layout makes the skill easy to install and easy for tools to detect:

release-notes-skill/
├── README.md
├── LICENSE
└── skills/
    └── writing-release-notes/
        ├── SKILL.md
        ├── scripts/
        │   └── collect_commits.py
        └── references/
            └── STYLE.md

In the README, cover what the skill does, the install command for each tool, what the script runs, and whether it needs network access. Users can install it in Claude Code by copying the folder:

git clone https://github.com/your-name/release-notes-skill.git
cp -r release-notes-skill/skills/writing-release-notes ~/.claude/skills/

For claude.ai, they can zip the writing-release-notes folder and upload it under Settings, then Features.

Step 8: Get it listed and scanned

People are right to be careful about installing skills, because skills can run code. Make it easy for them to check yours. Submit your repo to AgenticMart for a free scan. The marketplace detects a skill repo by a SKILL.md file or a skills/ folder, so the layout above works as is. Your skill then appears in the agent skills directory.

The scan runs on a fresh GitHub Actions virtual machine that is destroyed afterwards. It runs Semgrep plus custom rules for eval, shell execution, hidden environment variable reads and obfuscation, checks for secrets with Gitleaks and for known-vulnerable dependencies with OSV-Scanner, then starts the code in a locked-down Docker container and records outbound network traffic. The results become badges, a 0 to 100 trust score and a public report. See how the free scan works for details. A scan is a malware and risk check, not a safety guarantee. A clean scan helps users decide, but it is not proof of safety, so keep your scripts small and readable.

If you later want to charge for skills or premium versions, our guide to turning an open-source project into a paid agent covers that path.

FAQ

Why is my skill not triggering?

Almost always the description. Make it specific, write it in the third person, and include the exact phrases users type. Also check the folder name matches the name field and that the file is in a path your agent scans.

Do I need a script in my skill?

No. Many skills are instructions only. Add a script when a step must be exact every time, such as parsing, validation or data collection, because scripts are more reliable than generated code and only their output uses context.

Can one skill work in Claude, Codex and Copilot?

Yes, if you stick to the core spec fields and avoid tool-specific features. Place it where each tool looks, such as .claude/skills/ for Claude Code or .agents/skills/ for Codex, VS Code and Gemini CLI.

How do I update a published skill?

Push changes to the repo and note them in the README or release notes. On the Claude API, each new version must be a full upload of every file, because versions are complete snapshots.

How many test prompts does a skill need?

Anthropic recommends at least three evaluation scenarios for output quality. For triggering, Atlan suggests 5 to 7 prompts that should trigger and 3 to 5 that should not, with about 40% held out to check that your fixes generalize.

Can Claude write a skill for me?

Yes. Anthropic's best practices say you can ask Claude to create a skill and it produces a properly structured SKILL.md. Anthropic's skill-creator skill goes further and also builds evals and tunes the description. Review the result for length and test it before you publish.