Skill Creator

---
license: Apache-2.0
name: skill-creator
description: 'Use this skill when creating a new Claude skill from scratch, editing or improving an existing skill, or measuring skill performance with evals and benchmarks. Invoke whenever the user says things like ''make a skill for X'', ''turn this workflow into a skill'', ''test my skill'', ''improve my skill'', ''run evals'', ''benchmark this'', or ''optimize my skill description''. Also use proactively when the conversation has produced a repeatable workflow that would benefit from being captured as a skill. Covers the full lifecycle: capture intent, draft SKILL.md, run evals, review with user, iterate, optimize description, package. NOT for general coding help, debugging runtime errors, building MCP servers, writing Claude hooks, or creating plugins - use domain-specific skills for those.'
metadata:
  tags:
    - skill-creation
    - evals
    - benchmarking
    - skill-improvement
    - meta
category: Agent & Orchestration
tags:
  - skill-creation
  - authoring
  - templates
  - meta
  - development
---

# Skill Creator

A skill for creating new skills and iteratively improving them through systematic testing and evaluation.

## Decision Points

Navigate based on the user's entry point and current skill state:

```
Entry Assessment
├── "Make a skill for X" → Capture Intent → Draft SKILL.md
├── "Turn this workflow into a skill" → Extract from conversation → Confirm intent
├── "Test/improve my skill" → Locate skill → Run test cases
├── "Optimize description" → Load skill → Generate trigger evals
└── Proactive detection → Assess if workflow is repeatable → Suggest skill creation

Skill State Check
├── No skill exists → Full creation flow (intent → draft → test → iterate)
├── Draft exists, no tests → Create test cases → Run evaluations
├── Has tests, poor performance → Analyze failures → Improve skill → Retest
└── Working skill → Description optimization → Package

Test Case Strategy
├── Objectively verifiable outputs (code, data, files) → Create quantitative assertions
├── Subjective outputs (writing, design) → Skip assertions, focus on qualitative review
└── Mixed outputs → Assertions for verifiable parts, human review for subjective

Evaluation Scope
├── First iteration → 2-3 focused test cases
├── Subsequent iterations → Rerun same cases for comparison
└── Final validation → Expand to 5-10 diverse cases

Improvement Trigger
├── User feedback identifies specific issues → Target those areas
├── Quantitative benchmarks show regression → Analyze root cause
├── No clear feedback but low pass rates → Review transcripts for patterns
└── Consistent success across tests → Move to description optimization
```

## Failure Modes

### Vague Intent Capture (Symptom: Poor skill performance on diverse inputs)
**Detection**: If skill works on test cases but fails when users try different variations of the same task type.
**Diagnosis**: Intent interview was too narrow or focused only on specific examples rather than the general capability.
**Fix**: Restart intent capture with broader questions: "What are all the ways someone might ask for this?" and "What edge cases should we handle?"

### Weak Test Coverage (Symptom: High pass rates but skill doesn't work in practice)
**Detection**: If assertions are passing but user feedback consistently identifies problems, or if skill works only on the exact test prompts.
**Diagnosis**: Test cases don't represent real usage patterns, or assertions are non-discriminating.
**Fix**: Apply discriminating assertion test - would a clearly wrong output also pass this assertion? Rewrite assertions to check specific content only a correct output would have.

### Iteration Tunnel Vision (Symptom: Endless tweaking without improvement)
**Detection**: If 3+ iterations show no meaningful progress, or changes only address test cases without improving general capability.
**Diagnosis**: Over-optimization on specific examples rather than addressing root capability gaps.
**Fix**: Step back and analyze failure patterns across all test cases. Consider fundamental approach changes rather than incremental tweaks.

### Missing Baseline Reality Check (Symptom: False confidence in skill value)
**Detection**: If you're only testing with the skill and never comparing to without-skill performance.
**Diagnosis**: No objective measure of whether the skill actually improves outcomes over baseline Claude.
**Fix**: Always run baseline comparisons (without-skill for new skills, old-version for improvements) to measure actual value added.

### Premature Description Optimization (Symptom: Great triggering, poor execution)
**Detection**: If spending time on description optimization before the skill reliably works on test cases.
**Diagnosis**: Optimizing discoverability before ensuring quality.
**Fix**: Complete the test-improve cycle first. Only optimize descriptions once the skill consistently passes evaluations.

## Worked Examples

### Example 1: Creating a Documentation Skill from Scratch

**Initial Request**: "I keep having to explain our API structure to new developers. Can we make a skill for that?"

**Intent Capture Decision Tree**:
- What should skill do? → Generate API documentation from code/schemas
- When should it trigger? → "document the API", "create API docs", "explain our endpoints"
- Output format? → Markdown with examples and parameter tables
- Test cases needed? → Yes (verifiable structure and content)

**Draft Process**:
- Interview revealed: Uses OpenAPI specs, needs examples for each endpoint, must include authentication section
- Created SKILL.md with sections for parsing specs, generating examples, formatting output
- Identified need for a proposed `scripts/<openapi_parser>.py` based on repetitive parsing logic

**Test Cases Design**:
```json
{
  "skill_name": "api-documenter", 
  "evals": [
    {
      "id": 1,
      "prompt": "Document our user authentication API endpoints",
      "files": ["evals/files/auth_spec.yaml"],
      "expectations": [
        "Output includes authentication section with bearer token example",
        "Each endpoint has request/response examples",
        "Parameter tables include required/optional indicators"
      ]
    }
  ]
}
```

**First Iteration Results**:
- With-skill: Generated docs but missing parameter validation rules
- Without-skill: Basic endpoint list, no examples or detailed parameters
- User feedback: "The examples are great, but developers need to know which fields are validated how"

**Improvement Applied**:
- Added validation parsing to OpenAPI script
- Updated skill instructions to emphasize validation documentation
- Added section on error response formats

**Second Iteration**: Pass rate improved from 60% to 95%, user satisfied with comprehensive coverage.

### Example 2: Improving an Existing Code Review Skill

**Entry Point**: User says "My code review skill keeps missing security issues"

**State Assessment**: Skill exists, has test cases, but low performance on security-related assertions.

**Failure Analysis**:
- Reviewed transcripts: Skill was checking syntax/style but not analyzing security patterns
- Baseline comparison showed without-skill Claude actually caught more security issues
- Root cause: Skill was too focused on formatting, not enough on vulnerability detection

**Improvement Strategy**:
- Added security-focused checklist with OWASP patterns
- Included examples of common vulnerabilities (SQL injection, XSS, auth bypasses)
- Created `references/security_patterns.md` with detailed detection guidance

**Test Case Iteration**:
- Reran existing cases: General improvement across all assertions
- Added security-specific test case with intentionally vulnerable code
- New assertions focused on specific vulnerability detection rather than general "security check"

**Key Learning**: The skill was solving the wrong problem - automating style checks instead of augmenting security analysis. Success required fundamental reframing, not incremental improvement.

## Quality Gates

- [ ] Intent is clearly defined with specific trigger phrases and expected outputs identified
- [ ] Skill description includes both what it does AND when to use it (addresses undertriggering)
- [ ] At least 2 realistic test cases created that represent actual user scenarios
- [ ] If outputs are verifiable, assertions check specific content only correct execution would produce
- [ ] Each test case includes both with-skill and baseline runs for comparison
- [ ] Quantitative results show meaningful improvement over baseline (>20% better pass rate or significant time/quality gains)
- [ ] User feedback from viewer review is incorporated into skill improvements
- [ ] Skill instructions explain the "why" behind requirements, not just rigid rules
- [ ] Any repeated helper scripts across test cases are bundled in scripts/ directory
- [ ] Final iteration achieves user satisfaction or 95%+ pass rate on discriminating assertions

## NOT-FOR Boundaries

**What this skill should NOT handle:**

- **Runtime debugging**: For "my skill crashes when I run it" → Use debugging/troubleshooting skills instead
- **MCP server development**: For "build an MCP server that uses my skill" → Use MCP development skills instead  
- **Plugin architecture**: For "integrate this with VS Code/JetBrains" → Use IDE integration skills instead
- **General coding help**: For "help me write Python code" → Use language-specific programming skills instead
- **Infrastructure deployment**: For "deploy my skill to production" → Use deployment/DevOps skills instead

**When to delegate:**
- For skill hosting/distribution → Use deployment skills
- For testing frameworks beyond basic eval scripts → Use testing framework skills  
- For integration with external APIs → Use API integration skills
- For performance optimization of skill code → Use optimization/profiling skills

This skill focuses exclusively on the skill creation lifecycle: capturing requirements, drafting instructions, creating test cases, running evaluations, and iterating based on results.
## Imported bundle navigation

These preserved source files add depth when their stated topic is needed.

- [agents/analyzer.md](agents/analyzer.md) — Analyzer Agent.
- [agents/comparator.md](agents/comparator.md) — Blind Comparator Agent.
- [agents/grader.md](agents/grader.md) — Grader Agent.
- [eval-viewer/generate_review.py](eval-viewer/generate_review.py) — Files to exclude from output listings.
- [references/description-optimization.md](references/description-optimization.md) — Description Optimization: Detailed Guide.
- [references/eval-patterns.md](references/eval-patterns.md) — Eval Assertion Patterns.
- [references/schemas.md](references/schemas.md) — JSON Schemas.
- [references/troubleshooting.md](references/troubleshooting.md) — Troubleshooting.
- [scripts/__init__.py](scripts/__init__.py) — __init__.py.
- [scripts/aggregate_benchmark.py](scripts/aggregate_benchmark.py) — aggregate_benchmark.py.
- [scripts/generate_report.py](scripts/generate_report.py) — generate_report.py.
- [scripts/improve_description.py](scripts/improve_description.py) — improve_description.py.
- [scripts/package_skill.py](scripts/package_skill.py) — Patterns to exclude when packaging skills..
- [scripts/quick_validate.py](scripts/quick_validate.py) — quick_validate.py.
- [scripts/run_eval.py](scripts/run_eval.py) — run_eval.py.
- [scripts/run_loop.py](scripts/run_loop.py) — run_loop.py.
- [scripts/utils.py](scripts/utils.py) — utils.py.