SkillEvaluator: Tiered evaluation for AI agent skill quality and impact
For Agent Skill developers, it checks quality, security, and live impact by comparing agent runs with and without a skill.
GitHub NVIDIA/SkillEvaluator Updated 2026-11-16 Branch main Stars 1.7K Forks 229
Python AI agent skill evaluation SkillSpector Docker

🧭 Decision Guide

Try it if you

  • You maintain a skill following the Agent Skills specification and need Tier 1 static quality checks.
    The README section “Three-tier overview” says Tier 1 checks safety, structure, and harness compatibility.
  • You want to check changed skills automatically on GitHub pull requests.
    The README section “Get started” provides `skillevaluator ci init` and says it opens a setup PR.
  • You need to compare agent task results with and without a skill and can provide a provider, agent, and Docker sandbox.
    The README says it compares runs with and without a skill; “Three-tier overview” lists Tier 3's additional requirements.

Skip it if you

  • You need a production evaluation service with an SLA or NVIDIA enterprise support.
    The README section “Support” explicitly says the support level is Experimental, with no SLA or NVIDIA enterprise support entitlement.
  • Your environment cannot run Docker and your target is Tier 3 Live Evaluation.
    The README section “Three-tier overview” says Tier 3 uses a running Docker sandbox by default.
  • You only need basic security scanning and do not need the Agent Skill Tier 1–3 pipeline.
    The README describes SkillSpector as specialized skill security scanning and as a Tier 1 component used by SkillEvaluator.

Requirements

  • Standalone installation uses `pip install skillevaluator`; source checkout uses Git and uv, which provisions Python 3.13.
  • Full Tier 1 static coverage needs external scanners; Tier 2 needs embeddings and chat; Tier 3 needs provider access, a supported agent, and a running Docker sandbox.
  • NVIDIA Build `rubric-eval` requires NVIDIA Developer Program eligibility and `NVIDIA_API_KEY`.
  • Tier 1 fetches pinned SkillSpector and Gitleaks releases; Semgrep is optional.

First step (verbatim from README)

pip install skillevaluator

Watch out

  • `--no-llm` does not remove scanner requirements; an unavailable scanner produces INCOMPLETE.
    README section “Three-tier overview”
  • `doctor` checks readiness and does not prove inference success.
    README section “Run the combined pipeline”
  • Tier 3 autopilot only reuses sources or creates one case with limited coverage.
    README section “Three-tier overview”
  • Generated templates omit unauthored negatives; `evals/EVAL.md` and `--prompt` inputs matter.
    README section “Run the combined pipeline”

Alternatives

  • SkillSpector:When you only need specialized skill security scanning rather than SkillEvaluator's Tier 2 deduplication or Tier 3 agent evaluation.
    README section “Three-tier overview”
  • Harbor:When you need to use the open-source agent evaluation framework and sandbox execution directly rather than SkillEvaluator's integrated CLI.
    README section “Three-tier overview”

Not stated in the README

  • The README does not specify which agent runtimes and versions Tier 3 supports.
  • The README does not provide exact pricing or quotas for NVIDIA Build, model calls, or managed sandboxes.
  • The README does not provide runtime, resource, or concurrency data for different skill sizes.
  • The README does not identify the embedding and chat models used by Tier 2.

💡 Deep Analysis

6
Yes I only have Git and uv, with no API key, GPU, or Docker. I currently want a deterministic Skill quality score. Can I use SkillEvaluator directly?
For: An individual developer without a GPU who wants to validate Skill documentation first, using Python 3.13, uv, and Git

Yes, the README explicitly supports Tier 1 quality checks without an LLM, API key, GPU, or Docker, but that does not complete security or agent-impact evaluation.

  • The Quickstart requires Git and uv; uv provisions Python 3.13 and dependencies.
  • quality-check produces a deterministic quality score and findings. The example scores 86.2/100, exits with code 0, and writes a JSON report.
  • The no-LLM mode may still require SkillSpector and Gitleaks. If a scanner cannot run, the result is INCOMPLETE, so no API key does not mean no external-tool requirements.
  • The README states that no-LLM execution covers static quality only; security requires scanners and agent impact requires live evaluation.
  • Quickstart: `Run without an LLM`; “No API key, scanner, GPU or Docker is needed”
  • Quickstart: `uv run --python 3.13 skillevaluator quality-check ...`
  • Quickstart: the example scores “86.2/100 (B)” and saves `skillevaluator-quality.json`
  • Three-tier overview: `--no-llm` does not remove scanner requirements
uv run --python 3.13 skillevaluator quality-check \
  src/skillevaluator/tier3/reference_skills/text-analyzer \
  -r cli -r json -o reports/first-quality
Not stated in the README:The README does not specify scanner download sizes or runtime on a GPU-less machine.;It does not say whether a custom Skill can receive a fully comparable score using only static rules when scanners are unavailable.
Yes I own the Skill security gate. I require at least SkillSpector and Gitleaks, and I want to run static checks without an LLM first. Can Tier 1 meet this constraint?
For: A test and security engineer responsible for Skill security gates, using SkillSpector and Gitleaks with optional Semgrep

Yes, provided that SkillSpector and Gitleaks are available as runnable external scanners; Tier 1’s no-LLM mode does not bypass scanner requirements.

  • The README defines Tier 1 around safety, formatting, and harness compatibility, making it suitable for a static quality gate.
  • When missing, SkillEvaluator fetches pinned and verified releases of SkillSpector and Gitleaks; Semgrep is optional.
  • --no-llm disables model checks, but the README specifically says it does not remove scanner requirements.
  • An unavailable scanner becomes INCOMPLETE and causes a nonzero exit. The security gate must therefore distinguish “scan passed” from “scan incomplete” rather than relying only on a quality score.
  • Three-tier overview: Tier 1 checks safety, well-formedness, and harness compatibility
  • Three-tier overview: Tier 1 fetches pinned, verified SkillSpector and Gitleaks releases; Semgrep is optional
  • Three-tier overview: `--no-llm` does not remove scanner requirements
  • Three-tier overview: an unavailable scanner is `INCOMPLETE` and exits nonzero
uv run skillevaluator tier1 ./my-skill --no-llm
Not stated in the README:The README does not list exact compatibility matrices for SkillSpector, Gitleaks, and Semgrep versions.;It does not describe enterprise-proxy support, offline caching, or custom scanner rules.;It does not explain how findings are categorized by false positive status, severity, or acceptable risk level.
Yes I maintain a Skill repository using GitHub Pull Requests. My team wants Tier 1 checks in CI and wants failing Skills to block merges. Is SkillEvaluator suitable?
For: A DevOps engineer maintaining a GitHub Pull Request workflow who must check every changed Agent Skill

Yes, because the README provides a GitHub CI bootstrap command and Tier 1 is the default gate for validate.

  • Running skillevaluator ci init opens a setup PR, which fits an existing Pull Request workflow.
  • Tier 1 checks whether a Skill is safe, well formed, and harness-compatible, and validate --tiers 1 requires no API key.
  • CLI and JSON reports can be retained by CI; however, missing scanners produce INCOMPLETE and a nonzero exit code.
  • Tier 2 can also block deduplication issues, while Tier 3 blocks only when --block-on-agent-eval is enabled, so the gate scope is configurable.
  • Get started: `skillevaluator ci init`; “Check changed skills on every pull request”
  • Three-tier overview: Tier 1 asks whether the Skill is “safe, well-formed and harness-compatible”
  • Run the combined pipeline: “Tier 1 gates validate”; “Tier 3 gates only with --block-on-agent-eval”
  • Three-tier overview: an unavailable scanner is `INCOMPLETE` and exits nonzero
skillevaluator ci init
Not stated in the README:The README does not specify the exact GitHub permissions or organization branch-protection settings required after the setup PR is created.;It does not describe CI runtime or retry behavior when scanner downloads fail.
Yes I need to prove whether a Skill actually improves task results in Claude Code or Codex, and I can run a Docker sandbox and a supported agent. Is Tier 3 worth integrating into release evaluation?
For: A test engineer responsible for release quality of Claude Code or Codex Skills, with Docker sandbox access and a supported agent

Yes. Tier 3 specifically compares task outcomes with and without the Skill, making it closer to release behavior than static document scoring; however, it requires a complete runtime environment and has cost and nondeterminism concerns.

  • The README defines Tier 3 as “Does the skill help the agent?” and uses Harbor for sandboxed agent runs.
  • Docker is the default sandbox, with provider access, a supported agent, and a running sandbox required.
  • validate does not block on Tier 3 by default; --block-on-agent-eval must be configured to make agent evaluation a release gate.
  • Model and managed-sandbox usage can incur charges. Autopilot may only reuse a source or create one Skill-grounded case, so coverage can be limited.
  • Three-tier overview: Tier 3 asks “Does the skill help the agent?”; Harbor powers sandboxed agent runs
  • Three-tier overview: it requires a provider, supported agent, and running sandbox, with Docker by default
  • Run the combined pipeline: Tier 3 blocks only with `--block-on-agent-eval`
  • Run the combined pipeline: model and managed-sandbox usage may incur charges; autopilot coverage is limited
uv run --python 3.13 --extra all skillevaluator doctor --env-mode docker
Not stated in the README:The README does not list a complete compatibility matrix for Claude Code, Codex, and other agents.;It does not specify average Tier 3 runtime, model-call counts, or Docker resource requirements.;It does not define statistical significance or repetition mechanisms for with-Skill versus without-Skill comparisons.
It depends I maintain an organization-wide Skill catalog that keeps growing. I want to use embeddings and chat models to find duplicated guidance across Skills. Is Tier 2 suitable, and what should I watch for?
For: A platform engineer maintaining an organization-wide Skill catalog who needs to identify duplicated guidance

It depends. Tier 2 is suitable for detecting semantic overlap, but it cannot by itself prove that two Skills conflict functionally or cannot coexist in a business context.

  • The README defines Tier 2 as “Does guidance repeat or overlap?” and requires embeddings and chat models.
  • Cross-Skill comparison requires --catalog ./skill-catalog.json; without a catalog, analysis is more local and limited.
  • Tier 2 reports whether inter-Skill comparison actually ran, while semantic similarity still needs interpretation against organizational rules.
  • In the combined pipeline, Tier 2 can block deduplication issues by default, but --no-block-on-dedup disables that gate. It is therefore better treated as a catalog-governance signal than as the sole release decision.
  • Three-tier overview: Tier 2 asks “Does guidance repeat or overlap?”
  • Three-tier overview: it requires “Embeddings and chat”; `--catalog ./skill-catalog.json` enables inter-Skill comparison
  • Three-tier overview: Tier 2 reports whether inter-Skill comparison ran
  • Run the combined pipeline: Tier 2 gates unless `--no-block-on-dedup`
uv run --extra tier2 skillevaluator tier2 ./my-skill
Not stated in the README:The README does not provide catalog size limits, embedding-model choices, or similarity thresholds.;It does not describe Tier 2 accuracy across languages, long documents, or structured reference files.
It depends We have substantive `evals/EVAL.md` guidance and a prompt file containing task requests and outcomes, but few human-retained traces. Can SkillEvaluator generate a Tier 3 dataset for us?
For: A small AI-agent team generating Skill evaluation datasets with EVAL.md and prompt files but few human-authored traces

It depends. The README supports generating a Tier 3 dataset from EVAL.md or a prompt file, but coverage depends on whether the inputs actually define tasks, outcomes, and success criteria.

  • Standalone generation requires substantive evals/EVAL.md, a --prompt file with task requests and outcomes, or matching retained traces.
  • --force replaces the autopilot starter, while --refine refines it; --grounding skill permits a Skill-only starter.
  • The README explicitly says to review cases and that templates omit negative cases not authored by the user, so an automatically generated set is not complete negative coverage.
  • A dataset can be generated without many real traces, but its representativeness as a release gate remains uncertain.
  • Run the combined pipeline: standalone generation accepts `evals/EVAL.md`, `--prompt`, or retained traces
  • Run the combined pipeline: `--force` replaces the autopilot starter; `--refine` refines it; `--grounding skill` is supported
  • Run the combined pipeline: “Review cases; templates omit unauthored negatives”
  • Project insight: generated dataset quality depends on real tasks and clearly defined outcomes
uv run --python 3.13 --extra tier3 skillevaluator tier3 create-eval-dataset ./my-skill --full --prompt eval-guidance.md --force
Not stated in the README:The README does not define the minimum content or case count for a “substantive EVAL.md”.;It does not specify the dataset-generation model, per-case cost, or recovery behavior after generation failures.;It does not explain how to quantify coverage of production tasks.

✨ Highlights

  • Tiers 1–3 cover validation, deduplication, and live agent evaluation
  • Tier 1 produces an 86.2/100 example score without an API key
  • Supports GitHub CI, GitLab CI, and Claude Code
  • Tier 3 requires a supported agent and Docker sandbox by default

🔧 Engineering

  • Use validate to run the Tier 1, 2, and 3 evaluation pipeline
  • Use SkillSpector, Gitleaks, and optional Semgrep for security checks
  • Use rubric-eval and NVIDIA Build for model-backed feedback

⚠️ Risks

  • The project is Experimental, with no SLA or NVIDIA enterprise support
  • Tier 3 requires provider access, an agent, and a Docker sandbox
  • Model and managed-sandbox usage can incur charges
  • A missing scanner produces INCOMPLETE and a nonzero exit

👥 For who?

  • Teams maintaining skills under the Agent Skills specification
  • Developers needing skill checks on GitHub pull requests
  • Teams using Claude Code, Codex, or compatible agents