🧭 Decision Guide
Why trending now: 无法从材料判断
Try it if you
-
You maintain a skill following the Agent Skills specification and need Tier 1 static quality checks.The README section “Three-tier overview” says Tier 1 checks safety, structure, and harness compatibility.
-
You want to check changed skills automatically on GitHub pull requests.The README section “Get started” provides `skillevaluator ci init` and says it opens a setup PR.
-
You need to compare agent task results with and without a skill and can provide a provider, agent, and Docker sandbox.The README says it compares runs with and without a skill; “Three-tier overview” lists Tier 3's additional requirements.
Skip it if you
-
You need a production evaluation service with an SLA or NVIDIA enterprise support.The README section “Support” explicitly says the support level is Experimental, with no SLA or NVIDIA enterprise support entitlement.
-
Your environment cannot run Docker and your target is Tier 3 Live Evaluation.The README section “Three-tier overview” says Tier 3 uses a running Docker sandbox by default.
-
You only need basic security scanning and do not need the Agent Skill Tier 1–3 pipeline.The README describes SkillSpector as specialized skill security scanning and as a Tier 1 component used by SkillEvaluator.
Requirements
- Standalone installation uses `pip install skillevaluator`; source checkout uses Git and uv, which provisions Python 3.13.
- Full Tier 1 static coverage needs external scanners; Tier 2 needs embeddings and chat; Tier 3 needs provider access, a supported agent, and a running Docker sandbox.
- NVIDIA Build `rubric-eval` requires NVIDIA Developer Program eligibility and `NVIDIA_API_KEY`.
- Tier 1 fetches pinned SkillSpector and Gitleaks releases; Semgrep is optional.
First step (verbatim from README)
pip install skillevaluator
Watch out
-
`--no-llm` does not remove scanner requirements; an unavailable scanner produces INCOMPLETE.README section “Three-tier overview”
-
`doctor` checks readiness and does not prove inference success.README section “Run the combined pipeline”
-
Tier 3 autopilot only reuses sources or creates one case with limited coverage.README section “Three-tier overview”
-
Generated templates omit unauthored negatives; `evals/EVAL.md` and `--prompt` inputs matter.README section “Run the combined pipeline”
Alternatives
-
SkillSpector:When you only need specialized skill security scanning rather than SkillEvaluator's Tier 2 deduplication or Tier 3 agent evaluation.README section “Three-tier overview”
-
Harbor:When you need to use the open-source agent evaluation framework and sandbox execution directly rather than SkillEvaluator's integrated CLI.README section “Three-tier overview”
Not stated in the README
- The README does not specify which agent runtimes and versions Tier 3 supports.
- The README does not provide exact pricing or quotas for NVIDIA Build, model calls, or managed sandboxes.
- The README does not provide runtime, resource, or concurrency data for different skill sizes.
- The README does not identify the embedding and chat models used by Tier 2.
💡 Deep Analysis
6
Yes
I only have Git and uv, with no API key, GPU, or Docker. I currently want a deterministic Skill quality score. Can I use SkillEvaluator directly?
Yes, the README explicitly supports Tier 1 quality checks without an LLM, API key, GPU, or Docker, but that does not complete security or agent-impact evaluation.
- The Quickstart requires Git and uv; uv provisions Python 3.13 and dependencies.
quality-checkproduces a deterministic quality score and findings. The example scores 86.2/100, exits with code 0, and writes a JSON report.- The no-LLM mode may still require SkillSpector and Gitleaks. If a scanner cannot run, the result is
INCOMPLETE, so no API key does not mean no external-tool requirements. - The README states that no-LLM execution covers static quality only; security requires scanners and agent impact requires live evaluation.
- Quickstart: `Run without an LLM`; “No API key, scanner, GPU or Docker is needed”
- Quickstart: `uv run --python 3.13 skillevaluator quality-check ...`
- Quickstart: the example scores “86.2/100 (B)” and saves `skillevaluator-quality.json`
- Three-tier overview: `--no-llm` does not remove scanner requirements
uv run --python 3.13 skillevaluator quality-check \
src/skillevaluator/tier3/reference_skills/text-analyzer \
-r cli -r json -o reports/first-quality
Yes
I own the Skill security gate. I require at least SkillSpector and Gitleaks, and I want to run static checks without an LLM first. Can Tier 1 meet this constraint?
Yes, provided that SkillSpector and Gitleaks are available as runnable external scanners; Tier 1’s no-LLM mode does not bypass scanner requirements.
- The README defines Tier 1 around safety, formatting, and harness compatibility, making it suitable for a static quality gate.
- When missing, SkillEvaluator fetches pinned and verified releases of SkillSpector and Gitleaks; Semgrep is optional.
--no-llmdisables model checks, but the README specifically says it does not remove scanner requirements.- An unavailable scanner becomes
INCOMPLETEand causes a nonzero exit. The security gate must therefore distinguish “scan passed” from “scan incomplete” rather than relying only on a quality score.
- Three-tier overview: Tier 1 checks safety, well-formedness, and harness compatibility
- Three-tier overview: Tier 1 fetches pinned, verified SkillSpector and Gitleaks releases; Semgrep is optional
- Three-tier overview: `--no-llm` does not remove scanner requirements
- Three-tier overview: an unavailable scanner is `INCOMPLETE` and exits nonzero
uv run skillevaluator tier1 ./my-skill --no-llm
Yes
I maintain a Skill repository using GitHub Pull Requests. My team wants Tier 1 checks in CI and wants failing Skills to block merges. Is SkillEvaluator suitable?
Yes, because the README provides a GitHub CI bootstrap command and Tier 1 is the default gate for validate.
- Running
skillevaluator ci initopens a setup PR, which fits an existing Pull Request workflow. - Tier 1 checks whether a Skill is safe, well formed, and harness-compatible, and
validate --tiers 1requires no API key. - CLI and JSON reports can be retained by CI; however, missing scanners produce
INCOMPLETEand a nonzero exit code. - Tier 2 can also block deduplication issues, while Tier 3 blocks only when
--block-on-agent-evalis enabled, so the gate scope is configurable.
- Get started: `skillevaluator ci init`; “Check changed skills on every pull request”
- Three-tier overview: Tier 1 asks whether the Skill is “safe, well-formed and harness-compatible”
- Run the combined pipeline: “Tier 1 gates validate”; “Tier 3 gates only with --block-on-agent-eval”
- Three-tier overview: an unavailable scanner is `INCOMPLETE` and exits nonzero
skillevaluator ci init
Yes
I need to prove whether a Skill actually improves task results in Claude Code or Codex, and I can run a Docker sandbox and a supported agent. Is Tier 3 worth integrating into release evaluation?
Yes. Tier 3 specifically compares task outcomes with and without the Skill, making it closer to release behavior than static document scoring; however, it requires a complete runtime environment and has cost and nondeterminism concerns.
- The README defines Tier 3 as “Does the skill help the agent?” and uses Harbor for sandboxed agent runs.
- Docker is the default sandbox, with provider access, a supported agent, and a running sandbox required.
validatedoes not block on Tier 3 by default;--block-on-agent-evalmust be configured to make agent evaluation a release gate.- Model and managed-sandbox usage can incur charges. Autopilot may only reuse a source or create one Skill-grounded case, so coverage can be limited.
- Three-tier overview: Tier 3 asks “Does the skill help the agent?”; Harbor powers sandboxed agent runs
- Three-tier overview: it requires a provider, supported agent, and running sandbox, with Docker by default
- Run the combined pipeline: Tier 3 blocks only with `--block-on-agent-eval`
- Run the combined pipeline: model and managed-sandbox usage may incur charges; autopilot coverage is limited
uv run --python 3.13 --extra all skillevaluator doctor --env-mode docker
It depends
I maintain an organization-wide Skill catalog that keeps growing. I want to use embeddings and chat models to find duplicated guidance across Skills. Is Tier 2 suitable, and what should I watch for?
It depends. Tier 2 is suitable for detecting semantic overlap, but it cannot by itself prove that two Skills conflict functionally or cannot coexist in a business context.
- The README defines Tier 2 as “Does guidance repeat or overlap?” and requires embeddings and chat models.
- Cross-Skill comparison requires
--catalog ./skill-catalog.json; without a catalog, analysis is more local and limited. - Tier 2 reports whether inter-Skill comparison actually ran, while semantic similarity still needs interpretation against organizational rules.
- In the combined pipeline, Tier 2 can block deduplication issues by default, but
--no-block-on-dedupdisables that gate. It is therefore better treated as a catalog-governance signal than as the sole release decision.
- Three-tier overview: Tier 2 asks “Does guidance repeat or overlap?”
- Three-tier overview: it requires “Embeddings and chat”; `--catalog ./skill-catalog.json` enables inter-Skill comparison
- Three-tier overview: Tier 2 reports whether inter-Skill comparison ran
- Run the combined pipeline: Tier 2 gates unless `--no-block-on-dedup`
uv run --extra tier2 skillevaluator tier2 ./my-skill
It depends
We have substantive `evals/EVAL.md` guidance and a prompt file containing task requests and outcomes, but few human-retained traces. Can SkillEvaluator generate a Tier 3 dataset for us?
It depends. The README supports generating a Tier 3 dataset from EVAL.md or a prompt file, but coverage depends on whether the inputs actually define tasks, outcomes, and success criteria.
- Standalone generation requires substantive
evals/EVAL.md, a--promptfile with task requests and outcomes, or matching retained traces. --forcereplaces the autopilot starter, while--refinerefines it;--grounding skillpermits a Skill-only starter.- The README explicitly says to review cases and that templates omit negative cases not authored by the user, so an automatically generated set is not complete negative coverage.
- A dataset can be generated without many real traces, but its representativeness as a release gate remains uncertain.
- Run the combined pipeline: standalone generation accepts `evals/EVAL.md`, `--prompt`, or retained traces
- Run the combined pipeline: `--force` replaces the autopilot starter; `--refine` refines it; `--grounding skill` is supported
- Run the combined pipeline: “Review cases; templates omit unauthored negatives”
- Project insight: generated dataset quality depends on real tasks and clearly defined outcomes
uv run --python 3.13 --extra tier3 skillevaluator tier3 create-eval-dataset ./my-skill --full --prompt eval-guidance.md --force
✨ Highlights
-
Tiers 1–3 cover validation, deduplication, and live agent evaluation
-
Tier 1 produces an 86.2/100 example score without an API key
-
Supports GitHub CI, GitLab CI, and Claude Code
-
Tier 3 requires a supported agent and Docker sandbox by default
🔧 Engineering
-
Use validate to run the Tier 1, 2, and 3 evaluation pipeline
-
Use SkillSpector, Gitleaks, and optional Semgrep for security checks
-
Use rubric-eval and NVIDIA Build for model-backed feedback
⚠️ Risks
-
The project is Experimental, with no SLA or NVIDIA enterprise support
-
Tier 3 requires provider access, an agent, and a Docker sandbox
-
Model and managed-sandbox usage can incur charges
-
A missing scanner produces INCOMPLETE and a nonzero exit
👥 For who?
-
Teams maintaining skills under the Agent Skills specification
-
Developers needing skill checks on GitHub pull requests
-
Teams using Claude Code, Codex, or compatible agents