nvidia/skillevaluator
A three-stage checkup for the 'skill folders' that plug new abilities into AI agents
SkillEvaluator is an open-source tool that checks agent skills, which are folders of instructions and files that extend AI agents. It runs a validation stage for safety and formatting, a deduplication stage to catch overlap with existing skills, and a live evaluation stage that tests whether a skill actually helps an agent perform. Built by NVIDIA under Apache 2.0, it's currently labeled as experimental, community-supported software.
- agent-evaluation
- agent-security
- agent-skills
- agentic-ai
- benchmark
- claude-code
- codex
- evaluate
- evaluation
- security-scanner
- skill-eval
- skill-evals
What it does
- A skill is a folder containing a SKILL.md file plus supporting materials that extend an AI agent, following the Agent Skills specification
- Tier 1 validation checks schema, personal data (PII), licensing, quality, and security issues, and several of these checks run without any API key
- Tier 2 deduplication uses embeddings, a technique that turns text into numeric vectors, to measure how much a new skill overlaps with existing ones
- Tier 3 live evaluation runs the skill with a real agent inside a Docker, local, or cloud sandbox to see how it changes agent behavior, and can auto-generate its own test dataset
- Each tier can be run independently, and the tool integrates with other open-source projects like NVIDIA's SkillSpector security scanner and the Harbor agent evaluation framework
Why it matters
As teams start attaching arbitrary skill folders to AI agents, there's a real risk of hidden security flaws, leaked personal data, or redundant instructions that hurt performance, so a standardized pre-deployment check matters. This gives skill authors and agent-pipeline teams a concrete way to vet quality and safety before shipping.
Terms in this repo
- Agent Skill · A folder of instructions and files, including a SKILL.md, that extends what an AI agent can do
- Quality gate · A checkpoint that blocks progress unless certain criteria are met
- Embedding · A technique that converts text meaning into numeric vectors so similarity can be measured
- Sandbox · An isolated environment for running code safely without affecting the real system
- PII scan · A check for personally identifiable information, like names or addresses, hidden in a document
Repository description (English)
Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior.
Open on GitHubProject siteCoverage
Trending repos
- cathrynlavery/diagram-designA skill that makes AI coding tools draw magazine-quality diagrams instead of generic rounded boxes
- public-apis/public-apisA giant crowd-curated directory of free APIs for developers
- semantica-agi/semanticaAn open-source graph infrastructure that lets AI agents show their work, not just their answers
- cactus-compute/needleA 14MB AI model small enough to run tool-calling on a phone or watch, without internet
- unslothai/unslothA desktop app that lets you run and train AI models on your own computer, no coding required
- macro-inc/macroAn all-in-one workspace where email, chat, docs, tasks, and CRM are cross-linked and share one AI memory
- harry0703/MoneyPrinterTurboAn open-source tool that turns a single topic or keyword into a finished short video, complete with script, footage, subtitles, and music
- basecamp/omarchyA ready-made, opinionated Linux setup built by DHH
Latest from METAL LAB
- Hermes Agent gets a dedicated remote computer, priced at a few dollars a month
- ElevenLabs launches 'v3 Conversational' voice model for real-time dialogue
- Terence Tao: "AI Could Trigger Math's Biggest Crisis Since Gödel"
- Modly: Open-Source App Converts Photos to 3D Models Using Only a GPU
- Grok leaks conversations, location data via commands hidden in ciphertext