---
title: Does Your Skill Actually Help? A Practical Guide to Agent Skill Evals
description: >-
  Build a small release-summary evaluation with negative controls, traces, hard
  checks, and baselines, and understand where agent-skills-eval fits.
canonical_url: 'https://onevium.com/blog/skill-evals-practical-guide'
language: en
published_at: '2026-09-14'
publisher: Onevium
---
# Does Your Skill Actually Help? A Practical Guide to Agent Skill Evals

Build a small release-summary evaluation with negative controls, traces, hard checks, and baselines, and understand where agent-skills-eval fits.

## How do you know a Skill change helped?

Longer instructions are easy to write. Evidence of improvement takes more work. A convincing answer can hide a missed source, an unintended edit, or a failure that appears only when the request is phrased differently.

OpenAI’s January 22, 2026 article proposes four evaluation dimensions: Outcome, Process, Style, and Efficiency. Its demo-app example includes test-04, an adjacent request to add Tailwind to an existing application. This is a negative control designed to catch unwanted scaffolding, not a report of an observed incident.

Our original exercise uses a smaller task: a release-brief Skill that reads synthetic release notes and returns two summaries. The downloadable starter separates three questions: is the answer acceptable, was the Skill used correctly, and did adding the Skill help?

- [OpenAI: Testing Agent Skills Systematically with Evals](https://developers.openai.com/blog/eval-skills)

## Write the acceptance table first

The fictional release is version 0.4.0. CSV export and upload retries have shipped; offline editing is still planned. The Skill may read the notes and summarize them. It may not edit files or publish a message.

Turn that scope into checks. Code handles exact constraints. A reviewer handles meaning. When a run fails, the distinction tells you what to investigate.

Original example: acceptance criteria for release-brief

| Dimension  | Requirement                                                                                       | Evidence                                                   |
| ---------- | ------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- |
| Outcome    | Version 0.4.0; exactly export and retry                                                           | Parse the final JSON; review summary accuracy separately   |
| Process    | Read the right source; leave files untouched; do not summarize releases for terminology questions | Inspect the trace and command results; compare file hashes |
| Style      | Short summaries; never claim offline editing shipped                                              | Check the schema, then review meaning                      |
| Efficiency | No pointless loops or unrelated calls                                                             | Retain completed commands, turn usage, and wall time       |

## Understand the evaluation loop

A case includes both the request and the starting workspace. Capture the execution trace, final answer, and resulting files. The final statement “done” is not enough to establish what happened.

The starter follows this order: collect evidence, apply hard checks, review process and quality, then preserve failures as regression cases.

![Acceptance criteria and fixed inputs lead to an agent run, evidence capture, hard checks and review, then regression cases.](https://onevium.com/assets/blog/skill-evals-loop-en.svg)

Original diagram: a run produces evidence; defined checks produce an acceptance decision.

## Start with four cases, then grow the set

The download contains four cases to make the loop easy to understand. Add a few variations in each category to build an initial set of 10–20 cases. Treat this as a starting size, not a reliability guarantee. Real failures and closely related requests are particularly useful additions.

A negative case must specify the behavior to avoid. A terminology question should not read local release notes. A blocked write attempt also fails this read-only task, even if the sandbox prevents any final file change. For a Skill that legitimately edits files, compare changes against an explicit scope instead of requiring an unchanged directory.

The starter’s four requests

| Case     | Request                                                | Question being tested                                                      |
| -------- | ------------------------------------------------------ | -------------------------------------------------------------------------- |
| explicit | Use $release-brief to summarize this project’s release | Does direct invocation work?                                               |
| implicit | Summarize shipped changes as JSON                      | Does the host select the Skill without its name?                           |
| context  | Prepare a release handoff for support                  | Does added business context cause drift?                                   |
| negative | Define release notes; do not read local release files  | Does it answer the concept question without entering the release workflow? |

## Prepare a small, isolated exercise

Download and unzip the starter, then open a terminal in skill-evals-starter. You need Node.js 20 or later and an installed, authenticated Codex CLI. The offline test command makes no model calls. Running run.mjs uses your Codex configuration and quota.

The package includes synthetic RELEASE\_NOTES.md, a Skill, four cases, a JSON Schema, a runner, and a grader. The Skill lives in .agents/skills/release-brief/SKILL.md, following the current documentation. Each run uses a fresh temporary workspace with an explicitly requested read-only sandbox. Artifacts are written outside that workspace so logging does not count as an agent edit.

Your usual configuration and global Skills can still affect the run. Use a controlled evaluation environment for formal comparisons and check for a same-named global Skill. Removing a local Skill alone does not disable every copy.

### Check the environment, then run one case

```text
node --version
codex --version
codex exec --help

# Offline grader tests; no model calls
node --test grade.test.mjs

# One real model run
node run.mjs explicit
```

- [Download the original starter · ZIP](https://onevium.com/assets/downloads/skill-evals-starter.zip)
- [Codex: Skill discovery and locations](https://learn.chatgpt.com/docs/build-skills)

## Read the trace, answer, and file state

The terminal prints the artifact directory. Start with grade.json, then inspect trace.jsonl and answer.txt. meta.json records the request and CLI version; stderr.txt preserves diagnostics. Failed and timed-out runs retain whatever evidence was captured.

The runner constructs the command below with real paths and the selected prompt. --json emits JSONL events; -o saves the final answer separately. Positive cases also share an output schema. The negative case deliberately does not use a release-summary schema, which would steer it toward the wrong workflow.

The without-Skill baseline uses the same positive-case schema. Correct formatting is therefore a basic check, not evidence that the Skill added value.

### Command shape inside the runner; it supplies the paths

```text
codex exec --json --sandbox read-only \
  --skip-git-repo-check \
  -C <temporary-workspace> \
  --output-schema <absolute-schema-path> \
  -o <absolute-answer-path> \
  '<case prompt>'
```

- [Codex: non-interactive execution and structured output](https://learn.chatgpt.com/docs/non-interactive-mode)

## Hard checks are only one part of acceptance

The starter checks successful completion, unchanged workspace contents, and basic output constraints. Positive cases require the correct version and two change IDs. The negative case only checks for a nonempty answer without the local version. Semantic accuracy and a useful definition still need review.

Process requires its own evidence. This starter does not invent a universal skill\_invoked event or equate a mention of SKILL.md with compliance. Review host loading information where available, actual commands, and their results. Keep the verdict pending when the trace cannot establish it.

A command containing npm install does not prove installation succeeded. A zero exit code does not prove an application works. A working application does not prove that edits stayed in scope. For a scaffolding Skill, add target-directory checks, a build, and relevant browser checks.

### Illustrative result: hard checks pass; review remains

```text
{
  "checks": {
    "run_completed": true,
    "workspace_unchanged": true,
    "output_contract": true
  },
  "hard_pass": true,
  "process_review_required": true,
  "quality_review_required": true
}
```

## Give the judge specific questions

Provide the original notes alongside the two summaries. Ask whether either shipped feature is misrepresented, whether offline editing is incorrectly promised, and whether each summary clearly explains a change. Require a verdict and supporting evidence for each question, not just an overall score.

If a model assists with review, pin its model, grading prompt, and output schema. Missing fields or an unusable response should fail the review step. Calibrate it against a few human judgments, including an answer that tries to instruct the judge to award full marks.

A high quality score must not override a failed hard constraint. File naming, JSON keys, and other directly checkable conventions should stay in code rather than becoming matters of model opinion.

## Compare the Skill against a fair baseline

Run the remaining cases, then pair implicit with its without-Skill baseline. Both receive identical release notes and the same output schema; only the local Skill differs. Do not use the explicit Skill invocation as a without-Skill control.

Hold the model, CLI version, permissions, source data, and initial workspace fixed. Record the Skill commit or content hash. EVAL\_MODEL can pin a model available to your account. Repeat important boundary cases and report correctness, false activation, and tool behavior separately.

Treat token counts and elapsed time as diagnostic signals. Saving tokens by skipping a required source is not an improvement. Missing usage is not zero usage. The starter deduplicates completed commands by item ID instead of counting started and completed events twice.

### Each command creates a fresh workspace and calls the model

```text
node run.mjs implicit
node run.mjs context
node run.mjs negative

# Compare this with the implicit run above
node run.mjs implicit --without-skill
```

## Where agent-skills-eval fits

agent-skills-eval is an unofficial project maintained by darkrishabh. In the reviewed commit, b60eebe, with\_skill injects Skill content into the request; without\_skill omits it. The default Provider calls an OpenAI-compatible chat endpoint and returns tool\_calls for assertions. That path does not execute those tools and feed their results back into an agent loop.

This supports comparisons of instructed outputs and tool-call arguments. Testing actual Codex discovery, workspace edits, and build outcomes requires a real execution environment and corresponding checks.

One limitation matters for baseline fairness: this commit reads case attachments only for with\_skill. If the task depends on attachments, the two modes receive different information. Inspect the saved requests or adapt the runner before attributing an improvement to the Skill.

![Injected-instruction output evaluation and real agent workflow evaluation cover different layers, from requests and responses to discovery, execution and filesystem checks.](https://onevium.com/assets/blog/skill-evals-layers-en.svg)

Original diagram: choose a tool after deciding which behavior you need to measure.

- [agent-skills-eval repository](https://github.com/darkrishabh/agent-skills-eval)
- [Reviewed run-eval.ts](https://github.com/darkrishabh/agent-skills-eval/blob/b60eebe3c6edaa917a284e13b9b0e9fa00f1c957/src/run-eval.ts)
- [Reviewed default Provider](https://github.com/darkrishabh/agent-skills-eval/blob/b60eebe3c6edaa917a284e13b9b0e9fa00f1c957/src/openai-compatible-provider.ts)

## Make regressions actionable in CI

Version the Skill, cases, fixtures, and grading code together. Keep traces, answers, file differences, and grading results for each change. Separate infrastructure errors from task failures and quality failures so the team knows whether to retry or revise the Skill.

The starter exits nonzero when a hard check fails. Exit code zero still requires process and quality review. Before using it as a CI gate, implement those checks, configure authentication and a budget, and retain the evidence needed to investigate failures.

This is a tested starting point, not a claimed model benchmark. The grader passed 15 offline tests, including truncated traces, incorrect versions, and workspace changes. We did not run a live model evaluation or establish full acceptance of the example Skill.

Whenever a failure matters, preserve the input and a check that catches it. The Skill can change while the team’s definition of correct behavior remains explicit.

Source: [OpenAI — Testing Agent Skills Systematically with Evals](https://developers.openai.com/blog/eval-skills)
