How do you know a Skill change helped?
Longer instructions are easy to write. Evidence of improvement takes more work. A convincing answer can hide a missed source, an unintended edit, or a failure that appears only when the request is phrased differently.
OpenAI’s January 22, 2026 article proposes four evaluation dimensions: Outcome, Process, Style, and Efficiency. Its demo-app example includes test-04, an adjacent request to add Tailwind to an existing application. This is a negative control designed to catch unwanted scaffolding, not a report of an observed incident.
Our original exercise uses a smaller task: a release-brief Skill that reads synthetic release notes and returns two summaries. The downloadable starter separates three questions: is the answer acceptable, was the Skill used correctly, and did adding the Skill help?
Write the acceptance table first
The fictional release is version 0.4.0. CSV export and upload retries have shipped; offline editing is still planned. The Skill may read the notes and summarize them. It may not edit files or publish a message.
Turn that scope into checks. Code handles exact constraints. A reviewer handles meaning. When a run fails, the distinction tells you what to investigate.
| Dimension | Requirement | Evidence |
|---|---|---|
| Outcome | Version 0.4.0; exactly export and retry | Parse the final JSON; review summary accuracy separately |
| Process | Read the right source; leave files untouched; do not summarize releases for terminology questions | Inspect the trace and command results; compare file hashes |
| Style | Short summaries; never claim offline editing shipped | Check the schema, then review meaning |
| Efficiency | No pointless loops or unrelated calls | Retain completed commands, turn usage, and wall time |
Understand the evaluation loop
A case includes both the request and the starting workspace. Capture the execution trace, final answer, and resulting files. The final statement “done” is not enough to establish what happened.
The starter follows this order: collect evidence, apply hard checks, review process and quality, then preserve failures as regression cases.
Start with four cases, then grow the set
The download contains four cases to make the loop easy to understand. Add a few variations in each category to build an initial set of 10–20 cases. Treat this as a starting size, not a reliability guarantee. Real failures and closely related requests are particularly useful additions.
A negative case must specify the behavior to avoid. A terminology question should not read local release notes. A blocked write attempt also fails this read-only task, even if the sandbox prevents any final file change. For a Skill that legitimately edits files, compare changes against an explicit scope instead of requiring an unchanged directory.
| Case | Request | Question being tested |
|---|---|---|
| explicit | Use $release-brief to summarize this project’s release | Does direct invocation work? |
| implicit | Summarize shipped changes as JSON | Does the host select the Skill without its name? |
| context | Prepare a release handoff for support | Does added business context cause drift? |
| negative | Define release notes; do not read local release files | Does it answer the concept question without entering the release workflow? |
Prepare a small, isolated exercise
Download and unzip the starter, then open a terminal in skill-evals-starter. You need Node.js 20 or later and an installed, authenticated Codex CLI. The offline test command makes no model calls. Running run.mjs uses your Codex configuration and quota.
The package includes synthetic RELEASE_NOTES.md, a Skill, four cases, a JSON Schema, a runner, and a grader. The Skill lives in .agents/skills/release-brief/SKILL.md, following the current documentation. Each run uses a fresh temporary workspace with an explicitly requested read-only sandbox. Artifacts are written outside that workspace so logging does not count as an agent edit.
Your usual configuration and global Skills can still affect the run. Use a controlled evaluation environment for formal comparisons and check for a same-named global Skill. Removing a local Skill alone does not disable every copy.
node --version
codex --version
codex exec --help
# Offline grader tests; no model calls
node --test grade.test.mjs
# One real model run
node run.mjs explicitRead the trace, answer, and file state
The terminal prints the artifact directory. Start with grade.json, then inspect trace.jsonl and answer.txt. meta.json records the request and CLI version; stderr.txt preserves diagnostics. Failed and timed-out runs retain whatever evidence was captured.
The runner constructs the command below with real paths and the selected prompt. --json emits JSONL events; -o saves the final answer separately. Positive cases also share an output schema. The negative case deliberately does not use a release-summary schema, which would steer it toward the wrong workflow.
The without-Skill baseline uses the same positive-case schema. Correct formatting is therefore a basic check, not evidence that the Skill added value.
codex exec --json --sandbox read-only \
--skip-git-repo-check \
-C <temporary-workspace> \
--output-schema <absolute-schema-path> \
-o <absolute-answer-path> \
'<case prompt>'Hard checks are only one part of acceptance
The starter checks successful completion, unchanged workspace contents, and basic output constraints. Positive cases require the correct version and two change IDs. The negative case only checks for a nonempty answer without the local version. Semantic accuracy and a useful definition still need review.
Process requires its own evidence. This starter does not invent a universal skill_invoked event or equate a mention of SKILL.md with compliance. Review host loading information where available, actual commands, and their results. Keep the verdict pending when the trace cannot establish it.
A command containing npm install does not prove installation succeeded. A zero exit code does not prove an application works. A working application does not prove that edits stayed in scope. For a scaffolding Skill, add target-directory checks, a build, and relevant browser checks.
{
"checks": {
"run_completed": true,
"workspace_unchanged": true,
"output_contract": true
},
"hard_pass": true,
"process_review_required": true,
"quality_review_required": true
}Give the judge specific questions
Provide the original notes alongside the two summaries. Ask whether either shipped feature is misrepresented, whether offline editing is incorrectly promised, and whether each summary clearly explains a change. Require a verdict and supporting evidence for each question, not just an overall score.
If a model assists with review, pin its model, grading prompt, and output schema. Missing fields or an unusable response should fail the review step. Calibrate it against a few human judgments, including an answer that tries to instruct the judge to award full marks.
A high quality score must not override a failed hard constraint. File naming, JSON keys, and other directly checkable conventions should stay in code rather than becoming matters of model opinion.
Compare the Skill against a fair baseline
Run the remaining cases, then pair implicit with its without-Skill baseline. Both receive identical release notes and the same output schema; only the local Skill differs. Do not use the explicit Skill invocation as a without-Skill control.
Hold the model, CLI version, permissions, source data, and initial workspace fixed. Record the Skill commit or content hash. EVAL_MODEL can pin a model available to your account. Repeat important boundary cases and report correctness, false activation, and tool behavior separately.
Treat token counts and elapsed time as diagnostic signals. Saving tokens by skipping a required source is not an improvement. Missing usage is not zero usage. The starter deduplicates completed commands by item ID instead of counting started and completed events twice.
node run.mjs implicit
node run.mjs context
node run.mjs negative
# Compare this with the implicit run above
node run.mjs implicit --without-skillWhere agent-skills-eval fits
agent-skills-eval is an unofficial project maintained by darkrishabh. In the reviewed commit, b60eebe, with_skill injects Skill content into the request; without_skill omits it. The default Provider calls an OpenAI-compatible chat endpoint and returns tool_calls for assertions. That path does not execute those tools and feed their results back into an agent loop.
This supports comparisons of instructed outputs and tool-call arguments. Testing actual Codex discovery, workspace edits, and build outcomes requires a real execution environment and corresponding checks.
One limitation matters for baseline fairness: this commit reads case attachments only for with_skill. If the task depends on attachments, the two modes receive different information. Inspect the saved requests or adapt the runner before attributing an improvement to the Skill.
Make regressions actionable in CI
Version the Skill, cases, fixtures, and grading code together. Keep traces, answers, file differences, and grading results for each change. Separate infrastructure errors from task failures and quality failures so the team knows whether to retry or revise the Skill.
The starter exits nonzero when a hard check fails. Exit code zero still requires process and quality review. Before using it as a CI gate, implement those checks, configure authentication and a budget, and retain the evidence needed to investigate failures.
This is a tested starting point, not a claimed model benchmark. The grader passed 15 offline tests, including truncated traces, incorrect versions, and workspace changes. We did not run a live model evaluation or establish full acceptance of the example Skill.
Whenever a failure matters, preserve the input and a check that catches it. The Skill can change while the team’s definition of correct behavior remains explicit.