The code is ready, but the release is not
Imagine a small team shipping an invoice feature. AI helps finish the implementation, but the pull request waits for a runner, reruns a flaky check, and sits through unrelated tests. Continuous integration (CI) is the process that automatically builds and tests each code change. Faster code generation has moved the queue; it has not removed the work needed to release safely.
Anthropic's September 14 article describes pressure on its deterministic test-selection service: the component collecting results fell behind, leaving selection decisions based on stale history. The redesign separated shared state from workers so the system could scale. That is a lesson about the reliability of the decision process, not permission to let a model guess which tests to delete.
Our independent exercise starts much smaller: inspect three fictional timing records, select tests for two tiny modules, and compare the proposal with a full run. Try it once your team has a repeatable task and a known test command. If the slowest part is waiting for a runner, a clever test selector may not be the first thing to build.
Open a small project you can inspect completely
Download the original starter and unzip it. It needs Node.js 20 or later; there are no dependencies to install or API calls to make. The folder contains invoice and notification modules, two business-test files and one smoke-test file, a handwritten map, timing data, and small analysis scripts. All business data and timing records are synthetic.
In Onevium, click Add project beside Projects, choose Open existing folder, and select ci-test-selection-starter. Create a conversation from that project's row. At the top of the conversation, Browse all files opens the files and the terminal icon opens the built-in terminal. Confirm the directory before running commands; an existing terminal can retain its previous working directory. You can also use your usual terminal.
Start with the six fixture checks and two business tests and one smoke check below. The first command verifies the exercise's selection and arithmetic rules; the second runs its fictional business code. A downloaded folder does not need to be a Git repository to browse files. Neither command is a model evaluation or a full Onevium workflow test.
node --version
node --test fixture.test.mjs
node --test tests/invoice.test.mjs tests/notification.test.mjs tests/smoke.test.mjsFind out where the minutes go
Run node analyze.mjs. The report separates initial queue time, first-attempt execution, and additional retry time. In this toy dataset those periods do not overlap; retries include any extra waiting and execution after the first attempt. The three runs total 9 minutes queued, 15 executing, and 2 retrying. Their combined 26 minutes are an aggregate across runs, not one release's latency.
DEMO-B takes 12 minutes, of which 6 are the initial queue. DEMO-C spends 8 of its 9 minutes executing. They suggest different investigations: available runners and concurrency for B, expensive test steps for C. The retry record tells you to inspect why the first attempt failed before recommending automatic retries.
For your own CI, use actual timestamps and record cancellations and incomplete runs separately. Parallel jobs overlap, so adding every job's duration does not give end-to-end latency. First compare a few similar changes with the same timing definitions; do not present this synthetic example as measured time saved.
| Run | Queue | First execution | Extra retries | Total |
|---|---|---|---|---|
| DEMO-A | 2 | 3 | 0 | 5 |
| DEMO-B | 6 | 4 | 2 | 12 |
| DEMO-C | 1 | 8 | 0 | 9 |
node analyze.mjsMake every test choice explainable
Open dependency-map.json. It explicitly maps src/invoice.mjs to the invoice test and src/notification.mjs to the notification test. Every selection also includes a smoke test that checks the two entry points load. That small smoke check does not validate their behavior.
The command below prints a JSON plan with mode, changed files, reasons, and tests. For an invoice change, the mode is affected and the list contains invoice.test.mjs plus smoke.test.mjs. It does not run tests or alter a CI configuration. Passing filenames is a declaration of what changed; the script does not obtain a Git diff.
This is deliberately a handwritten map, not Anthropic's historical-result service or an automatic coverage graph. An AI assistant can help explain the paths and propose missing relationships, but a plausible explanation is not proof that the mapping is complete.
node select.mjs src/invoice.mjs
# Selected test files:
# tests/invoice.test.mjs
# tests/smoke.test.mjsWhen the information is uncertain, run the full set
Try the four commands below. Each returns mode: full and all three test files. Unknown files and shared/configuration changes also return full. An empty change list, an invalid map, or an incompatible layout revision cannot silently select no tests.
The two uncertainty flags matter. The demo cannot detect a deleted or renamed file, an incomplete diff, or a stale but structurally valid map by itself. The caller must flag those conditions. The map's layoutRevision is only a manual compatibility marker; it is not a commit check. A real system needs a maintained source of dependency and revision evidence before trusting a smaller set.
node select.mjs new-file.mjs
node select.mjs --uncertain-history src/invoice.mjs
node select.mjs --map-stale src/invoice.mjs
node select.mjs --map missing.json src/invoice.mjsUse a full run to catch a confidently wrong selection
Run the normal shadow command below. It actually executes the selected set and all three tests in separate Node processes. Both pass in the supplied example. The printed output field identifies a new .results/run-* directory containing summary.json and individual test logs.
Next, run the deliberate fault case. It declares only an invoice change but makes the notification module return an incorrect message. The selected invoice and smoke tests still pass; the full run fails the notification test. summary.json records that file in omittedFailures, and the command exits 1. This failure is the expected result of the exercise, not a reason to remove the notification assertion.
The flag simulates incorrect impact information; it neither discovers a real dependency nor modifies source files. Its purpose is to make a failure visible that a reduced green result would otherwise hide. Passing the normal case only validates this tiny local example.
node shadow.mjs src/invoice.mjs
# Expected exit code 1; inspect omittedFailures
node shadow.mjs --simulate-missed-dependency src/invoice.mjsGive Onevium a reviewable investigation
With the exercise folder open, reference ci-runs.json and dependency-map.json from the composer's @ file list, then send the request below. Replace the result-folder placeholder with the actual path printed by your run. The useful deliverable is a short explanation tied to files and command output; you can check it against the terminal yourself.
For a real team, begin with exported timing data and a proposed selection report while existing full CI remains authoritative. Compare on the same revision and environment. Track omitted failures, fallback frequency, selection overhead, and queue time before proposing a change. A quieter dashboard alone cannot establish that fewer bugs escaped.
Read this exercise's ci-runs.json, dependency-map.json,
and .results/<actual-run-folder>/summary.json.
Explain which time category dominates each run.
Explain why the selection missed the simulated failure,
citing the test paths and result fields.
Separate observed output from your recommendations.
Do not edit files, delete tests, change CI, commit, or push.
If evidence is missing, list it instead of guessing.Keep the next experiment small
The checked deliverable here is an offline Node exercise: selection rules, fallback behavior, timing arithmetic, actual local test runs, and an intentionally missed failure caught by the full comparison. It does not establish real-model quality, production safety, or a time-saving percentage.
A practical order for the series is to define who owns a complex task, investigate the test bottleneck here, preserve a repeatable method in a team Skill, and then measure the whole workflow. You can start with this article if CI is already the obvious delay. Your next step is one shadow report on a real, representative change, with full tests still in place and a named person reviewing the difference.