Start with a decision, not a usage chart
A support team uses AI to update its help center. Drafts arrive quickly, but a colleague still checks instructions, fixes omissions and prepares the final handoff. At the end of the week, the team lead needs a practical answer: should we expand this workflow, improve it, or stop?
You will produce a short review with three things: the observed human effort, whether the finished work passed the same quality standard, and a cost estimate only when the inputs support it. The downloadable exercise uses four fictional documentation tasks. It does not contain customer results or claim to measure Onevium performance.
Our earlier reporting tutorial taught you to reconcile messy records. This one adds a matched comparison, an explicit quality gate and a small calculator that refuses to turn missing costs into zero.
What we take from OpenAI’s article
In its September 16 article, OpenAI connects AI usage with the work a team wants to improve. It recommends establishing a baseline, counting review and correction time, and comparing benefits with AI, setup and support costs. It also distinguishes freed capacity from the business outcomes that capacity may eventually enable.
We apply that measurement idea to a small, original file-based exercise. OpenAI’s Admin Console, plugin and API are features described in its article; this tutorial does not assume they are available inside Onevium. Your own task records and a plain local script are enough to practice the method.
Agree on what a fair comparison means
For each task brief, record one manual result and one AI-assisted result. Keep the scope and final acceptance standard the same. Here, a reviewer requires complete coverage, facts checked against the approved source, usable steps and no known blocking error. A faster draft that fails that standard is unfinished work.
Count active human work, checking and corrections separately, without counting the same person’s minute twice. Include preparation and handoff. Passive waiting is a separate column: several simultaneous waits cannot be added together and called the delivery time. If delivery speed is the goal, record actual start and finish times separately.
The CSV matches rows by explicit task IDs and checks their type and difficulty. Those checks cannot decide whether the real tasks are comparable. Using the same person twice can introduce familiarity effects; record execution order and use separate workers or balanced ordering where feasible.
| Question | What to record |
|---|---|
| What work is being compared? | One agreed task brief and scope per pair |
| What counts as complete? | The same human-reviewed quality checklist |
| Which effort and cost are included? | Preparation, execution, review, rework, model/tool fees, setup and support |
Open the starter in Onevium and run it
Download and extract the ZIP. Use Add project → Open existing folder in Onevium, choose the extracted directory and create a conversation there. Browse all files lets you inspect sample-runs.csv, pilot-costs.json and README.md before running anything. The task examples concern help-center updates; the package contains their fictional measurement records, not generated help articles.
Use Node.js 20 or later. Check the terminal directory with pwd on macOS or Get-Location in PowerShell; it must be the extracted starter folder. Then run node --version. If Node is unavailable, complete your normal environment setup first. No package installation, analytics account or business-system connector is needed.
Run the first command below and check that all 14 tests pass. Then run the second command. It writes summary.json and both report languages inside results. These commands do not call a model or access the network; running them again overwrites those generated reports. Choose a new output directory when keeping a history.
node --test test.mjs
node summarize.mjs sample-runs.csv pilot-costs.json resultsCheck the human effort first
Open results/report-en.md and compare it with the CSV. Each manual task takes 30 minutes of work, eight of review and two of correction: 40 minutes. The AI-assisted rows also include human preparation, review and rework. The script adds those components; it does not treat AI waiting time as employee labor.
Across the four fictional pairs, human effort is 160 minutes manually and 115 minutes with AI assistance, an observed difference of 45 minutes. All four final results are marked pass after their listed corrections. Those quality labels are supplied teaching assumptions: the calculator does not inspect a document and certify its quality.
| Help-center task | Manual minutes | AI-assisted minutes | Final quality |
|---|---|---|---|
| Installation prerequisites | 40 | 25 | Both pass |
| Provider connection instructions | 40 | 25 | Both pass |
| Permissions troubleshooting | 40 | 35 | Both pass |
| Release/documentation consistency | 40 | 30 | Both pass |
Read README.md, sample-runs.csv, pilot-costs.json,
and results/summary.json in this project.
Explain the 160 and 115 minute totals using row IDs.
Separate human work, review, rework and waiting.
Check whether quality labels and costs are complete.
Show the cost arithmetic and each assumption.
Do not change inputs, fill missing values,
claim measured ROI, or extrapolate to a year.
List what a team lead still needs to verify.Forty-five minutes less effort can still be insufficient
All compared records use the same fictional hourly labor-cost assumption of USD 60, including relevant employment costs. Forty-five minutes therefore has a labor-cost proxy of USD 45. We assume only half of that freed capacity can be used for other useful work, giving USD 22.50. The 50% figure is a stated assumption to challenge, not a measured productivity rate.
The four AI runs cost USD 4.50 in model/tool fees. Setup takes another 15 human minutes at USD 60 per hour, and the exercise allocates USD 5 for support/subscription costs: USD 20 in total. The net capacity estimate is therefore USD 22.50 − USD 4.50 − USD 20.00 = USD −2.00.
That result supports improving the pilot before expanding it under these assumptions. It does not establish a cash loss, payroll reduction or revenue effect. Record what the freed time actually enables; only business evidence can connect it with an avoided expense, extra completed work or revenue. Never spread today’s setup cost across imaginary future tasks.
| Component | USD |
|---|---|
| Usable capacity assumption | 22.50 |
| Incremental model/tool costs | −4.50 |
| Setup and support costs | −20.00 |
| Net capacity estimate; not cash profit | −2.00 |
Let incomplete evidence stop the conclusion
The tests also check the uncomfortable cases. A missing fee leaves the time comparison visible but withholds the money calculation. A failed or unreviewed final result remains in the all-pairs time totals, and financial valuation is withheld for the whole pilot. A separate qualified-pairs subtotal is labeled so excluded failures cannot disappear into a success claim.
Duplicate runs, missing counterparts and a task ID reused under another pair cause an error. Resolve those records rather than choosing whichever run makes AI look best. The ID check prevents an obvious duplicate; it cannot prove statistical independence between tasks performed by the same team.
This introductory calculator also withholds capacity and net-value estimates when hourly rates differ anywhere in the comparison. Someone working longer at a lower rate may cost less without freeing time. Keep the actual rates and review that staffing or cost change separately; do not alter the records to pass the check. Mixed currencies are never added together. Use one documented currency basis if you need a combined comparison; this script does no currency conversion. Unknown fees or labor rates stay blank, and unknown JSON assumptions stay null. A zero means you have evidence for zero. Count subscriptions and setup once, with a clear allocation.
Replace the fictional records with one real pilot
Copy blank-runs.csv and blank-costs.json, choose one recurring task with an owner, and agree on the observation window and quality checklist before collecting results. The README explains every column. Keep the failures, difficult tasks and slow runs from that window, alongside the successful ones.
Use approved records with only the fields you need. Obtain model costs from the relevant bill or usage export; this exercise does not fetch them automatically. A local file does not determine what your configured model provider receives when you ask an assistant to read it.
At review time, inspect the actual work, not just the totals. Continue a workflow that repeatedly meets its quality bar with manageable complete effort. Improve it when review and setup dominate, and pause it when important errors persist. A small observational pilot can guide the next experiment; it cannot establish a general productivity percentage or annual ROI.
Use this as the fourth step in a team workflow
Begin by breaking one complex request into responsibilities and acceptance criteria. For a coding workflow, check the test bottleneck next. Once the method is useful, make its shared Skills understandable and testable. Measure value after a pilot has produced work you can inspect.
This order keeps the measurement grounded in an actual workflow. For the next review, bring the task records, the finished outputs, the cost evidence and one decision: expand, improve or pause.