Measuring an AI workflow: cost per accepted outcome, including rework

An execution is a technical event. An outcome is the useful result the business accepts. For an invoice workflow, the outcome might be a reviewed record ready for the next authorized step. For a reporting workflow, it might be a report accepted by its owner with no unresolved required fields.

Write the acceptance rule in operational language. Identify who decides, what evidence they inspect, and which defects require rework. Without that rule, the team can improve its success rate simply by changing what “success” means.

Track separate states for completed processing, accepted output, rejected output, and unresolved cases. A timeout should not disappear from the denominator because it never reached the final node. A rejected draft should not count as an accepted outcome because the model returned valid JSON.

I would keep the acceptance rule versioned. If the business changes its standard during a pilot, the report should identify which cases were judged under each version.

Give each business case a stable identifier and each execution attempt its own identifier. Record the relationship between them. A case that succeeds after three retries is one accepted outcome with the costs of four attempts.

Capture review and correction events against the same case. If someone fixes a draft manually outside the workflow, that effort still belongs in the measurement. Make the capture method simple enough that people will use it, such as a small review form with a duration category and correction reason.

OpenTelemetry describes traces as a way to follow a request through operations represented by spans. That structure can help connect technical work across services, but the application still needs its business case identifier and acceptance event. A trace does not automatically know that a manager approved a report. OpenTelemetry traces

Avoid placing sensitive case content in telemetry just to make reporting easier. Identifiers and bounded reason codes may be sufficient for the cost analysis.

Variable costs can include model requests, document extraction, external API calls, compute, storage, and human review. Record the actual billing basis where available. Estimated costs should remain labeled estimates until reconciled with invoices or provider usage records.

Human effort needs an agreed valuation method. A finance owner may choose a loaded internal rate for planning. That is an assumption to document, not a claim that every minute saved becomes cash in the bank.

Include failed attempts and rework. If a workflow produces an incorrect draft that takes longer to repair than starting again, its low inference cost is not the economically important fact.

Also account for operating effort that arrives in batches: incident handling, connector maintenance, access reviews, and prompt or rule changes. Allocate those costs over an explicit period and show the allocation method. Hiding them in a separate engineering budget makes the workflow appear cheaper than it is to operate.

A practical calculation is total attributable operating cost for the measurement period divided by the number of accepted outcomes in that period. Show the components alongside the result so a reader can challenge the assumptions.

Consider a hypothetical month with 100 submitted cases, 80 accepted outcomes, 200 units of service cost, and 600 units of review and correction cost. The operating cost per accepted outcome is 800 divided by 80, or 10 units. Dividing by submitted cases would give 8 units and answer a different question.

The remaining twenty cases still matter. Report whether they were rejected, pending, or routed to the old process. If the old process completed them, include that handling cost or show it as an explicit excluded cost. Do not let the boundary change silently between reports.

Keep implementation cost separate from recurring operating cost, then show a scenario that includes it. Amortization over a chosen period is a planning assumption. A short pilot cannot establish a durable payback period without evidence about future volume and maintenance.

The baseline should use cases similar in complexity to the automated cohort. Comparing easy automated cases with a historical average dominated by difficult exceptions can exaggerate improvement.

Define categories before reviewing the results. For example, separate clean structured submissions from scanned documents that require manual interpretation. Report results by category and disclose which cases the automation does not handle.

Use the same acceptance standard for the baseline and the new workflow. If people previously delivered a fully reviewed record and the new system produces only a draft, the outputs are not equivalent.

I would record both elapsed time and active work time. A case may spend two days waiting for approval while requiring only a few minutes of effort. Reducing the waiting time can be valuable, but it is different from eliminating paid work.

A cost chart without a quality chart can reward a worse system. Track the correction rate, critical errors, missing evidence, and unresolved cases alongside cost per accepted outcome.

Some failures deserve a stop condition regardless of the average. An unauthorized disclosure or an unapproved external action should not be averaged away by a large volume of inexpensive successful cases. Define those conditions with the process owner before launch.

For ordinary defects, classify the cause. A stale source, an extraction error, and a confusing approval screen require different fixes. A single “AI failed” category makes the report less useful to the team responsible for improvement.

Measure reviewer agreement on a small sample if acceptance is subjective. If two reviewers disagree frequently, the acceptance rule may need clarification before the model is blamed for inconsistent results.

The report should show submitted cases, accepted outcomes, unresolved cases, total cost, cost per accepted outcome, correction effort, and any stop-condition events. Include the measurement window and data completeness.

Then identify one concrete decision: continue the pilot, repair a specific failure, narrow the eligible case set, or stop the workflow. Avoid turning every weekly report into a dashboard expansion project.

When a metric changes, inspect the underlying mix. A lower cost may reflect fewer difficult cases rather than an engineering improvement. A higher cost may reflect a temporary cleanup effort that improves later operation. The report should preserve that explanation with evidence.

I would include case identifiers, acceptance events, cost capture, and a basic operating report in the original implementation scope. Adding them after launch can leave the team unable to reconstruct what happened during the pilot.

Start with the fields needed to answer whether the workflow is useful and supportable. More telemetry is helpful only when someone can use it to make a decision or investigate a failure.

Bring a sample of the current process, its review criteria, and the costs you can actually observe.