Some companies recently ranked employees by the number of AI tokens they consumed. The pattern became known as tokenmaxxing, followed by tokenminning, which favoured using as few tokens as possible.
Both measures miss the point. If you are tracking AI ROI, the useful question is whether work ships with better quality and supports a business outcome. This piece shows why usage metrics fail, where the bottleneck moves and how to set a more useful finish line.
AI investment becomes meaningful when usage data is connected to a business result that leaders can inspect and improve.
- Token volume, adoption and feature counts can increase while delivery constraints remain unchanged.
- AI can amplify strong or fragile workflows, so quality and delivery measures must sit beside usage data.
- Founders should define the outcome first, baseline it, then test whether AI-assisted work changes the result.
AI ROI needs a finish line
The fuel-gauge analogy explains the management error. A racing team that celebrates fuel consumption may tune its car to burn more, while a team that celebrates minimal fuel use may leave the car at the starting line. Neither measure tells leadership who finished well.
The same distortion appears when companies reward token volume, graphics processing unit (GPU) utilisation or the number of AI features shipped. High GPU utilisation shows that compute is busy, but it does not show that a customer problem was solved. Adding a summarisation feature to every legacy product may increase release counts while creating more review work and support complexity.
Adoption can mislead in a quieter way. A company may report 95% deployment of a coding assistant and infer that developer productivity has improved. If pull requests become larger, human review takes longer and defect rates rise, the 95% figure describes distribution rather than effective use.
What caught my attention was how quickly the metric can become the work. Uber acknowledged that it had spent its entire AI budget for the year within a matter of months, while the central question remained whether that spending improved the work being delivered.
The tokenmaxxing cycle exposes the measurement trap
The recent swing from tokenmaxxing to tokenminning provides a useful pattern. First, leaders treated higher consumption as evidence of progress. Then the backlash treated lower consumption as evidence of discipline. Both choices put an input at the centre of the scorecard instead of defining the result the business needed.
This is why a technically sophisticated number can still become a vanity metric. A vanity metric is a number that looks impressive on a dashboard but does not change how the business is run. GPU utilisation, AI features shipped per quarter and software licences deployed can all be measured precisely while remaining detached from the work customers or colleagues actually need.
The comparison is straightforward: twice as much generated code that takes twice as long to ship is more work, not necessarily more efficiency. Likewise, a high adoption rate without a change in cycle time or rework may show that a tool was distributed, not that the workflow improved.
The practical test is simple. If leadership cannot name the business result that should move when a usage number changes, the usage number is telemetry without a finish line.
AI amplifies the workflow around it
The 2025 DORA report found that many teams deploying AI into underprepared systems experienced weaker delivery stability. The finding does not make AI inherently harmful; it shows that the surrounding engineering practice affects what higher output produces.
The effect chain is visible: AI generates more first-draft code, pull requests grow, human review becomes the constraint, and cycle time may remain flat while rework or defects increase. More generation has created more material to inspect, so the bottleneck has moved rather than disappeared.
The tool can increase output while leaving the business waiting at the same constraint.
For established engineering teams, DORA examines delivery health through measures including deployment frequency, change lead time, recovery from failure and change-related incidents. SPACE covers five dimensions of developer productivity, including outcomes and satisfaction. Neither framework was designed as an AI usage scoreboard, which makes each more useful for testing whether the work improved.
Engineering. Pair AI usage with cycle time, rework rate and cost per change. A 25% increase in adoption alongside weaker delivery stability should prompt investigation rather than celebration.
Support. Track first-response time beside reopen rates. Faster replies may improve service, but a rise in reopened cases would show that the workflow shifted the burden instead of reducing it.
Other functions. Marketing, finance and operations can start with an existing outcome such as response time, pipeline conversion or forecast accuracy. The AI measure describes the intervention; the business measure determines whether the intervention deserves further attention.
Measure the outcome before the intervention
A credible measurement design begins with the result the team already owns. Record the baseline before changing the workflow, then compare the outcome afterwards while checking the quality measure that could expose a hidden cost.
For example, a support team began with a first-response baseline of nine hours. After agents used an AI assistant to draft replies, response time was three hours two months later, while reopen rates had not changed. That comparison says more than a count of generated replies because it connects speed with a quality check.
Usage data still has a role, but it should support the investigation rather than settle it. Adoption can show how widely a tool was deployed; token data can show consumption; neither establishes that the business improved without an outcome moving alongside it.
Keep human review explicit. AI-generated code, replies or analysis can contain errors, and review may be where those errors are found. Removing or weakening that step to improve an activity metric can make the dashboard look healthier while the underlying service becomes harder to manage.
Choose the measurement model according to the workflow. Engineering teams may pair DORA or SPACE with AI usage and business measures, while support, marketing, finance and operations may use a simpler pair such as response time with reopen rates or forecast accuracy with rework.
What AI ROI means for your next decision
Define the finish line. Write the business outcome before selecting the AI measure. If the goal is faster delivery, name cycle time or recovery from failure rather than token consumption.
Baseline the workflow. Record the current result, such as the nine-hour first-response time, before changing the process. The baseline gives later movement a reference point.
Pair speed with quality. Use reopen rates, rework or incidents beside the primary outcome. A faster process with weaker quality may have moved the constraint rather than solved it.
Inspect the constraint. If AI generates more code but review expands, examine the review workflow before increasing usage. The next unit of output waits wherever the process is slowest.
Use usage as evidence. Adoption and token data can help describe what changed, but the decision to continue should consider the outcome and its associated costs.
AI ROI is therefore a management discipline, not a consumption score: value is assessed at the finish line, while usage is supporting telemetry. To compare measurement approaches or examine an AI-assisted workflow around a specific bottleneck, read more from Yanisa Execution or speak with the team about the operating detail behind the metric.
Frequently Asked Questions
How should a company measure AI ROI?
Start with a business outcome already owned by the team, record its baseline, then compare results after the AI-assisted workflow changes. Pair the primary measure with a quality check such as rework, reopen rates or delivery incidents.
Why can AI usage metrics become vanity metrics?
Usage metrics can show that tokens, licences or compute are being consumed without showing that the work improved. They become more useful when connected to an outcome that changes how the business is managed.
Is high AI adoption a sign of success?
High adoption shows that a tool has been distributed or used, but it does not establish effective use. If delivery stability, cycle time or quality does not improve, adoption alone may be an expensive distribution measure.
What should engineering teams measure alongside AI usage?
Engineering teams can pair usage data with deployment frequency, change lead time, recovery from failure, incidents, cycle time, rework rate or cost per change. DORA and SPACE can help examine delivery health and developer productivity.
How can support teams measure an AI assistant?
Track a service outcome such as first-response time and pair it with a quality measure such as reopen rates. A move from nine hours to three hours over two months, with reopen rates unchanged, provides a clearer signal than counting generated replies.