Case Insight #9: Why Organizations Struggle to Measure the Value of AI

Christina Sitaras

Oct 2

5 min read

Organizations are investing heavily in AI.

Yet a recurring question continues to appear:

How do we measure whether AI is actually creating value?

Organizations can often measure adoption.

They know how many employees are using a tool.

How many licenses have been deployed.

How many tasks have been automated.

How frequently particular capabilities are being accessed.

Some can estimate hours saved.

But translating those measures into organizational value is proving more difficult.

Perhaps part of the difficulty is not AI.

Perhaps it is the sequence through which AI was adopted.

Measurement Begins Before Implementation

It is difficult to evaluate an intervention when the outcome it was intended to produce was never clearly defined.

Consider a relatively traditional technology investment.

An organization implementing an ERP system might be trying to standardize fragmented processes, improve information visibility, integrate disconnected operations, reduce manual reconciliation, or create a more scalable operating environment.

The ERP is the technology. But the desired organizational improvement exists independently of the technology selected to enable it. That distinction creates the basis for evaluation.

Did reconciliation time decrease?

Did data become more consistent?

Did cycle times improve?

Did previously fragmented processes become standardized?

Did information become more accessible?

Not every benefit will be perfectly quantifiable.

But there is at least a theory connecting the investment to an intended outcome.

AI should not be fundamentally different.

Adoption Is Not the Same as Value

Suppose an organization announces that it wants to increase AI adoption. Six months later, usage has increased substantially.

Was the initiative successful? If adoption was the objective, yes. But why was adoption the objective?

Perhaps employees now produce documents faster.

Does that mean productivity improved?

Maybe.

But perhaps the volume of documents also increased.

Perhaps managers now spend more time reviewing AI-generated work.

Perhaps errors or inconsistencies require additional validation.

Perhaps employees use the time saved to perform higher-value activities.

Or perhaps the saved time simply disappears into additional meetings and email.

Knowing that people used AI does not tell us what changed in the organization because they did.

Tool-First Adoption Creates a Measurement Problem

When organizations begin with:

Where can we use AI?

they can identify dozens of applications.

A pilot is launched.

Then another.

Features are added to existing systems.

Employees begin using general-purpose tools.

Teams experiment independently.

Eventually leadership asks:

What value are we getting from all of this?

The question is reasonable.

But the organization may not have established the conditions required to answer it.

What was the original problem?

What was the baseline?

What outcome was expected to change?

Over what period?

What other parts of the system could be affected?

What would success have looked like without AI?

Without that information, evaluation becomes retrospective.

The organization is trying to determine whether an outcome occurred without having clearly defined which outcome it was trying to produce.

Hours Saved May Not Be the Outcome

One of the most intuitive measures of AI productivity is time saved.

A task that previously required two hours now takes thirty minutes.

That seems valuable.

And it may be.

But the organizational value depends on what happens to the remaining ninety minutes.

If the employee uses that time to complete additional valuable work, capacity may genuinely have increased.

If the faster output creates additional review requirements elsewhere, some of the capacity may simply have moved.

If greater production creates more downstream work than the organization can absorb, the system may become less efficient even while the individual task becomes faster.

If quality deteriorates and creates rework, apparent savings may disappear altogether.

The relevant unit of analysis therefore matters.

A tool can improve the productivity of an individual activity without improving the performance of the system in which that activity exists.

AI May Move the Constraint

Imagine AI reduces the time required to develop an initial analysis from several hours to twenty minutes.

The organization can now produce substantially more analyses.

But perhaps each analysis still requires review by someone with scarce expertise.

The constraint has not disappeared. It has moved.

And because upstream production is now faster, the review constraint may become more severe than it was before.

From the perspective of the person creating the analysis, productivity improved dramatically.

From the perspective of the reviewer, workload increased.

From the perspective of the overall process, cycle time may barely change.

All three observations can be true.

This is why AI value cannot always be evaluated at the point where AI is introduced.

Moving Quickly Can Still Be Intentional

Organizations have legitimate reasons to move quickly.

AI capabilities are changing rapidly.

Competitors are experimenting.

Customers' expectations are changing.

Employees may already be using tools regardless of whether the organization has formally adopted them.

Waiting for complete certainty may itself carry risk.

In competitive environments, organizations can find themselves in a Red Queen dynamic: moving simply to maintain their relative position while everyone around them is moving too.

But urgency does not eliminate the need for intent.

An organization may reasonably decide that the objective of an AI initiative is learning, rather than immediate financial return.

Can this technology reduce cycle time?

Can it improve first-draft quality?

Which activities still require human judgment?

What new risks emerge?

What capabilities do employees need?

What happens downstream when output accelerates?

Those are measurable questions.

The experiment does not need a guaranteed return.

It needs a reason for existing.

Define the Outcome Before Measuring the Tool

Consider again an AI-enabled product search capability.

If the organization begins with:

We want AI search.

Success becomes difficult to define beyond whether the capability was built and whether customers use it.

But suppose the original problem was:

Customers struggle to find relevant products within our catalogue.

Now the organization has something to evaluate.

Did search abandonment decrease?

Did customers find relevant products faster?

Did conversion improve?

Did customer effort decline?

Did support inquiries related to product discovery change?

AI may be the mechanism producing the improvement.

But the outcome being measured is the customer or organizational result the technology was intended to create.

The same logic applies internally.

If AI is introduced to reduce repetitive administrative work, measure whether that work declined and what happened to the capacity released.

If AI is intended to improve decision support, determine what improvement in decision-making actually means.

If AI is intended to accelerate software development, examine not only output but quality, rework, deployment, and downstream maintenance.

The KPI follows the intent.

Perhaps AI Is Not the Measurement Problem

AI certainly creates measurement challenges.

Some benefits are difficult to quantify.

Causal relationships can be complicated.

Capabilities evolve rapidly.

Benefits may emerge in unexpected places.

Not everything valuable can be reduced to a financial return.

But those realities do not eliminate the need to define what an investment or experiment is intended to accomplish.

If organizations adopt AI because they believe they need to adopt AI, then struggle later to determine whether AI created value, the difficulty may have begun much earlier than the measurement exercise.

It may have begun when the tool became the starting point.

Closing Reflection

Before asking:

How do we measure the ROI of AI?

it may be useful to ask:

What outcome did we expect AI to change? What did that outcome look like before the intervention? What else in the system could change as a result?

And:

What evidence would convince us that the intervention was worth continuing?

Perhaps the difficulty is not that AI is uniquely difficult to measure.

Perhaps we are trying to measure an outcome we never defined.