KPIs have guided organizational decision-making for over a century, and for good reason. In an industrial world, performance was tangible and measurable: weight limits, throughput, speed. Today’s digital ecosystems operate differently. With cloud, AI, and hybrid architectures, improvement is no longer a straight line. Decisions involve tradeoffs across cost, resilience, speed, and risk, and many traditional KPIs struggle to capture that reality.
On the Oteemo CIO Podcast, Justin Fanelli, Chief Technology Officer for the U.S. Department of the Navy, explained why leaders need to rethink metrics and introduced the Navy’s Outcome-Driven Metrics framework.
“KPIs are a 120-year-old metric system. They worked in the Industrial Age, but they don’t always help us compare new technology tradeoffs.” — Justin Fanelli
Why KPIs and OKRs Aren’t Enough
KPIs excel at measuring discrete outputs: uptime, defect rates, and cost per unit. But they often push leaders to focus optimization on the wrong things, missing the bigger picture of resilience, adaptability, and user experience.
OKRs (Objectives and Key Results), a more recent management tool, stretch organizations to set aspirational goals. They’re useful for startups or teams launching something new. But in large, mission-driven enterprises like the Navy, neither KPIs nor OKRs are sufficient to measure whether technology investments truly “move the needle” and a new paradigm must be deployed.
The Five Outcome-Driven Metrics
To accomplish this, Fanelli’s team adopted a portfolio-based approach anchored in five outcome-driven metrics:
- User Time Lost — How much time do end-users waste waiting for systems to work?
- Operational Resilience — Can services withstand outages, cyber threats, or degraded conditions
- Cost per User — Are we delivering value relative to peers and benchmarks, not just driving costs down?
- Adaptability — How quickly can contracts, code, or processes shift when requirements change?
- Customer Experience — Are users actually satisfied and able to accomplish their missions?
We’re using outcome-driven metrics… the five are: user time lost; operational resilience; cost per user; adaptability; and customer experience. — Justin Fanelli
Five is deliberate: one metric risks tunnel vision, ten creates confusion.
If you have one thing you’re measuring, you’re probably over-optimizing. If you have ten, no one remembers them. — Justin Fanelli
From Dashboards to Tradeoffs
The power of outcome-driven metrics lies in decision-making. Instead of comparing vendors on feature checklists, the Navy evaluates how each option shifts the five outcomes.
For example: in military communications, traditional satellites (MILSAT) and proliferated low-earth orbit (LEO) constellations offer very different advantages. MILSAT offers hardened, high-assurance channels; LEO offers flexible bandwidth and easier replacement. By measuring tradeoffs in terms of resilience, adaptability, and cost per user, leaders can decide how much of each to fund, rather than defaulting to a single solution.
If you have a dollar, what is it going to improve? If you can’t explain it, you don’t even get in the room. — Justin Fanelli
Governance Without the Gridlock
One of the biggest advantages of outcome-driven metrics is how they streamline conversations. Instead of endless debates about pet projects, leaders align on five outcomes upfront.
If a proposal doesn’t measurably improve one of them, it never makes it past discussion.
This creates a culture where impact trumps preference, and where data, not emotion, drives portfolio decisions.
Why This Matters for CIOs and CTOs
Outcome-driven metrics provide leaders with a common language for tradeoffs:
- Security vs. Performance: captured under resilience.
- Innovation vs. cost control: measured under adaptability and cost per user.
- User complaints vs. uptime reports: reflected in user time lost and customer experience.
By reframing metrics around outcomes, technology executives can move faster, defend investment decisions, and ensure that “success” isn’t defined by vanity dashboards but by real-world impact.
Oteemo’s Perspective: Engineering Metrics Into Transformation
At Oteemo, we see outcome-driven metrics as a bridge between technical detail and business value. Too often, program dashboards are cluttered with dozens of KPIs that don’t tell decision-makers what really matters.
Our approach is to help clients map every technology initiative to one of the five outcomes Fanelli describes, then instrument their systems so results can be measured in real time. Whether it’s reducing user wait time through automation, increasing adaptability through modular architecture, or improving customer experience via human-centered design, success must be tangible.
This mirrors the Navy’s philosophy: stop chasing activity metrics, and start measuring whether technology is actually moving the needle.
