What CMM Taught Me About Measuring Institutional Effectiveness
Some of the most useful ideas in my career did not come from a new framework. They came from seeing an old one clearly enough to use it in a different setting.
Early in my technology career, in the early 2000s, I became well versed in the Carnegie Mellon Capability Maturity Model, especially Levels 2 and 3. At the time, I was focused on software and process improvement. The language was technical, but the lesson was broader than technology.
Organizations do not become capable all at once.
They move through stages. They begin with informal practices, often driven by individual effort. Then they start to define common processes. Over time, those processes become repeatable, measured, managed, and eventually embedded in the way the organization works.
That way of thinking stayed with me.
Decades later, while developing an institutional effectiveness framework, I found myself returning to the same core idea. Different programs were starting from different places. Some were still defining their mission and outcomes. Others already had measures, data, and evaluation practices in place. Treating them the same would have been unfair, and it would not have helped them improve.
So I built a maturity model.
Not as a grading system. Not as a compliance checklist. Not as a way to sort programs into winners and losers.
I built it as a practical way to understand current capability, identify the next capability to build, and help each program move forward deliberately.
What CMM taught me about capability
The Capability Maturity Model made a simple point in a disciplined way. Before an organization can manage a process, it must define it. Before it can improve a process, it must repeat it. Before it can compare performance, it must know what performance means.
That may sound obvious. In practice, it is easy to forget.
In technology work, I saw organizations try to jump ahead. They wanted measurement before they had common practices. They wanted predictability before they had defined roles. They wanted improvement before they had agreed on what the process was supposed to produce.
CMM helped expose that mismatch.
Levels 2 and 3 were especially instructive. Level 2 focused on repeatable practices. The organization may still depend on specific teams or people, but it has enough structure to manage work with some consistency. Level 3 moved toward defined processes across the organization. Practices were documented, shared, trained, and expected.
That shift mattered.
At Level 2, a team might succeed because experienced people know what to do. At Level 3, the organization begins to succeed because it has defined how work should be done. The capability moves from personal memory into organizational practice.
That distinction shaped how I later thought about institutional effectiveness.
Many organizations want evidence of effectiveness. That is reasonable. Leaders need to know what is working, what is not, and where to invest attention. But evidence does not appear by command. It depends on a chain of capabilities that have to be built in order.
If a program has not defined its outcomes, asking for performance indicators can turn into wordsmithing. If it has not selected indicators, asking for data produces confusion. If the data are weak, asking for evaluation findings can lead to false confidence.
The maturity lens helps slow that rush. It gives leaders a way to ask a better question.
What is the next capability this program needs in order to become more effective?
Why institutional effectiveness needed a maturity model
When I began developing an institutional effectiveness framework years later, I saw a familiar problem. Programs were not all at the same stage.
Some had a clear mission, but had not translated it into measurable outcomes. Some had outcomes, but no clear logic connecting activities to results. Others had indicators, but collected data inconsistently. A few had strong evidence and were beginning to use findings in planning and resource discussions.
A single standard would have missed those differences.
If I expected every program to produce the same level of evidence at the same time, I would reward those that already had mature practices and discourage those still building the basics. If I lowered the standard for everyone, I would fail to challenge the more mature programs.
Neither approach made sense.
The maturity model gave me a middle path. It allowed each program to be assessed against a developmental path, rather than a flat checklist. That made the conversation more honest and more useful.
A program at an early level could say, “We are still defining our outcomes,” without feeling that the statement was an admission of failure. A more mature program could focus on whether it was actually using evidence to improve decisions, rather than simply proving that it had measures.
That was the real value.
A good maturity model changes the tone of the conversation. It moves the discussion away from judgment and toward growth. It helps leaders see that lower maturity is not a defect. It is information.
A lower maturity score should point to the next capability to build, not serve as a label of failure.
That principle became central to the model.
The four levels I used
I created a four-level model that was simple enough to explain and strong enough to guide improvement.
The four levels were:
Level | What it means |
Ad Hoc | Practices are informal, inconsistent, or dependent on individual effort. |
Defined | Expectations, concepts, and basic processes are described and shared. |
Managed | Practices are implemented with consistency, evidence, and oversight. |
Integrated | Practices are embedded in planning, decisions, improvement, and governance. |
I chose these terms with care.
“Ad Hoc” describes a real starting point. It does not mean people are careless. In many cases, people are working hard and doing thoughtful things. The issue is that practices are not yet common, visible, or repeatable.
“Defined” marks an important step. At this point, people have agreed on language, expectations, outcomes, and basic methods. The work becomes easier to discuss because it has a shared structure.
“Managed” means the work has moved from paper to practice. Data are collected. Measures are used. Evaluations happen with enough discipline to support interpretation.
“Integrated” is the highest level because the work has become part of how the organization thinks and acts. Evidence informs decisions. Roles are clear. Improvement is routine. Institutional effectiveness is no longer an annual exercise. It is part of management and governance.
I wanted the levels to be easy to remember because models that require constant explanation rarely survive contact with busy organizations. The model had to support real conversations with program leaders, not just satisfy framework designers.
The seven areas the model assesses
The next decision was what to assess. I did not want a model that only measured data collection. Data matter, but they are only one part of effectiveness.
I built the model around seven areas.
Area | Core question |
Strategic alignment | Does the program connect clearly to organizational priorities and mission? |
Outcomes and logic | Are intended results defined, and is there a clear logic for how activities produce those results? |
Indicators and measures | Are there meaningful ways to track progress and performance? |
Evidence and data | Are data collected with enough quality and consistency to be useful? |
Evaluation and interpretation | Are findings analyzed in context and interpreted responsibly? |
Decision use and continuous improvement | Do findings influence planning, resource choices, and improvement actions? |
Governance and roles | Are responsibilities, review processes, and decision rights clear? |
Each area represents a capability. Together, they describe how institutional effectiveness works from strategy through use.
I included strategic alignment first because effectiveness has to be tied to purpose. A program can measure many things and still miss what matters. Without alignment, measurement can become busy work.
Outcomes and logic come next because an organization needs to define what success looks like before it chooses how to measure success. That sounds basic, but I have seen many teams start with data because data feel concrete. The problem is that data without outcomes can become a search for meaning after the fact.
Indicators and measures translate outcomes into observable signals. They do not have to be perfect, but they have to be clear enough to guide collection and interpretation.
Evidence and data involve the quality, availability, and consistency of information. This is where many organizations discover that good measures still require disciplined collection.
Evaluation and interpretation turn data into meaning. Numbers rarely explain themselves. People have to ask what changed, why it changed, what else may have influenced results, and how much confidence the evidence supports.
Decision use and continuous improvement ask whether the work affects choices. If findings never change plans, budgets, priorities, or practices, the system is incomplete.
Governance and roles hold the structure together. Someone has to own the process. Someone has to review findings. Someone has to decide what changes.
Why sequence matters more than speed
The biggest lesson I carried from CMM into institutional effectiveness was that sequence matters.
Organizations often want to move fast. I understand the pressure. Leaders need answers. External stakeholders may ask for evidence. Internal teams may want dashboards, scorecards, or evaluation reports.
But speed can create weak foundations.
The sequence I used was straightforward:
Establish strategic alignment.
Build outcomes and logic.
Define indicators and measures.
Collect evidence and data.
Evaluate and interpret findings.
Use results for decisions and improvement.
Clarify governance and roles across the cycle.
The sequence does not mean every step must be perfect before the next begins. Real work is more iterative than that. But the order matters because each capability depends on the previous one.
Strategic alignment comes first because it answers the purpose question. Why does the program exist, and how does it support broader priorities?
Outcomes come next because they describe the intended change or result. Without outcomes, indicators have no anchor.
Indicators follow outcomes because they translate intent into something observable. They help answer, “What would we expect to see if this program is working?”
Evidence follows indicators because data collection should serve a measurement plan. Collecting data before defining indicators often leads to too much information and not enough meaning.
Evaluation follows evidence because interpretation requires something credible to interpret. Weak data can still be useful, but only if people understand the limits.
Decision use follows evaluation because findings should inform action. This is where institutional effectiveness earns its keep.
Governance supports the whole sequence. It clarifies who does what, when reviews happen, and how decisions get made.
This sequence also protects programs from unrealistic expectations. A program that has not yet defined outcomes should not be pushed immediately into advanced evaluation claims. Its next step is to define outcomes well. Once that capability exists, the next step becomes clearer.
That is how maturity models help. They make improvement visible in manageable increments.
How the model changed the conversation
The four-level model helped turn institutional effectiveness into a developmental conversation.
Instead of asking, “Is this program effective?” as a yes-or-no question, I could ask more precise questions:
Has the program defined its mission and outcomes?
Are the outcomes connected to strategy?
Do the indicators match the outcomes?
Is the evidence good enough to support interpretation?
Are findings used in decisions?
Are roles clear enough to sustain the work?
Those questions are harder to game and easier to act on.
They also respect the reality that programs mature at different rates. A newer program may need to focus on alignment and outcomes. A long-running program may need to improve its use of findings. A program with plenty of data may need stronger interpretation practices.
The model also helped reduce defensiveness. When people hear assessment, they often hear criticism. A maturity model can reduce that reaction if it is used well. It creates a shared language for where things stand and what comes next.
That last phrase matters: what comes next.
The model should always lead to a next step. If a maturity assessment only produces a score, it has not done enough. The score is the beginning of the improvement conversation, not the end.
What made the model practical
I learned from CMM that models have to be both structured and usable. If a model is too vague, it does not guide action. If it is too complicated, people avoid it.
So I tried to keep the model practical in four ways.
The levels were intuitive. People could understand Ad Hoc, Defined, Managed, and Integrated without a long briefing.
The areas followed the work. The seven areas reflected the natural path from mission to outcomes, measures, evidence, interpretation, decisions, and governance.
The scoring was developmental. Lower maturity showed where support was needed. It did not assign blame.
The model supported different starting points. Programs did not have to pretend they were all in the same place.
That last point may be the most important. Institutional effectiveness work often fails when it assumes uniform readiness. Mature programs get bored. Emerging programs get overwhelmed. Leaders get uneven information and struggle to compare progress.
A maturity model does not solve all of that, but it helps. It gives the organization a common frame while allowing programs to build capability in sequence.
The lesson that still holds up
Looking back, I do not think the lasting value of CMM was the specific terminology. The deeper value was the discipline of staged improvement.
It taught me that capability has structure. It taught me that repeatability matters. It taught me that defined processes create a foundation for measurement and improvement. It also taught me that organizations need to be met where they are.
Years later, those lessons helped me build an institutional effectiveness maturity model that could do more than assess. It could guide.
The model gave leaders a way to see uneven maturity without treating it as failure. It gave programs a path to improve without asking them to skip steps. It made sequence visible, from strategic alignment to outcomes, indicators, evidence, interpretation, decision use, and governance.
The lesson still feels current to me.
Improvement starts by understanding where you are. Then you define the next level of capability. Then you build toward it deliberately.
That was true in technology process improvement decades ago. It remains true in institutional effectiveness today.
Comments