Impact measurement: a wild & final frontier.
Insights for changemakers trying to articulate & demonstrate impact in the wild west
Impact measurement is a wild frontier.
There’s not a common framework or language around impact measurement in the social sector. Even popular concepts like the theory of change, logic model, and SROI have many formats and approaches. How is impact even defined in the first place, anyway? Much less, how do you track it?
There are slick online tools like Sopact and Clear Impact. Acumen spun out its measurement play called 60 Decibels and it hopes to ‘reboot the way we all understand impact.’ The Impact Management Project is working to ‘build global consensus on how to measure and manage impact’ and 2,000 big hitters are backing the effort. Then more than 50,000 businesses (like us!) have used The B Impact Assessment to measure impact on workers, communities, the environment, and customers. To name a few.
No wonder so many social entrepreneurs are confused.
But many funders don’t appear to be that much better off in terms of clarity or consistency. Most foundations and investors have proprietary reporting requirements (and obligatory blog posts about ‘how we measure impact’). Major funds have tried their hand at efforts like an SDG measurement toolkit. Even Bono got in the game earlier this year with the launch of Y Analytics alongside a middle-market investment firm. Then Sorenson Impact Foundation recently concluded in Forbes that impact measurement is investing’s final frontier. So it’s no surprise certain impact investors have asked if it’s time to admit defeat.
After speaking on a panel this month at the East Africa Impact Investment Summit (with a room full of both doers and donors), one thing remained obvious. Changemakers feel like they’re in the wild west by trying to articulate and demonstrate impact.
The reality is that this stuff is hard. Meanwhile, the private sector has it easy. Nobody is asking if Apple is driving impact when it sells millions of iPhones. Even bad-actor fast food brands don’t have to answer to much other than shareholder value. Success is measured by revenue or profit. That’s it. The balance sheet has been around for more than 500 years. And even more modern tools like the business model canvas have become ubiquitous.
So what’s a changemaker to do? Most growth-stage NGOs and social enterprises don’t have full-fledged monitoring and evaluation teams. But they need evidence to scale. While we’re not an M&E firm, we do help clients design and document impact models on the front end. Then, more importantly, we aim to articulate and communicate that impact to the outside world after the good work is done.
So looking through our brand strategy, org health, and marcom lens, a few key tenets apply for any changemaker embarking upon the journey of impact measurement and articulation.
Five insights for growth-stage changemakers.
If you’re a changemaker reading this, you’re likely not yet an M&E whiz. That’s ok. You have to start somewhere. So here are five insights we gave at the Impact Investment Summit for growth-stage changemakers like you.
DON’T SWEAT THE FORMATS & FRAMEWORKS
Use the tool or template that works well for your organization and your funders, even if it’s a rudimentary whiteboard diagram. One of the most prestigious funders out there – Mulago Foundation – uses an exceptionally simple document called the Design Iteration Format. While the questions take great thought, the DIF itself is just text on a page with an arrow or two. At Mighty Ally, we work to clarify an impact model through 12 big questions then a one-page summary blueprint at the end (plus sometimes a graphical representation). So don’t labor over what it looks like; sweat the thinking behind the model and details within.
12 IMPACT MODEL questions
1. What is the problem being addressed?
2. Which group of people are you serving?
3. What are you trying to achieve?
4. What is your central, distinctive idea?
5. Which behaviors do you need to influence?
6. What resources are required?
7. What are your key interventions or activities?
8. What immediate results will these activities produce?
9. How will these results lead to changes in conditions?
10. What is the ultimate end-game vision?
11. Which partnerships are needed outside your core work?
12. How will you replicate and fund the model?
FIND, MEASURE, AND COMMUNICATE THE CORE
Orville Redenbacher spent 40 years perfecting his popcorn recipe. Then later famously said, “Do one thing and do it better than anyone.” Truth is, most changemakers have multiple programs or products, because complex social change requires complex solutions. But there are typically one or two elements that the rest hinges on. Identify that core, develop evidence measures around that core, and communicate that core. Then add to it and innovate around it over time. ‘The core’ to us means the things you have to do always everywhere vs. those you can do sometimes in some places. Here’s one example of great discipline and clarity. America’s oft-admired nonprofit, Charity:water, also builds latrines. But you wouldn’t know it because their impact is wisely measured and communicated around one digestible thing… water.
GRADUATE BEYOND ACTIVITIES & OUTPUTS
Despite the classic logic model existing for decades, many changemakers stop short at measuring outputs alone. Outputs are the immediate, often numerical results of activities. While these data points are important to link causes and effects, outputs are just means to an end. And most savvy funders are now demanding evidence around outcomes and impact: actual changes in conditions over time. Often changemakers think it’s too hard to measure outcomes without an M&E team. But it’s not too hard to start. If you’re working with at-risk youth, your theory of change might say an outcome is an increase in confidence. So don’t just measure the number of young adults in your program. That’s a vanity metric. At least survey participants at the beginning and end of the intervention and ask on a five-point Likert scale, “Do you feel in control of your future?” And there you have your first outcome measurement around increased confidence.
MIX STORIES AND DATA WHEN M&E IS WEAK
As much as this piece focuses on real metrics, the reality is that stories matter too. As a changemaker, your job is to articulate, measure, and communicate impact. Like the tree falling in the woods – all the impact in the world won’t matter without telling a compelling story to funders, volunteers, partners, and other audiences. And because these are all humans on the receiving end who think and feel, you have to balance the head stuff with heart. Head and heart, stories and data. Aggregate data to show repeatability and reach, with individual examples to show depth and real change. Kenyan community-health NGO Lwala is a great example: university-backed evidence, with beautiful photography and personal stories of impact. But here’s the tough love: stories get you into the room, but data lands the deal. You can hook funders with a good story like other homo sapiens. But when it’s time to compare your work against others in the space or ask for a check, the measurement conversation will come up.
BE SURE TO OPERATIONALIZE THE IMPACT MODEL
At some point, you might have the next game-changing idea on your hands and even burgeoning M&E systems to measure it. But unless you have the operations in place to implement the model, it will all be for naught. This is where you – as a leader – turn from working in the business to on the business. Meaning, you can no longer stay up at night toiling over your programs, products, and beneficiaries alone. You now must fixate on building a healthy organization. This includes an equal focus on leadership, people, priorities, rhythms, and key performance indicators. And sadly, too many changemakers don’t make this all-important leap. So most promising social ventures stall. In fact, over a 40 year period, Bridgespan Group found that 200,000 nonprofits started in the United States and only 144 reached $50MM in annual revenue. The missing link is organizational health: both a philosophical approach and a deliberate set of actions through which a business sets a solid foundation for everything else to grow. Org health is the not-so-hidden key to building a high-performing NGO or social enterprise. One that gives you the best chance of impact and scale alike.
Try these resources on for size.
In addition to the links above, here are some resources that might help in the quest.
Oxford Impact Measurement Programme: webinar on Impact Measurement in Practice – Lean Data & Impact Management.
Mulago Foundation: 'simple enough to do, rigorous enough to mean something' impact measurement.
Philanthropy University: one-month online course, Planning for Monitoring and Evaluation.
Coursera: Introduction to Impact Measurement, from the University of Pennsylvania.
Wishing you well in the wild and final frontier.