the perversity of measures

Setting impact measures is one of the most difficult tasks in results-based management

​One time I was working for a client that  I am helping get back on track when it comes to performance measurement framework. The conversation went into the loftiness of some of the goals that was set up by a funding institution that they are supposed to align to with their project. This is a standard BHIG- big, hairy, audacious goal that nobody can actually put a critical baseline measure on because in the grand scheme of things, it is way up there. It is complex. It accounts for very wide dimension of change that one can have a hundred of discrete  interpretations of the phenomenon to render any meaningful judgment on what constitute that change.

In this day and age, when funding companies and organizations mandate specific outcomes for applying organizations to align begs the question of forcing the alignment. Harmonized goals are good for standard-setting, for measuring collective impact and to promote joint accountability across various projects and actors working in the same field.

Sometimes, the interpretation of change of   beneficiaries does not necessarily conform to the worldviews and expectations of the bureaucrats who are designing these tools. Most of the time, this top-down arrangement defeats the very purpose of meaningful performance measurement.

Which perspective counts, for whom, and for what purpose? And whose perspectives dominate the narrative of change? Programmers must be aware and astute in managing their horizontal and vertical alliances and ensure that there is an equilibrium state. Programmers must be able to bridge the gap between local knowledge and wisdom and those powers that be that tend to know it all. Complex change tend to resist boxed solutions. It needs complexity-based solutions.

I quote Bruce Cameron when he said, not everything that can be counted counts. Not everything that counts can be counted. This is precisely the point about measuring key performance in the organization. The executives need to count the things that contribute to their outcomes and that within their control. The outputs of activities are lower level indicators of how busy the organization is, but not how it made an impact in that community.  Number of trainings conducted  or number of wells built is just that. It can lead to incentivizing outputs over what matters at the end of the day- the impact it had on the beneficiaries. Staff will focus their efforts on meeting their targets or exceeding their targets, resources poured to achieve activity levels, contractors and consultants being brought over from developed country to assist the change processes, etc.

​Yet when the time comes to assess and evaluate the project, there is always that feeling of frustration, that nagging feeling that it wasn’t enough. Blame it on the M & E or the lack of it,  the problem is not because they did not deliver, (sometimes they over-deliver) but there measures failed them.

Busyness is the number one problem of most non-profits and initiatives with social mission. Doing things over and over again and expecting a different outcome is not a good trait. Translate that to organizations that make activities after activities, falling flat from the weight of their own doing without  getting near to their destination: the outcomes they exist or work for.

 In this day and age when everything that gets counted, gets done or managed is absolutely spot-on. W hat is not counted goes  to the backburner and rarely seen the light of day. Remember polio when no body knows what it is and now according to World Health Organization, they are close to its eradication by 99%. That precision is a  result of collective work with shared success measurement.  While there is a tendency to quantify the outcomes, some outcomes do not lend itself easily to quantification. Qualitative, context-based method will be needed to provide a fuller and robust picture of the performance story and provide the how that is badly required by all stakeholders these days. The basket of indicators approach tends to be the best way to calibrate between contending measures to provide counterfactuals.

When presented with a few options for indicators, one should be careful not to put too much emphasis on one over the other, as it can create perverse consequences. Take the fact of malaria. The global efforts to end malaria has had direct unexpected, unpredicted, dangerous consequences that investment in malaria prevention has risen to other diseases ( that were not at critical levels before) because of the under funding of health institutions and underinvestment of practices that promote integrated health and well-being of people and communities in the developing countries.

​Setting performance measures is one of the most difficult tasks in developing a project. Take time to consult the beneficiaries in terms of the longer and stable changes they want to see happen in their lives and put that specificity in the outcomes. If something has to give, costs should give way to quality. It will save you dollars done the road when it is done properly and having in mind the right approach, method, tools, and mindset  in place.

Let me know what you think. Did I miss an important point?

​Cheers!