- 11 марта 2013, 19:00
- Harvard Business Review
Everyone agrees that social enterprises need to use data to assess their effectiveness. But precisely what data should organizations capture? How should they incorporate it into meaningful evaluations that prove their approaches work? And, how should potential funders analyze these evaluations to determine which enterprises are most effective?
At GiveWell (where I'm co-founder) we've analyzed data and evaluations from hundreds of social enterprises and programs to identify standout organizations. We serve donors who want to be confident that their money is going to the most effective programs they can find. The best data and evaluations we've seen have two characteristics:
- They're balanced. The data shared has a good chance of demonstrating that an organization's program is ineffective. Many evaluations we've seen appear to be stacked in the organization's favor from the start, and we wonder whether the evaluation could possibly have reached a negative conclusion and, if it did, whether it would ever have seen the light of day.
- They explore alternative causes. The evaluation should consider alternative hypotheses for the observed changes. For example, if the program provides training for farmers and observes rises in farmer incomes, the evaluation should question whether other factors — such as rainfall or changes in the accessibility of local markets — could have driven the changes.
Let's look at two charities — Village Reach and Against Malaria Foundation — that have produced evaluations that demonstrate these qualities.
VillageReach, an organization focused on the last mile delivery of vaccines to remote rural areas in Mozambique, produced the best evaluation we've seen from an organization itself (as opposed to academics). The evaluation stands out because it collects meaningful data directly connected to the program's social impact, describes in detail its methodology, and discusses its inherent limitations — all characteristics we've rarely seen. The data make the case that the program significantly increased immunizations rates in the province in which it worked, where childhood immunization rates increased from less than 70% to more than 90% during the course of the project.
Among other measures, the evaluation presents data on how often health clinics were out of vaccines when VillageReach monitors visited and the change in immunization rates. These data could have shown that the program failed: had immunization rates stayed flat or stockout rates stayed high, we would have known that the approach didn't work. In fact, the evaluation showed large changes in both measures.
Nevertheless, it still leaves some important questions unanswered. Did VillageReach cause the increase in immunization rates or did other factors, perhaps related to Northern Mozambique's emergence from a recent civil war and subsequent donor interest in the area, cause the improvement? To its credit, the organization addresses alternative hypotheses in its evaluation, even if it was not able to arrive at compelling answers to these questions. We don't have reason to believe that other factors were the primary driver of change, but we also have no way of knowing that they were not. The data made a case for impact but, necessarily and through no fault of VillageReach's, left important questions of possible alternative causes of the measured changes unanswered.
Against Malaria Foundation (AMF)
AMF, which funds distribution of malaria nets, also makes a compelling case but through the use of independent evidence. The organization cites more than 20 randomized controlled trials that demonstrate that distributing nets saves lives. They also rely on additional evidence collected from household surveys across many countries by MEASURE DHS and the World Health Organization to show that the people who receive nets distributed in the type of real-world, large-scale distributions that AMF supports use them about as consistently as those who received them in the rigorous trials.
The points above make a strong case that net distribution causes a reduction in malaria deaths. But AMF supplements the case with data it collects itself, data that would enable it (and outsiders) to know if its program were failing. Before distributing nets, the organization conducts pre-distribution surveys from all potential recipient households to determine whether they need additional nets. It then performs post-distribution surveys every six months from a sample of recipients to monitor net usage rates. The organization also collects malaria case rate data from local health centers to monitor malaria rates in the regions its serves.
The combination of independent evidence (which addresses potential alternative causes of impact) and AMF-collected data (which would tell us if its program were failing) makes a compelling case for impact.
Of the hundreds of organizations we've looked at, we've rarely seen cases for impact like those described above. Why is that?
Donors may be the problem. When funders give to organizations based on vague and superficial stories, rely on poor evaluations, or don't critically assess programs' impacts at all, they show organizations that there's no need to produce high-quality evaluations. Even worse, some donors give to organizations with the lowest "overhead ratios" as if that's a proxy for effectiveness, making investment in assessment near impossible. We want to change the conversation around giving from one that assumes that all well-intentioned people are accomplishing great things to a more open, critical discussion of what organizations do and how well it works.
Insights from HBR and The Bridgespan Group