Nonprofit Comes Clean: Our Program Doesn't Work

In the face of a problem or failure, too many charitable organizations duck and cover: hiding serious dysfunction, glossing over a disappointing result, ignoring deplorable behavior.

But in November 2018 “a major development charity did something remarkable: It admitted” that a pilot program “didn’t seem to work.”  Kelsey Piper at Vox.com writes that this nonprofit’s admission about its program “should be a model” for other 501(c)(3) groups. 

   Poverty Program Tested

Evidence Action is headquartered in Washington, D.C, with a distinguished slate of directors, staff, and advisory board. It operates in ten countries around the world.  The group’s mission is front and center on the website’s home page: “We measurably reduce the burden of poverty for hundreds of millions of people … and we have the data to prove it.”

The organization’s website is clear and transparent; indeed, the recent news about worse-than-expected data was posted immediately to its blog and on its media page. 

The 501(c)(3) was “launched formally in 2013 to scale up poverty interventions that have been proven to be effective in order to benefit millions of people.” The organization’s two key programs are Dispensers for Safe Water and DeWorm the World Initiative, “incubated initially” by Innovations for Poverty Action. “While Evidence Action grew out of a need to manage these programs, the larger goal of the organization was to design a process to bridge the gap between proven interventions that work and scaling them up to produce measurable impact for millions of people.”

Evidence Action Beta followed in 2014 to test other interventions that are “rigorously evaluated” and designed to be “implemented as sustainable programs at scale.” It scores projects “based on impact, cost-effectiveness, scalability, and strategic fit.”

The project in question here – the one that doesn’t work – is No Lean Season: an “innovative program that was created to help poor families in rural Bangladesh during the period between planting and harvesting (typically September to November).” During those months, there is hunger and deprivation because of the lull in jobs and income. This “seasonable poverty” is estimated to affect at least 300 million of Bangladesh’s rural poor. The goal of the No Lean Season project was to give workers small amounts of money during this period so they could travel to jobs available in towns and cities. 

“In small trials, it worked great.” Evidence Action funded the pilot programs and made a huge investment in scaling it up. In 2016, there was a No Lean Season program in 82 villages; that figure jumped to 699 villages the following year. No Lean Season was apparently a success; it made GiveWell’s list of top charities.

But Evidence Action wouldn’t rest on those early results about No Lean Season’s effectiveness. For more data about the program’s “benefits at scale,” it participated in what is considered the “gold standard” for testing effectiveness of these types of “interventions”: an RCT, a rigorous randomized controlled trial. 

   Program Results Come In

In mid-November 2018, the results came in and Evidence Action reported them, here, including why No Lean Season did not perform at scale as hoped. Simply put, the program didn’t have the expected effect on inducing rural workers to temporarily migrant to urban areas, so the hoped-for increase in income never happened. 

This admission was a big deal in development circles: “It is exceptionally rare for a charity to participate in research, conclude that the research suggests its program as implemented doesn’t work and publicize those results in a major announcement to donors.”

Evidence Action also rejected the understandable path of concealing the results, contesting them, or explaining them away; for example, blaming the outcome on unusual floods in 2017 in Bangladesh. The organization also avoided the temptation to put a positive spin on the results; for example, saying the results were “discouraging” but not “devastating.”

Putting into action the organization’s stated “values” of putting “evidence first,” it announced it will use existing results “to make program improvements and pivots.” Then it will continue rigorous testing to check out whether the changes have produced the originally expected outcomes.

Most significantly, Evidence Action will seek no additional funding for No Lean Season until and unless the program modifications test out as successful.

   Challenges to Transparency

Calling this “a stunning development,” Kelsey Piper, the Vox reporter who publicized this story adds that “it would be great if  every charity was as careful and honest as Evidence Action” but it’s “immensely challenging” for several reasons:

• • Most organizations are staffed with true believers in the mission;

• • They worry that transparency will spook donors;

• • “Research can sometimes be confusing instead of clarifying”; 

• • Research can be prohibitively expensive; and 
  • Many charities have a single project only; a negative report may doom the future of that project and the organization itself.

     Conclusion

Of course, Evidence Action was built from the ground up to be able to do the right thing: its “founding commitment” is to rigorously test programs to see if they scale up successfully. It “expects that some of the programs it incubates” will fail,” and has in-house expertise to do the testing and analyze the results. There is “institutional freedom” to admit failure in one or another pilot program.

Nevertheless, credit where credit is due: “Evidence Action’s transparency, good epistemics, and commitment to getting No Lean Season right is highly encouraging.”  The writer was so impressed, she says, that she made a donation.