Global Policy Forum

By Robert Glasser

No one could deny that the mushrooming NGO sector does good work in providing on-the-ground humanitarian relief. But Robert Glasser, Secretary General of CARE International, says that evaluations of their effectiveness "have been patchy at best" When disaster strikes, non-governmental organisations are among the first on the scene. It's a pattern that has become increasingly familiar. The United Nations Development Program (UNDP) estimates there are now more than 37,000 international NGOs following the surge in their numbers in the 1990s when major donors started to rely on them more and more. Inevitably there have been problems. Both the Rwandan genocide in 1994 and the 2004 Indian Ocean tsunami saw chaotic competition between hundreds of NGOs as they all scrambled to help. Yet their have also been landmark victories. More than 1,400 NGOs operating in 90 countries were able to exert sufficient political pressure to get 123 countries to ratify the treaty banning land mines; it earned the campaign organisers the Nobel peace prize.

It's not just the number of NGOs that is making a difference, but also the trend towards trans-national alliances to achieve common goals collectively. But the sheer scale of this disaster relief "industry" - plus the longer-term development efforts of NGOs - is raising serious issues about how to measure their performance. Among the questions being asked by NGOs and by UN and national donors, has been how to prevent past mistakes from being repeated. There are few clearly defined international rules on what an NGO actually is. Flexibility allows NGOs to be innovative in ways that organisations like the UN often cannot. But the same lack of control also leads to unpredictable consequences. In one recent case in Chad, the French NGO L'Arche de Zoé tried to smuggle children out of the country without obtaining permission from either their parents or the government. Less dramatic, but potentially more serious, are occasions when an inexperienced group leaps into a situation which it is unable to handle.

The wake-up call for most NGOs came after the Rwandan genocide, when hundreds of small organisations tried to set up ad hoc operations in refugee camps in the Democratic Republic of the Congo and in Tanzania. As they jostled for space, some camps turned into staging posts for armed factions. In the ensuing chaos, an estimated 50,000 refugees died from cholera. Many attempts to provide relief proved either redundant or ineffective. There was also mayhem in the relief efforts that followed the Indian Ocean tsunami. At one point, more than 400 NGOs were on the ground in Aceh, Indonesia, competing for resources, personnel and funding. Many of the lessons that had been learned during the Rwandan crisis were forgotten or simply ignored. The confusion was caused mostly by smaller NGOs with little or no experience in dealing with disasters.

The relief community did eventually take on board lessons from both Rwanda and the tsunami. The situation in Indonesia led the UN to adopt a new "cluster" system to improve coordination. And after a review of the Rwandan debacle, 400 NGOs and UN organisations working in 80 countries got together in the Sphere Project to develop a common humanitarian mandate and a handbook of standards. This outlines the minimum performance required of any NGO working in a disaster zone. At the same time, leading NGOs created the Active Learning Network for Accountability and Performance in Humanitarian Action, known as ALNAP, to exchange the latest ideas on accountability. It was followed by the Humanitarian Accountability Project, which evolved into the Humanitarian Accountability Partnership-International. This currently includes 18 NGOs as full members and focuses on reporting back to the people who have been directly affected by a disaster and who received humanitarian aid.

As the number of post-intervention reviews increased, some of the biggest NGOs appointed full time staff to improve the effectiveness of their evaluations. Seven of the largest NGOs have also joined forces in the Emergency Capacity Building Project, funded by the Gates Foundation, to explore inter-agency collaboration on assessments and "after action" reviews. This has helped create a rudimentary framework for evaluating the impact of NGOs' humanitarian work, but there is still considerable debate over what really needs to be evaluated, and how.

Over the last decade or so, there has been a major shift in the focus of NGO evaluations. Rather than simply looking at project inputs and outputs, the emphasis has turned towards measuring the overall impact of an operation. The basic idea is to find out if the lives of the people on the receiving end were changed for the better in any sustained way. More and more donors are also insisting that NGOs provide measurable proof that they make a difference. While this sounds fine in theory, in practice there are drawbacks. By demanding quantifiable results, donors may force programme managers to choose easily achieved targets in preference to actions which - though less measurable - accord with sound humanitarian principles. Or reporting of aid programmes may be skewed to keep donor funds flowing. The greatest danger is that humanitarian relief will be tailored to meet the demands of donors, rather than being dictated by the type of aid that is needed on the ground.

There are other pressures on NGOs too. If they take too much account of the demands of local beneficiaries, they may lose sight of the bigger picture. For instance, food aid may prevent an immediate famine, but it can also undercut local farmers and so contribute to future starvation. Thus NGOs would rather reduce the emphasis on measuring results of their programmes in favour of a more balanced mixture of evaluation methods.

Still, there is no doubt that, until recently, the record on evaluating responses to humanitarian emergencies has been patchy at best. There are many reasons for this. CARE, as both a relief and development agency, can take a long-term approach to disasters, matching emergency relief with a rehabilitation and recovery phase. But this is not an option for NGOs which focus only on emergency responses. Once their allotted time is up - or their funds run out - they tend to pack up and leave. Even for those NGOS that stick around, determining the impact of their relief efforts in the middle of a crisis is either difficult or well-nigh impossible. Emergencies are chaotic; staff and resources are stretched, and the local population is very unlikely to be able to provide meaningful feedback. Even if you manage to conduct an on-the-spot assessment, you're unlikely to be able to gather much pre-crisis baseline data, so comparisons are complicated. And, all too often, events move too quickly to be measured accurately. Also, until recently, donors who were willing to pay for relief were less likely to finance follow-up evaluations.

The result is that emergency relief evaluations often rely on little more than guesswork and assumptions. A 2004 report by the Humanitarian Policy Group cited a survey carried out in Ethiopia after UN agencies said that widespread famine had been averted in 2000 by humanitarian efforts. The claim sounded credible until the subsequent survey showed that the crude mortality rate in the area had actually risen to six times the normal base rate. Most of the deaths were from communicable diseases which malnourished people may well have contracted after crowding into feeding centres.

The HPG therefore recommended very long-term monitoring of humanitarian responses in future, and said that success or failure should be judged in a broad context rather than by a narrow focus on a specific project. Many people who survive an earthquake or a flood, for instance, may soon face another crisis if the disaster also destroys their only means of earning a living. New and more sophisticated analytical tools are needed to understand these long-term effects, along with sufficient training to make sure that new methods are applied properly in the field. A recent innovation has been the Coping Strategy Index, devised by the World Food Programme and CARE, which analyses the way people cope with short-term food crises while also taking into account their future vulnerability to hunger.

NGOs are now able to get closer than ever to local communities and offer a voice to some of the most disenfranchised people on earth. These NGOs also operate in politically sensitive environments that are closed to more formal institutions. The world's leading NGOs advise the UN and help to shape its current reform efforts; they are also on hand whenever the UN High Commissioner for Refugees, the World Food Programme and other international donors need to feed thousands of refugees. NGOs do the lion's share of the world's humanitarian work, so some mistakes are certainly made along the way. But as we deepen our experience of humanitarian relief and development, we learn the lessons of the past and understand how much more there is to know.