Global Policy Forum

By Celia Dugger

Wealthy nations and international organizations, including the World Bank, spend more than $55 billion annually to better the lot of the world's 2.7 billion poor people. Yet they have scant evidence that the myriad projects they finance have made any real difference, many economists say.

That important fact has left some critics of the World Bank, the largest financier of antipoverty programs in developing countries, dissatisfied, and they have begun throwing down an essential challenge. It is not enough, they say, just to measure how many miles of roads are built, schools constructed or microcredit loans provided. You must also measure whether those investments actually help poor people live longer, more prosperous lives.

It is a common-sense approach that is harder than it sounds, just like the question it seeks to answer: Does aid really work?

A small band of development economists, who a year ago founded the Poverty Action Lab at the Massachusetts Institute of Technology, have become influential advocates for randomized evaluations as the best way to answer that question. Such trials, generally regarded as the gold standard in social policy research, involve randomly assigning people eligible for an antipoverty program to get the help or not, then comparing outcomes to see whether those who got the help fared better than those who did not.

It is the same approach that has helped drug companies figure out what medicines are effective and Americans decide how best to reform welfare. Advocates for rigorous evaluations hope to make aid more effective, not by directing money to particular countries, but by spending it on programs proven to work.

The Poverty Action Lab scholars have made startling discoveries in their own randomized evaluations.

Adding an extra teacher to classrooms in rural India did not improve children's test scores. But hiring high-school graduates who were paid only $10 to $15 a month to give remedial tutoring to groups of lagging students in a Bombay slum markedly improved reading and math skills.

A series of education experiments in Kenya found that providing poor students with free uniforms or a simple porridge breakfast substantially increased attendance. But giving them drugs to treat the intestinal worms that infect more than a quarter of the world's population was more cost effective, with a price tag of only $3.50 for each extra year of schooling achieved. Healthier children are more likely to go to school. "You can't answer the general question: Does aid work?'' said Esther Duflo, an economist and co-founder of the Poverty Action Lab."You have to go project by project and accumulate the evidence.''

The World Bank, a lumbering giant that employs more than 1,200 Ph.D.'s, is beginning to listen to critics like her. This summer, it is organizing large-scale impact evaluations, including randomized trials, of programs to upgrade slums, improve the performance of schools and keep children healthy and in class. The programs will be tested in dozens of countries.

Franí§ois Bourguignon, the bank's chief economist, said he hoped this new effort would help the bank, other donors and developing countries "learn what does and does not work."

Rigorous impact evaluations should become part of the bank's culture, he said.

That will require deep change. A recent in-house review of bank projects during the past four to five years found that only 2 percent had been properly evaluated for whether they made a difference, according to Mr. Bourguignon.

When Lant Pritchett, an economist who has spent a dozen years at the bank, pondered why there was so little good evidence on the impact of projects it financed, a tune from an old game show spoof that his Mom used to sing popped into his head: "It pays to be ignorant, to be dumb, to be dense . . ."

Bank economists have recently produced assessments of huge development initiatives that acknowledge weaknesses in the evidence.

A critical review of the bank's $7 billion portfolio of programs that involve local communities in their design and management concluded recently that "there are, unfortunately, a dearth of well-designed evaluations of such projects."

Another review of a $1.3 billion initiative in India found similar problems. Bank economists in New Delhi examined more than 200 studies of projects in India that ranged from teacher training to school construction, enrollment drives to textbook revision.

They concluded that none of the studies were rigorous enough to measure whether the initiatives made a difference, except for one that found it increased enrollment by a disappointing 1.3 percent. "The World Bank spent more than a billion dollars without knowing why they were doing what they were doing - that's the tragedy,'' said Abhijit Banerjee, an M.I.T. economics professor and co-founder of the Poverty Action Lab.

But even as aid agencies lagged in conducting stringent evaluations, Professors Banerjee and Duflo at M.I.T., Michael Kremer at Harvard and other economists associated with the lab have been conducting randomized trials of antipoverty programs in India, Kenya, South Africa, Peru and the Philippines.

Even they acknowledge that random evaluations are not a panacea. For example, a program that works in Asia may not work in Africa. Still, they say, the trials offer the best evidence.

"This rigorous testing has made a huge difference in medicine and has improved human welfare due to better drugs,'' said Professor Kremer. "If we could use randomized evaluations to really find out what works, foreign aid donors could implement better health and education policies and so could developing countries.''

Mr. Pritchett, a veteran bank economist, tried to explain why rigorous evaluations were such a rarity in the culture of the bank. Its highly trained, well-meaning professionals too often think they know the solutions. "They have too little doubt,'' he said.

They also worry that modest, proven gains for the poor will lose out to inflated, unproven claims for, say, tax cuts to the rich or a new weapons system - a concern he shares. "You want to know what works and what doesn't, but until you subject the full range of government spending to the same discipline, why are you disadvantaging things for poor people?'' he asked.

But Professor Banerjee is optimistic that reliable evaluations will give advocates ammunition to lobby for increased foreign aid.

He pointed to the success of a rigorously studied Mexican program that paid poor mothers a small sum if they kept their children in school and got them immunized. The model has spread across Latin America in large measure because a large randomized trial, published in 2001, showed that the children who participated were healthier and stayed in school longer.

"In the development business,'' he said, "it would be really good to get away from the need to have people promising miracles.''