Survivor Satisfaction Survey

The aid groups that descended on Haiti after the January 2010 earthquake will most likely get high marks for their work. Why? They’ll be grading themselves.

That’s how it’s done in the disaster-relief industry. This poses two problems. First, self-assessment is subject to bias. Second, it’s quite possible for an aid group to feel confident of having met its goals (for instance, number of meals served) without ever asking if the services offered were what people needed. Disaster relief has been called “the world’s largest unregulated industry.” Who says that? The Red Cross.

Now, two Johns Hopkins researchers want to propel a post-Haiti paradigm shift. Their tool is a questionnaire that asks people affected by a disaster to rate the aid offered; the survey’s co-author, International Health doctoral student Paul C. Perrin, calls it a “customer-satisfaction survey.”

This fall, Perrin, MPH, and Tom Kirsch, MD, MPH ’87, co-director of the Bloomberg School’s Center for Refugee and Disaster Response, plan to travel to the quake zone to test-drive their questionnaire. They have arranged to survey Haitians in and around the post-January 12 tent cities—now home to 1.2 million people. Kirsch and Perrin will employ Creole interpreters to conduct 20-minute interviews of 300 people in 30 temporary settlements in Port-au-Prince and in Léogâne, where the earthquake destroyed 80 percent of the dwellings. They will also interview 300 Haitians who live just outside the camps and who therefore most likely received different forms of help, or none at all. The interviewers will ask people to grade the quality of post-earthquake health care.

If the survey proves useful in Haiti, says Kirsch, it could be used in the future to pinpoint problems early enough so that aid groups could promptly make adjustments. “We could evaluate on a routine basis. We could use this method of asking what needs are met—like, ‘Did you get food today?’—every day or every week, rather than waiting for long-term outcomes like deaths from malnutrition.

Kirsch believes that aid groups are interested in improving evaluation. “Everyone—from NGOs, to the federal government, to the UN—recognizes that having better data collection techniques is important. They recognize that the current methods are not necessarily ideal.”

Two characteristics of Kirsch and Perrin’s project set it apart from surveys that relief organizations might conduct, says Perrin. First, it will be  administered independently, rather than by an aid group scrutinizing its own work. Secondly, the people to be interviewed represent the population as a whole; the researchers will use multistage cluster sampling to locate subjects, potentially including people who received no aid: “We’re trying to do it in a scientifically rigorous way—population-based rather than targeting individual customers,” Perrin says.

The survey asks for ratings of the quality of services in 14 areas. In assessing the sustainability of a relief project, for instance, the researchers will ask whether clinics were staffed by local health workers. In rating efficiency, they will ask whether any health assistance was misused. To gauge participation, they will ask whether marginalized or vulnerable groups received help. And to rate staff professionalism, they will ask whether health workers respected confidentiality. Perrin says that the survey could be used in other countries if it is adapted to suit other cultures and circumstances. It could also be altered to assess the quality of aid in the form of shelter, food, water or sanitation.

Kirsch has seen firsthand the flaws in the Haitian relief effort. He spent two weeks immediately post-earthquake providing medical care in Haiti. One of the biggest problems, he says, was that some aid groups offered help without first asking what was needed. It was common for people to “randomly send garbage” like expired drugs, and for doctors and nurses to show up without knowing where—or whether—they were needed or who would feed and house them, says Kirsch, an associate professor in the Department of Emergency Medicine at the Johns Hopkins School of Medicine and the Bloomberg School’s Department of International Health.

Perrin points out that disasters are increasing because of population growth, urbanization and environmental degradation. According to David Fisher, an expert on legal aspects of disaster relief for the International Federation of Red Cross and Red Crescent Societies, between 2000 and 2006 alone, disasters affected nearly a third of the world’s population.