Tag Archives: Evaluation

Disaster Response, Self-Organization and Resilience: Shocking Insights from the Haiti Humanitarian Assistance Evaluation

Tulane University and the State University of Haiti just released a rather damming evaluation of the humanitarian response to the 2010 earthquake that struck Haiti on January 12th. The comprehensive assessment, which takes a participatory approach and applies a novel resilience framework, finds that despite several billion dollars in “aid”, humanitarian assistance did not make a detectable contribution to the resilience of the Haitian population and in some cases increased certain communities’ vulnerability and even caused harm. Welcome to supply-side humanitarian assistance directed by external actors.

“All we need is information. Why can’t we get information?” A quote taken from one of many focus groups conducted by the evaluators. “There was little to no information exchange between the international community tasked with humanitarian response and the Haitian NGOs, civil society or affected persons / communities themselves.” Information is critical for effective humanitarian assistance, which should include two objectives: “preventing excess mortality and human suffering in the immediate, and in the longer term, improving the community’s ability to respond to potential future shocks.” This longer term objective thus focuses on resilience, which the evaluation team defines as follows:

“Resilience is the capacity of the affected community to self-organize, learn from and vigorously recover from adverse situations stronger than it was before.”

This link between resilience and capacity for self-organization is truly profound and incredibly important. To be sure, the evaluation reveals that “the humani-tarian response frequently undermined the capacity of Haitian individuals and organizations.” This completely violates the Hippocratic Oath of Do No Harm. The evaluators thus “promote the attainment of self-sufficiency, rather than the ongoing dependency on standard humanitarian assistance.” Indeed, “focus groups indicated that solutions to help people help themselves were desired.”

I find it particularly telling that many aid organizations interviewed for this assessment were reluctant to assist the evaluators in fully capturing and analyzing resource flows, which are critical for impact evaluation. “The lack of transparency in program dispersal of resources was a major constraint in our research of effective program evaluation.” To this end, the evaluation team argue that “by strengthening Haitian institutions’ ability to monitor and evaluate, Haitians will more easily be able to track and monitor international efforts.”

I completely disagree with this remedy. The institutions are part of the problem, and besides, institution-building takes years if not decades. To assume there is even political will and the resources for such efforts is at best misguided. If resilience is about strengthening the capacity of affected communities to self-organize, then I would focus on just that, applying existing technologies and processes that both catalyze and facilitate demand-side, people-centered self-organization. My previous blog post on “Technology and Building Resilient Societies to Mitigate the Impact of Disasters” elaborates on this point.

In sum, “resilience is the critical link between disaster and development; monitoring it will ensure that relief efforts are supporting, and not eroding, household and community capabilities.” This explains why crowdsourcing and data mining efforts like those of Ushahidi, HealthMap and UN Global Pulse are important for disaster response, self-organization and resilience.

On Technology and Learning, Or Why the Wright Brothers Did Not Create the 747

I am continuously amazed by critics who outlaw learning curves, especially when it comes to new and innovative projects. These critics expect instantaneous perfection from the outset even when new technologies are involved. So when a new project doesn’t meet their satisfaction (based on their own, often arbitrary measurement of success), they publicly castigate the groups behind the projects for their “failures”.

What would the world look like if these critics were in charge? There would be little to no innovation, progress or breakthrough’s. Isaac Newton would never have spoken the words “If I have seen so far it is because I have stood on the shoulders of giants.” There would be no giants, no perspective, and no accumulation of knowledge.

Take the Wright brothers, Orville and Wilbur, who invented and built the world’s first successful airplane by developing controls that made fixed-wing powered flight possible. They too had their critics. Some of them in “the European aviation community had converted the press to an anti-Wright brothers stance. European newspapers, especially in France, were openly derisive, calling them bluffers.” The Wright brothers certainly failed on numerous occasions. But they kept tinkering and experimenting. It was their failures that eventually made them successful. As the Chinese proverb goes, “Failure is not falling down but refusing to get up.”

The Wright brothers did not create the 747 because they had to start from scratch. They first had to figure out the basic principles of flight. Today’s critics of new technology-for-social impact projects are basically blaming NGOs in the developing world for not creating 747s.

Finally, it is problematic that many critics of technology-for-social impact projects have little to no scholarly or professional background in monitoring and evaluation (M&E). These critics think that because they are technology or development experts they know how to evaluate projects—one of the most common mistakes made by self-styled evaluators as noted at the start of The Fletcher School’s introductory course on M&E.

The field of M&E is a science, not an amateur sport. M&E is an independent, specialized field of expertise in it’s own right, one that requires months (if not several semesters) of dedicated study and training.

Patrick Philippe Meier

From Baselines to Basemaps: Crisis Mapping for Monitoring & Evaluation (M&E)

I was just in Berlin for meetings with Transparency International and the topic of mapping for Monitoring & Evaluation came up yet again. Earlier this year, Mercy Corps and UNDP Sudan both expressed an interest in exploring the application of crisis mapping platforms for M&E. Problem is, the field of M&E—particularly with regards to peacebuilding and post-conflict reconstruction—is devoid of any references to mapping.

As part of my consulting work with UNDP Sudan, I therefore produced a short concept paper this topic back in June. Here’s a summary of what I wrote.

Introduction

Peacebuilding and post-conflict reconstruction programs necessarily operate within a dynamic environment. This means that they must adapt to changing circumstances or else run the risk of misallocating resources or worse, exacerbating tensions not to mention creating new sources of violent conflict. Hence the need for conflict sensitive programming, which UNDP’s TRMA initiative is designed to support in the Sudan.

The threat and risk maps produced by UNDP provide spatial risk assessments that can inform programmatic response in Sudan’s post-conflict states. This need not be a one-off decision-support exercise, however. Indeed, the use of spatial risk assessments updated over time is an even more compelling use of crisis maps for decision-support.

A changing post-conflict environment means that projects designed half-a-year ago may no longer be having the intended impact they were funded to have. To this end, it is important that UN and local-government partners have regular updates on the changing context in order to adapt programming respectively. Crisis mapping can play a pivotal role in this decision support process.

Basemapping

I therefore propose a new approach to crisis mapping called “basemapping”. The purpose of basemapping is to combine M&E and crisis mapping to produce basemaps against which projects can be monitored and evaluated. The basemapping process constitutes three distinct mapping steps:

1. Ideal World Basemapping: mapping the ideal world that a given project seeks to achieve over a given period of time.

2. Real World Basemapping: mapping the current state of affairs in the specific world that the project seeks to change

3. Changed World Basemapping: ongoing mapping to compare the change between the ideal world and real world basemaps.

The following section provides a brief background to M&E and formulates a proposed methodology for basemapping. As basemapping is a new concept that has little to no precedent, the purpose of the proposed methodology is to catalyze discussion on the subject and to make the methodology more sophisticated over time.

Monitoring and Evaluation (M&E)

A strong M&E framework will include theories and types of change, an achievable goal with clear objectives, outputs and activities as well as reliable indicators and baselines. However, M&E frameworks should be case-specific and must be tailored to the purpose of individual projects. This deliverable assumes that UNDP is already well versed in M&E frameworks and methodologies. The analysis that follows thus focuses specifically on the contributing role of crisis mapping to the M&E process.

Baselines are the most often forgotten component within design, monitoring and evaluation, yet they are key to proving that change has truly taken place. They also provide the most intuitive link to crisis mapping. While baselines typically represent a snapshot in time and space against which deviations represent progress or failure, the concept of “basemaps” can play the same role albeit dynamically.

Perhaps the closest analogy to baselines in the field of conflict analysis is the conflict assessment. A conflict assessment is an exploration of the realities of the conflict and an analysis of its underlying causes. An assessment can be done at any time, independently of a program or as a part of an existing program. Assessments are often conducted to determine whether an intervention is needed and, if so, what type of intervention.

Assessments (or risk assessments) are also typically carried out as a first step in the development of a conflict early warning system. They serve to identify appropriate conflict early warning indicators. In a sense, an assessment is the basis from which the programming will be designed. Conversely, a baseline identifies the status of the targeted change before the project starts but after it has been designed.

M&E experts caution that assessments and baselines should not be blended together. Nor do they suggest using one as a substitute for the other since their raison d’être, focus, and implementation are very different. We beg to differ and would even go so far as proposing that dynamic crisis mapping platforms can bring both conflict assessments and M&E baselines together with considerable added value.

To explain the potential of basemaps in more detail, a sound understanding of baselines is important. A baseline provides a starting point or reference from which a comparison can be made.  Baselines are conducted prior to the beginning of a program intervention and are the point of comparison for monitoring and evaluation data. The bulk of baseline studies focus on the intended outcomes of a project. They can also take into account secondary outcomes and assumptions, though these are not the primary emphasis.

Baseline information can be used in a number of ways. Perhaps the most intuitive application is the comparison of baseline information with subsequent information to show the change (or lack thereof) that has taken place over time (again, change over space is all-too often ignored). Baseline information can also be used to refine programming decisions or set achievable and realistic targets. Finally, baseline information enables monitoring data to have greater utility earlier in the project cycle.

Baselines have three possible focus areas: change, secondary outcomes and assumptions: change, secondary changes and assumptions. It is important that DAI and partners agree on the formulation of an M&E framework that clearly focuses on one of these focus areas. Note that the first area, change, is required of all baselines, while the other two are optional depending on the project.

Towards Dynamic Basemaps

Conflict, or threat and risk data, typically has a geographic dimension. Consequently, baseline data on conflict dynamics also have a geographic element. To this end, it makes far more sense to use basemaps than baselines since the former includes spatially relevant information that changes over time, which is necessarily important for decision-making vis-à-vis programmatic response in post-conflict environments.

UNDP’s TRMA enjoys a distinctive advantage in this respect since the project already maps threat and risk data (via hard-copy print-outs and a dynamic mapping tools called the 4W’s). The main handicap at the moment is the lack of regularly updated threat and risk data in order to visualize change over time. That said, the TRMA team is planning to shift towards more regular data collection, which will make the use of the 4Ws even more compelling for M&E purposes.

In the meantime, drawing on the use of baselines for M&E can guide the development of a general methodology for basemaps. This deliverable assumes that a standard M&E framework has already been developed for a given project. The challenge is to now translate this framework into a basemap. Recall that a strong M&E framework will include theories and types of change, an achievable goal with clear objectives, outputs and activities as well as reliable indicators and baselines. The first step, then, is to consider the theory (or theories) of change formulated for a given project.

Theories of change help planners and evaluators stay aware of the assumptions behind their choices, verify that the activities and objectives are logically aligned, and identify opportunities for integrated programming to spark synergies and leverage greater results. Types of change refer to specific changes expressed in the actual program design and/or evaluation, either as goals, objectives, or indicators.  Common examples include changes in behavior, practice, process, status, etc.  Both the theory of change and the types of changes sought should be evident in a well-designed program.

Although theories and types of change may not have obvious or intuitive geographical dimensions, it is important for “basemapping” to start thinking in geographical terms from the first step in the M&E process. Take “the reduction of violence theory of change,” which suggests that peace will result as we reduce the levels of violence perpetrated by combatants or their representatives. Methods include cease-fires, introduction of peacekeeping forces and conflict sensitive programming, for example. Cease-fires, peacekeeping operations and conflict sensitive development all have a geographical component. So the point is simply to ask the question “Where?” and to ensure the answer is woven into the theory of change.

Next, the basemapping process should consider the program design, e.g., the design hierarchy:

  • Goal: broadest change in the conflict.
  • Objectives: types of changes that are prerequisites to achieve stated goal.
  • Outputs: deliverables or products, often tangible from the activities.
  • Activities: concrete events or services performed.

Each element in the design hierarchy has a spatial component. The goal is to be achieved in a specific location or locations. So are the objectives, outputs and activities. Again, this geographical dimension needs to be explicitly articulated in each step of the hierarchy. The first phase of the basemapping process is about fully geographically mapping a project’s design hierarchy (social network mapping is also possible but not included in this deliverable).

In essence, the first phase of basemapping is to map the “ideal world” that a given project is mean to achieve. The second phase of basemapping comprises the mapping of a standard albeit georeferenced baseline of indicators. This phase seeks to capture an accurate picture of the current state of affairs in the “real world” and is in effect a conflict or risk assessment. Many conflict/risk assessment frameworks have already been developed and applied so this will not be duplicated here.

The third phase of basemapping comes after “Ideal World” and “Real World” mapping. The purpose of “Changed World” basemapping is to compare ideal and real world basemaps in order to isolate any positive changes that can be attributed to the project being implemented. The purpose of basemaps is not to prove causation but rather to suggest correlation.

Perhaps the most critical component of “Changed World” basemaps is the selection of the “change indicators”. That is, identifying those indicators that can geographically denote whether program activities are in fact change the real world as intended. Often, these indicators will already have been identified during standard baseline studies and simply need to be tied to specific geographic coordinates. In other situations, proxy indicators with a deliberate geographic dimension will need to be identified.

The temporal resolution of “change indicators” also needs to be a deliberate decision. For example, these geo-referenced indicators can be monitored on a daily, weekly or monthly basis. Mobile technology such as mobile phones and PDAs can be used to document change indicators and map them in quasi-real time. The Ushahidi mapping platform could be ideal for basemapping in this regards.

Basemapping and Decision-Support

Basemapping should not be considered distinct from decision-support processes and tools. This is particularly true of “Changed World Basemapping” which is meant to highlight in space and time the difference between the “Ideal World” and “Real World” basemaps. In other words, “Changed World Basemapping” should draw on geospatial analysis to create “heat maps” (and other relevant visualization techniques) to depict progress towards the ideal world in both time and space.

It is precisely because heat maps depict change that they should be considered as decision-support tools, particularly if these are viewed on a dynamic platform that permits the user to query and analyze the heat map. To this end, the purpose of basemapping is to combine M&E, conflict assessment and decision support by using one-and-only one integrated dynamic mapping tool.

Basemap Challenges

Like any new idea and methodology, there are important challenges that need to be addressed. For example, projects are likely to have impact at different spatial levels. How do we capture cross-scale effects? In addition, how do we introduce (spatial) control variables in order to isolate intervening (spatial) variables? Finally, can control groups be introduced in order to provide compelling evidence of impact or does this raise some important ethical issues as has happened in other fields?

Patrick Philippe Meier

Crisis Mapping for Monitoring & Evaluation

I was pleasantly surprised when local government ministry representatives in the Sudan (specifically Kassala) directly requested training on how to use the UNDP’s Threat and Risk Mapping Analysis (TRMA) platforms to monitor and evaluate their own programs.

Introduction

The use of crisis mapping for monitoring and evaluation (M&E) had cropped up earlier this year in separate conversations with the Open Society Institute (OSI) and MercyCorps. The specific platform in mind was Ushahidi, and the two organizations were interested in exploring the possibility of using the platform to monitor the impact of their funding and/or projects.

As far as I know, however, little to no rigorous research has been done on the use of crisis mapping for M&E. The field of M&E is far more focused on change over time than over space. Clearly, however, post-conflict recovery programs are implemented in both time and space. Furthermore, any conflict sensitivity programming must necessarily take into account spatial factors.

CartaMetrix

The only reference to mapping for M&E that I was able to find online was one paragraph in relation to the Cartametrix 4D map player. Here’s the paragraph (which I have split into to ease legibility) and below a short video demo I created:

“The Cartametrix 4D map player is visually compelling and fun to use, but in terms of tracking results of development and relief programs, it can be much more than a communications/PR tool. Through analyzing impact and results across time and space, the 4D map player also serves as a good program management tool. The map administrator has the opportunity to set quarterly, annual, and life of project indicator targets based on program components, regions, etc.

Tracking increases in results via the 4D map players, gives a program manager a sense of the pace at which targets are being reached (or not). Filtering by types of activities also provides for a quick and easy way to visualize which types of activities are most effectively resulting in achievements toward indicator targets. Of course, depending on the success of the program, an organization may or may not want to make the map (or at least all facets of the map) public. Cartametrix understands this and is able to create internal program management map applications alongside the publicly available map that doesn’t necessarily present all of the available data and analysis tools.”

Mapping Baselines

I expect that it will only be a matter of time until the M&E field recognizes the added value of mapping. Indeed, why not use mapping as a contributing tools in the M&E process, particularly within the context of formative evaluation?

Clearly, mapping can be one contributing tool in the M&E process. To be sure, baseline data can be collected, time-stamped and mapped. Mobile phones further facilitate this spatially decentralized process of information collection. Once baseline data is collected, the organization would map the expected outcomes of the projects they’re rolling out and estimated impact date against this baseline data.

The organization would then implement local development and/or conflict management programs  in certain geographical areas and continue to monitor local tensions by regularly collecting geo-referenced data on the indicators that said projects are set to influence. Again, these trends would be compared to the initial baseline.

These program could then be mapped and data on local tensions animated over time and space. The dynamic mapping would provide an intuitive and compelling way to demonstrate impact (or the lack thereof) in certain geographical areas where the projects were rolled out as compared to other similar areas with no parallel projects. Furthermore, using spatial analysis for M&E could also be a way to carry out a gap analysis and to assess whether resources are being allocated efficiently in more complex environments.

Next Steps

One of my tasks at TRMA is to develop a short document on using crisis mapping for M&E so if anyone has any leads on applied research in this area, I would be much obliged.

Patrick Philippe Meier