From Baselines to Basemaps: Crisis Mapping for Monitoring & Evaluation (M&E)

I was just in Berlin for meetings with Transparency International and the topic of mapping for Monitoring & Evaluation came up yet again. Earlier this year, Mercy Corps and UNDP Sudan both expressed an interest in exploring the application of crisis mapping platforms for M&E. Problem is, the field of M&E—particularly with regards to peacebuilding and post-conflict reconstruction—is devoid of any references to mapping.

As part of my consulting work with UNDP Sudan, I therefore produced a short concept paper this topic back in June. Here’s a summary of what I wrote.

Introduction

Peacebuilding and post-conflict reconstruction programs necessarily operate within a dynamic environment. This means that they must adapt to changing circumstances or else run the risk of misallocating resources or worse, exacerbating tensions not to mention creating new sources of violent conflict. Hence the need for conflict sensitive programming, which UNDP’s TRMA initiative is designed to support in the Sudan.

The threat and risk maps produced by UNDP provide spatial risk assessments that can inform programmatic response in Sudan’s post-conflict states. This need not be a one-off decision-support exercise, however. Indeed, the use of spatial risk assessments updated over time is an even more compelling use of crisis maps for decision-support.

A changing post-conflict environment means that projects designed half-a-year ago may no longer be having the intended impact they were funded to have. To this end, it is important that UN and local-government partners have regular updates on the changing context in order to adapt programming respectively. Crisis mapping can play a pivotal role in this decision support process.

Basemapping

I therefore propose a new approach to crisis mapping called “basemapping”. The purpose of basemapping is to combine M&E and crisis mapping to produce basemaps against which projects can be monitored and evaluated. The basemapping process constitutes three distinct mapping steps:

1. Ideal World Basemapping: mapping the ideal world that a given project seeks to achieve over a given period of time.

2. Real World Basemapping: mapping the current state of affairs in the specific world that the project seeks to change

3. Changed World Basemapping: ongoing mapping to compare the change between the ideal world and real world basemaps.

The following section provides a brief background to M&E and formulates a proposed methodology for basemapping. As basemapping is a new concept that has little to no precedent, the purpose of the proposed methodology is to catalyze discussion on the subject and to make the methodology more sophisticated over time.

Monitoring and Evaluation (M&E)

A strong M&E framework will include theories and types of change, an achievable goal with clear objectives, outputs and activities as well as reliable indicators and baselines. However, M&E frameworks should be case-specific and must be tailored to the purpose of individual projects. This deliverable assumes that UNDP is already well versed in M&E frameworks and methodologies. The analysis that follows thus focuses specifically on the contributing role of crisis mapping to the M&E process.

Baselines are the most often forgotten component within design, monitoring and evaluation, yet they are key to proving that change has truly taken place. They also provide the most intuitive link to crisis mapping. While baselines typically represent a snapshot in time and space against which deviations represent progress or failure, the concept of “basemaps” can play the same role albeit dynamically.

Perhaps the closest analogy to baselines in the field of conflict analysis is the conflict assessment. A conflict assessment is an exploration of the realities of the conflict and an analysis of its underlying causes. An assessment can be done at any time, independently of a program or as a part of an existing program. Assessments are often conducted to determine whether an intervention is needed and, if so, what type of intervention.

Assessments (or risk assessments) are also typically carried out as a first step in the development of a conflict early warning system. They serve to identify appropriate conflict early warning indicators. In a sense, an assessment is the basis from which the programming will be designed. Conversely, a baseline identifies the status of the targeted change before the project starts but after it has been designed.

M&E experts caution that assessments and baselines should not be blended together. Nor do they suggest using one as a substitute for the other since their raison d’être, focus, and implementation are very different. We beg to differ and would even go so far as proposing that dynamic crisis mapping platforms can bring both conflict assessments and M&E baselines together with considerable added value.

To explain the potential of basemaps in more detail, a sound understanding of baselines is important. A baseline provides a starting point or reference from which a comparison can be made.  Baselines are conducted prior to the beginning of a program intervention and are the point of comparison for monitoring and evaluation data. The bulk of baseline studies focus on the intended outcomes of a project. They can also take into account secondary outcomes and assumptions, though these are not the primary emphasis.

Baseline information can be used in a number of ways. Perhaps the most intuitive application is the comparison of baseline information with subsequent information to show the change (or lack thereof) that has taken place over time (again, change over space is all-too often ignored). Baseline information can also be used to refine programming decisions or set achievable and realistic targets. Finally, baseline information enables monitoring data to have greater utility earlier in the project cycle.

Baselines have three possible focus areas: change, secondary outcomes and assumptions: change, secondary changes and assumptions. It is important that DAI and partners agree on the formulation of an M&E framework that clearly focuses on one of these focus areas. Note that the first area, change, is required of all baselines, while the other two are optional depending on the project.

Towards Dynamic Basemaps

Conflict, or threat and risk data, typically has a geographic dimension. Consequently, baseline data on conflict dynamics also have a geographic element. To this end, it makes far more sense to use basemaps than baselines since the former includes spatially relevant information that changes over time, which is necessarily important for decision-making vis-à-vis programmatic response in post-conflict environments.

UNDP’s TRMA enjoys a distinctive advantage in this respect since the project already maps threat and risk data (via hard-copy print-outs and a dynamic mapping tools called the 4W’s). The main handicap at the moment is the lack of regularly updated threat and risk data in order to visualize change over time. That said, the TRMA team is planning to shift towards more regular data collection, which will make the use of the 4Ws even more compelling for M&E purposes.

In the meantime, drawing on the use of baselines for M&E can guide the development of a general methodology for basemaps. This deliverable assumes that a standard M&E framework has already been developed for a given project. The challenge is to now translate this framework into a basemap. Recall that a strong M&E framework will include theories and types of change, an achievable goal with clear objectives, outputs and activities as well as reliable indicators and baselines. The first step, then, is to consider the theory (or theories) of change formulated for a given project.

Theories of change help planners and evaluators stay aware of the assumptions behind their choices, verify that the activities and objectives are logically aligned, and identify opportunities for integrated programming to spark synergies and leverage greater results. Types of change refer to specific changes expressed in the actual program design and/or evaluation, either as goals, objectives, or indicators.  Common examples include changes in behavior, practice, process, status, etc.  Both the theory of change and the types of changes sought should be evident in a well-designed program.

Although theories and types of change may not have obvious or intuitive geographical dimensions, it is important for “basemapping” to start thinking in geographical terms from the first step in the M&E process. Take “the reduction of violence theory of change,” which suggests that peace will result as we reduce the levels of violence perpetrated by combatants or their representatives. Methods include cease-fires, introduction of peacekeeping forces and conflict sensitive programming, for example. Cease-fires, peacekeeping operations and conflict sensitive development all have a geographical component. So the point is simply to ask the question “Where?” and to ensure the answer is woven into the theory of change.

Next, the basemapping process should consider the program design, e.g., the design hierarchy:

• Goal: broadest change in the conflict.
• Objectives: types of changes that are prerequisites to achieve stated goal.
• Outputs: deliverables or products, often tangible from the activities.
• Activities: concrete events or services performed.

Each element in the design hierarchy has a spatial component. The goal is to be achieved in a specific location or locations. So are the objectives, outputs and activities. Again, this geographical dimension needs to be explicitly articulated in each step of the hierarchy. The first phase of the basemapping process is about fully geographically mapping a project’s design hierarchy (social network mapping is also possible but not included in this deliverable).

In essence, the first phase of basemapping is to map the “ideal world” that a given project is mean to achieve. The second phase of basemapping comprises the mapping of a standard albeit georeferenced baseline of indicators. This phase seeks to capture an accurate picture of the current state of affairs in the “real world” and is in effect a conflict or risk assessment. Many conflict/risk assessment frameworks have already been developed and applied so this will not be duplicated here.

The third phase of basemapping comes after “Ideal World” and “Real World” mapping. The purpose of “Changed World” basemapping is to compare ideal and real world basemaps in order to isolate any positive changes that can be attributed to the project being implemented. The purpose of basemaps is not to prove causation but rather to suggest correlation.

Perhaps the most critical component of “Changed World” basemaps is the selection of the “change indicators”. That is, identifying those indicators that can geographically denote whether program activities are in fact change the real world as intended. Often, these indicators will already have been identified during standard baseline studies and simply need to be tied to specific geographic coordinates. In other situations, proxy indicators with a deliberate geographic dimension will need to be identified.

The temporal resolution of “change indicators” also needs to be a deliberate decision. For example, these geo-referenced indicators can be monitored on a daily, weekly or monthly basis. Mobile technology such as mobile phones and PDAs can be used to document change indicators and map them in quasi-real time. The Ushahidi mapping platform could be ideal for basemapping in this regards.

Basemapping and Decision-Support

Basemapping should not be considered distinct from decision-support processes and tools. This is particularly true of “Changed World Basemapping” which is meant to highlight in space and time the difference between the “Ideal World” and “Real World” basemaps. In other words, “Changed World Basemapping” should draw on geospatial analysis to create “heat maps” (and other relevant visualization techniques) to depict progress towards the ideal world in both time and space.

It is precisely because heat maps depict change that they should be considered as decision-support tools, particularly if these are viewed on a dynamic platform that permits the user to query and analyze the heat map. To this end, the purpose of basemapping is to combine M&E, conflict assessment and decision support by using one-and-only one integrated dynamic mapping tool.

Basemap Challenges

Like any new idea and methodology, there are important challenges that need to be addressed. For example, projects are likely to have impact at different spatial levels. How do we capture cross-scale effects? In addition, how do we introduce (spatial) control variables in order to isolate intervening (spatial) variables? Finally, can control groups be introduced in order to provide compelling evidence of impact or does this raise some important ethical issues as has happened in other fields?

Patrick Philippe Meier