Crowdsourcing in Crisis: A More Critical Reflection

This is a response to Paul’s excellent comments on my recent posts entitled “Internews, Ushahidi and Communication in Crisis” and “Ushahidi: From Croudsourcing to Crowdfeeding.”

Like Paul, I too find Internews to be a top organization. In fact, of all the participants in New York, the Internews team in was actually the most supportive of exploring the crowdsourcing approach further instead of dismissing it entirely. And like Paul, I’m not supportive of the status quo in the humanitarian community either.

Paul’s observations are practical and to the point, which is always appreciated. They encourage me revisit and test my own assumptions, which I find stimulating. In short, Paul’s comments are conducive to a more critical reflection of crowdsourcing in crisis.

In what follows, I address all his arguments point by point.

Time Still Ignored

Paul firstly notes that,

Both accuracy and timeliness are core Principles of Humanitarian Information management established at the 2002 Symposium on Best Practices in Humanitarian Information Exchange and reiterated at the 2007 Global Symposium +5. Have those principles been incorporated into the institutions sufficiently? Short answer, no. Is accuracy privileged at the expense of timeliness? Not in the field.

The importance of “time” and “timeliness” was ignored during both New York meetings. Most field-based humanitarian organizations dismissed the use of  “crowdsourcing” because of their conviction that “crowdsourced information cannot be verified.” In short, participants did not privilege timeliness at the expense accuracy because they consider verification virtually impossible.

Crowdsourcing is New

Because crowdsourcing is unfamiliar, it’s untested in the field and it makes fairly large claims that are not well backed by substantial evidence. Having said that, I’m willing to be corrected on this criticism, but I think it’s fair to say that the humanitarian community is legitimately cautious in introducing new concepts when lives are at stake.

Humanitarian organizations make claims about crowdsourcing that are not necessarily backed by substantial evidence because crowdsourcing is fairly new and untested in the field. If we use Ushahidi as the benchmark, then crowdsourcing crisis informaiton is 15 months old and the focus of the conversation should be on the two Ushahid deployments (Kenya & DRC) during that time.

The angst is understandable and we should be legitimately cautious. But angst shouldn’t mean we stand back and accept the status quo, a point that both Paul and I agree on.

Conflict Inflamation

Why don’t those who take the strongest stand against crowdsourcing demonstrate that Ushahidi-Kenya and Ushahidi-DRC have led to conflict inflammation? As far we know, none of the 500+ crowdsourced crisis events in those countries were manufactured to increase violence. If that is indeed the case, then skeptics like Paul should explain why we did not see Ushahidi be used to propagate violence.

In any event, if we embrace the concept of human development, then the decision vis-à-vis whether or not to crowdsource and crowdfeed information ultimately lies with the crowd sourcers and feeders. If the majority of users feel compelled to generate and share crisis information when a platform exists, then it is because they find value in doing so. Who are we to say they are not entitled to receive public crisis information?

Incidentally, it is striking to note the parallels between this conversation and skeptics during the early days of Wikipedia.

Double Standards

I would also note that I don’t think the community is necessarily holding crowdsourcing to a higher standard, but exactly the same standard as our usual information systems – and if they haven’t managed to get those systems right yet, I can understand still further why they’re cautious about entertaining an entirely new and untested approach.

Cautious and dismissive are two different things. If the community were holding crowdsourcing to an equal standard, then they would consider both the timeliness and accuracy of crowdsourced information. Instead, they dismiss crowdsourcing without recognizing the tradeoff with timeliness.

What is Crisis Info?

In relation to my graphic on the perishable nature of time, Paul asks

What “crisis information” are we talking about here? I would argue that ensuring your data is valid is important at all times, so is this an attack on dissemination strategies rather than data validation?

We’re talking about quasi-real time and geo-tagged incident reporting, i.e., reporting using the parameters of incident type, location and time. Of course it is important that data be as accurate as possible. But as I have already argued, accurate information received late is of little operational value.

On the other hand information that has not been yet validated but received early gives those who may need the information the most (1) more time to take precautionary measures, and (2) more time to determine its validity.

Unpleasant Surprises

On this note, I just participated in the Harvard Humanitarian Initiative (HHI)’s Humanitarian Action Summit  (HAS) where the challenge of data validation came up within the context of public health and emergency medicine. The person giving the presentation had this to say:

We prefer wrong information to no information at all since at least we can at least take action in the case of the former to determine the validity of the information.

This reminds me of the known unknowns versus unknown unknowns argument. I’d rather know about a piece of information even though I’m unable to validate it rather than not know and be surprised later in case it turns out to be true.

We should take care not to fall into the classic trap exploited by climate change skeptics. Example: We can’t prove that climate change is really happening since it could simply be that we don’t have enough accurate data to arrive at the correct conclusion. So we need more time and data for the purposes of validation. Meanwhile, skeptics argue, there’s no need to waste resources by taking precautionary measures.

Privileging Time

It also strikes me as odd that Patrick argues that affected communities deserve timely information but not necessarily accurate information. As he notes, it may be a trade-off – but he provides no argument for why he privileges timeliness over accuracy.

I’m not privileging one over the other. I’m simply noting that humanitarian organizations in New York completely ignored the importance of timeliness when communicating with crisis-affected communities, which I still find stunning. It is misleading to talk about accuracy without talking about timeliness and vice versa. So I’m just asking that we take both variables into account.

Obviously the ideal would be to have timely and accurate information. But we’re not dealing with ideal situations when we discuss sudden onset emergencies. Clearly the “right” balance between accuracy and timeliness depends who the end users are and what context they find themselves in. Ultimately, the end users, not us, should have the right to make that final decision for themselves. While accuracy can saves lives, so can timeliness.

Why Obligations?

Does this mean that the government and national media have an obligation to report on absolutely every single violation of human rights taking place in their country? Does this mean that the government and national media have an obligation to report on absolutely every single violation of human rights taking place in their country?

I don’t understand how this question follows from any of my preceding comments. We need to think about information as an ecosystem with multiple potential sources that may or may not overlap. Obviously governments and national media may not be able to—or compelled to—report accurately and in a timely manner during times of crises. I’m not making an argument about obligation. I’m just making an observation about there being a gap that crowdsourcing can fill, which I showed empirically in this Kenya case study.

Transparency and Cooperation

I’m not sure it’s a constructive approach to accuse NGOs of actively “working against transparency” – it strikes me that there may be some shades of grey in their attitudes towards releasing information about human rights abuses.

You are less pessimistic than I am—didn’t think that was possible. My experience in Africa has been that NGOs (and UN agencies) are reluctant to share information not because of ethical concerns but because of selfish and egotistical reasons. I’d recommend talking with the Ushahidi team who desperately tried to encourage NGOs to share information with each other during the post-election violence.

Ushahidi is Innovation

On my question about why human rights and humanitarian organizations were not the one to set up a platform like Ushahidi, Paul answers as follows.

I think it might be because the human rights and humanitarian communities were working on their existing projects. The argument that these organisations failed to fulfill an objective when they never actually had that objective in the first place is distinctly shakey – it seems to translate into a protest that they weren’t doing what you wanted them to do.

I think Paul misses the point. I’m surprised he didn’t raise the whole issue of innovation (or rather lack thereof) in the humanitarian community since he has written extensively about this topic.

Perhaps we also have to start thinking in terms of what damage might this information do (whether true or false) if we release it.

I agree. At the same time, I’d like to get the “we” out of the picture and let the “them” (the crowd) do the deciding. This is the rationale behind the Swift River project we’re working on at Ushahidi.

Tech-Savvy Militias

Evidence suggests that armed groups are perfectly happy to use whatever means they can acquire to achieve their goals. I fail to see why Ushahidi would be “tactically inefficient, and would require more co-ordinating” – all they need to do is send a few text messages. The entire point of the platform is that it’s easy to use, isn’t it?

First of all, the technological capacity and sophistication of non-state armed groups varies considerably from conflict to conflict. While I’m no expert, I don’t know of any evidence from Kenya or the DRC—since those are our empirical test cases—that suggest tech-savvy militia members regularly browse the web to identify new Web 2.0 crowdsourcing tools they can use to create more violence.

Al Qaeda is a different story, but we’re not talking about Al Qaeda, we’re talking about Kenya and the DRC. In the case of the former, word about Ushahidi spread through the Kenyan blogosphere. Again, I don’t know of any Kenyan militia groups in the Rift Valley, for example, that monitors the Kenyan blogosphere to exploit violence.

Second of all, one needs time to learn how to use a platform like Ushahidi for conflict inflammation. Yes, the entire point of the platform is that it’s easy to use to report human rights violations. But it obviously takes more thinking to determine what, where and when to text an event in order to cause a particular outcome. It requires a degree of coordination and decision-making.

That’s why it would be inefficient. All a milita would need to do is fire a few bullets from one end of a village to have the locals run the other way straight into an ambush. Furthermore, we found no evidence of hate SMS submitted to Ushahidi even though there were some communicated outside of Ushahidi.

Sudan Challenges

The government of Sudan regularly accuses NGOs (well, those NGOs it hasn’t expelled) of misreporting human rights violations. What better tool would the government have for discrediting human rights monitoring than Ushahidi? All it would take would be a few texts a day with false but credible reports, and the government can dismiss the entire system, either by keeping their own involvement covert and claiming that the system is actually being abused, or by revealing their involvement and claiming that the system can be so easily gamed that it isn’t credible.

Good example given that I’m currently in the Sudan. But Paul is mixing human rights reporting for the purposes of advocacy with crisis reporting for the purposes of local operational response.

Of course government officials like those in Khartoum will do, and indeed continue to do, whatever the please. But isn’t this precisely why one might as well make the data open and public so those facing human rights violations can at least have the opportunity to get out of harms way?

Contrast this with the typical way that human rights and humanitarian organizations operate—they typically keep the data for themselves, do not share it with other organizations let alone with beneficiaries. How is data triangulation possible at all given such a scenario even if we had all the time in the world? And who loses out as usual? Those local communities who need the information.

Triangulation

While Paul fully agrees that local communities are rarely dependent on a single source of information, which means they can triangulate and validate, he maintains that this “is not an argument for crowdsourcing.” Of course it is, more information allows more triangulation and hence validation. Would Paul argue that my point is an argument against crowdsourcing?

We don’t need less information, we need more information and the time element matters precisely because we want to speed up the collection of information in order to triangulate as quickly as possible.

Ultimately, it will be a question of probability whether or not a given event is true, the larger your sample size, the more confident you can be. The quicker you collect that sample size, the quicker you can validate. Crowdsourcing is a method that facilitates the rapid collection of large quantities of information which in turn facilitates triangulation.

Laughing Off Disclaimers

The idea that people pay attention to disclaimers makes me laugh out loud. I don’t think anybody’s accusing affected individuals of being dumb, but I’d be interested to see evidence that supports this claim. When does the validation take place, incidentally? And what recourse do individuals or communities have if an alert turns out to be false?

Humanitarians often treat beneficiaries as dumb, not necessarily intentionally, but I’ve seen this first hand in East and West Africa. Again, if you haven’t read “Aiding Violence” then I’d recommend it.

Second, the typical scenario that comes up when talking about crowdsourcing and the spreading of rumors has to do with refugee camp settings. The DRC militia story is one that I came up with (and have already used in past blog posts) in order emphasize the distinction with refugee settings.

The scenario that was brought up by others at the Internews meeting was actually one set in a refugee camp. This scenario is a classic case of individuals being highly restricted in the variety of different information sources they have access to, which makes the spread of rumors difficult to counter or dismiss.

Crowdsourcing Response

When I asked why field-based humanitarian organizations that directly work with beneficiaries in conflict zones don’t take an interest in crowdsourced information and the validation thereof, Paul responds as follows.

Yes, because they don’t have enough to do. They’d like to spend their time running around validating other people’s reports, endangering their lives and alienating the government under which they’re working.

I think Paul may be missing the point—and indeed power—of crowdsourcing. We need to start thinking less in traditional top-down centralized ways. The fact is humanitarian organizations could subscribe to specific alerts of concern to them in a specific and limited geographical area.

If they’re onsite where the action is reportedly unfolding and they don’t see any evidence of rumors being true, surely spending 15 seconds to text this info back to HQ (or to send a picture by camera phone) is not a huge burden. This doesn’t endanger their lives since they’re already there and quelling a rumor is likely to calm things down. If we use secure systems, the government wouldn’t be able to attribute the source.

The entire point behind the Swift River project is to crowdsource the filtering process, ie, to distribute and decentralizes the burden of data validation. Those organizations that happen to be there at the right time and place do the filtering, otherwise they don’t and get on with their work. This is the whole point behind my post last year on crowdsourcing response.

Yes, We Can

Is there any evidence at all that the US Embassy’s Twitter feed had any impact at all on the course of events? I mean, I know it made a good headline in external media, but I don’t see how it’s a good example if there’s no actual evidence that it had any impact.

Yes, the rumors didn’t spread. But we’re fencing with one anecdote after the other. All I’m arguing is that two-way communication and broadcasting should be used to counter misinformation;  meaning that it is irresponsible for humanitarian organizations to revert to one-way communication mindsets and wash their hands clean of an unfolding situation without trying to use information and communication technology to do something about it.

Many still don’t understand that the power of P2P meshed communication can go both ways. Unfortunately, as soon as we see new communication technology used for ill, we often react even more negatively by pulling the plug on any communication, which is what the Kenyan government wanted to do during the election violence.

Officials requested that the CEO of Safaricom switch off the SMS network to prevent the spread of hate SMS, he chose to broadcast text messages calling for peace, restraint and warning that those found to be creating hate SMS would be tracked and prosecuted (which the Kenyan Parliament subsequently did).

Again, the whole point is that new communication technologies present a real potential for countering rumors and unless we try using them to maximize positive communication we will never get sufficient evidence to determine whether using SMS and Twitter to counter rumors can work effectively.

Ushahidi Models

In terms of Ushahidi’s new deployment model being localized with the crowdsourcing limited to members of a given organization, Paul has a point when he suggests this “doesn’t sound like crowdsourcing.” Indeed, the Gaza deployment of Ushahidi is more an example of “bounded crowdsourcing” or “Al Jazeera sourcing” since the crowd is not the entire global population but strictly Al Jazeera journalists.

Perhaps crowdsourcing is not applicable within those contexts since “bounded crowdsourcing” may in effect be an oxymoron. At the same time, however, his conclusion that Ushahidi is more like classic situation reporting is not entirely accurate either.

First of all, the Ushahidi platform provides a way to map incident reports, not situation reports. In other words, Ushahidi focuses on the minimum essential indicators for reporting an event. Second, Ushahidi also focuses on the minimum essential technology to communicate and visualize those events. Third, unlike traditional approaches, the information collected is openly shared.

I’m not sure if this is an issue of language and terminology or if there is a deeper point here. In other words, are we seeing Ushahidi evolve in such a way that new iterations of the platform are becoming increasingly similar to traditional information collection systems?

I don’t think so. The Gaza platform is only one genre of local deployment. Another organization might seek to deploy a customized version of Ushahidi and not impose any restrictions on who can report. This would resemble the Kenya and DRC deployments of Ushahidi. At the moment, I don’t find this problematic because we haven’t found signs that this has led to conflict inflammation. I have given a number of reasons in this blog post why that might be.

In any case, it is still our responsibility to think through some scenarios and to start offering potential solutions. Hence the Swift River project and hence my appreciating Paul’s feedback on my two blog posts.

Patrick Philippe Meier