Tag Archives: validation

Three Common Misconceptions About Ushahidi

Posted on December 16, 2009 | 1 comment

Cross posted on Ushahidi

Here are three interesting misconceptions about Ushahidi and crowdsourcing in general:

  1. Ushahidi takes the lead in deploying the Ushahidi platform
  2. Crowdsourced information is statistically representative
  3. Crowdsourced information cannot be validated

Lets start with the first. We do not take the lead in deploying Ushahidi platforms. In fact, we often learn about new deployments second-hand via Twitter. We are a non-profit tech company and our goal is to continue developing innovative crowdsourcing platforms that cater to the growing needs of our current and prospective partners. We provide technical and strategic support when asked but otherwise you’ll find us in the backseat, which is honestly where we prefer to be. Our comparative advantage is not in deployment. So the credit for Ushahidi deployments really go the numerous organizations that continue to implement the platform in new and innovative ways.

On this note, keep in mind that the first downloadable Ushahidi platform was made available just this May, and the second version just last week. So implementing organizations have been remarkable test pilots, experimenting and learning on the fly without recourse to any particular manual or documented best practices. Most election-related deployments, for example, were even launched before May, when platform stability was still an issue and the code was still being written. So our hats go off to all the organizations that have piloted Ushahidi and continue to do so. They are the true pioneers in this space.

Also keep in mind that these organizations rarely had more than a month or two of lead-time before scheduled elections, like in India. If all of us have learned anything from watching these deployments in 2009, it is this: the challenge is not one of technology but election awareness and voter education. So we’re impressed that several organizations are already customizing the Ushahidi platform for elections that are more than 6-12 months away. These deployments will definitely be a first for Ushahidi and we look forward to learning all we can from implementing organizations.

The second misconception, “crowdsourced information is statistically representative,” often crops up in conversations around election monitoring. The problem is largely one of language. The field of election monitoring is hardly new. Established organizations have been involved in election monitoring for decades and have gained a wealth of knowledge and experience in this area. For these organizations, the term “election monitoring” has specific connotations, such as random sampling and statistical analysis, verification, validation and accredited election monitors.

When partners use Ushahidi for election monitoring, I think they mean something different. What they generally mean is citizen-powered election monitoring aided by crowdsourcing. Does this imply that crowdsourced information is statistically representative of all the events taking place across a given country? Of course not: I’ve never heard anyone suggest that crowdsourcing is equivalent to random sampling.

Citizen-powered election monitoring is about empowering citizens to take ownership over their elections and to have a voice. Indeed, elections do not start and stop at the polling booth. Should we prevent civil society groups from crowdsourcing crisis information on the basis that their reports may not be statistically representative? No. This is not our decision to make and the data is not even meant for us.

Another language-related problem has to due with the term “crowdsourcing”. The word  “crowd” here can literally mean anyone (unbounded crowdsourcing) or a specific group (bounded crowdsourcing) such as designated election monitors. If these official monitors use Ushahidi and they are deliberately positioned across a country for random sampling purposes, then this becomes no different at all to standard and established approaches to election monitoring. Bounded crowdsourcing can be statistically representative.

The third misconception about Ushahidi has to do with the tradeoff between unbounded crowdsourcing and the validation of said crowdsourced information. One of the main advantages of unbounded crowdsourcing is the ability to collect a lot of information from a variety of sources and media—official and nonofficial sources—in near real time. Of course, this means that a lot more of information can be reported at once, which can make the validation of said information a challenging process.

A common reaction to this challenge is to dismiss crowdsourcing altogether because unofficial sources may be unreliable or at worse deliberately misleading. Some organizations thus find it easier to write off all unofficial content because of these concerns. Ushahidi takes a different stance. We recognize that user-generated content is not about to disappear any time soon and that a lot of good can come out of such content, not least because official information can too easily become proprietary and guarded instead of shared.

So we’re not prepared to write off user-generated content because validating information happens to be challenging. Crowdsourcing crisis information is our business and so is (obviously) the validation of crowdsourced information. This is why Ushahidi is fully committed to developing Swift River. Swift is a free and open source platform that validates crowdsourced information in near real-time. Follow the Ushahidi blog for exciting updates!

1 Comment

Posted in Crowdsourcing

Tagged , ,

Twitter vs. Tyrants: Ushahidi and Data Verification

Posted on October 27, 2009 | 4 comments

My colleague Chris Doten asked me to suggest panelists for this congressional briefing on the role of new media in authoritarian states. I blogged about the opening remarks of each panelist here. But the key issues really came to fore during the Q/A session.

These issues addressed Ushahidi, data validation, security and education. This blog post addresses the issues raised around Ushahid and data validation. The text below includes my concerns with respect to a number of comments and assumptions made by some of the panelists.

Nathan Freitas (NYU):

  • It’s [Ushahidi] a crisis-mapping platform that  has grown out of the movement in Africa after the Kenyan elections.  It’s akin  to a blog system, but for mapping crisis, and what’s unique about it is it allows you to capture unverified and verified information.

Me: Many thanks to Nathan for referencing Ushahidi in the Congressional Briefing. Nathan’s comments are spot on. One of the unique features of Ushahidi is that the platform allows for the collection of both unverified and verified information.

But what’s the difference between these two types of information in the first place? In general, unverified information simply means information reported by “unknown sources” whereas verified tends to be associated with known sources of reporting, such as official election monitors.

The first and most important point to understand is that both approaches to information collection are compatible and complementary. Official election monitors, like professional journalists, cannot be everywhere at the same time. The “crowd” in crowdsourcing, on the other hand, has a comparative advantage in this respect (see supporting *empirical evidence here).

Clearly, the crowd has many more eyes and ears than any official monitoring network ever will. So discounting any and all information originating from the crowd is hard to justify. One would have to entirely dismiss the added value of all the Tweets, photos and YouTube footage generated by the “crowd” during the post-election violence in Iran.

  • And what’s interesting, I think we’ve seen the first round, the 1.0 of a lot of  this election monitoring.  As these systems come in place, they’ll be running  all the time, and they’ll be used in local elections and in state-level  elections, and the movement for – these tools will be easier, just like blogs.   Everyone blogs; in a few years, everyone’s got their own crisis-mapping  platform.

Me: What a great comment and indeed one of Ushahidi’s goals: for everyone to have their own crisis mapping platform in the near future. That’s what I call an iRevolution. Nathan’s point about the first round of these systems is also really important. The first version of the Ushahidi platform only became downloadable in May of this year; that’s just 5 months ago. We’re just getting started.

Daniel Calingaert (Freedom House):

  • [T]here’s a very critical component […] often  overlooked in these kinds of programs:  The information needs to be verified. It is useless or even counterproductive to simply be passing around rumors, and  rumor-mongering is very big in elections, and especially Election Day.

Me: Daniel certainly makes an important point although I personally don’t think that the need for verification is often overlooked in election monitoring. In any case, one should note that  rumors themselves need to be monitored and documented pre, during and post-elections. To be sure, if the information collection protocol is too narrow (say using only official monitors are allowed to submit evidence), then rumors (and other important information) may simply be dismissed and go unreported even though they could fuel conflict.

  • So it’s  important as part of the structure that you have qualified people to sort through the information and call what is credible reporting from citizens from very unsubstantiated information.

Me: Honestly, I’m always a little weary when I read comments along the lines of “you need to have qualified people” or “only experts should carry out the task.” Why? Because they tend to dismiss the added value that hundreds of bystanders can bring to the table. As Chris Spence noted about their operations in Moldova, NDI’s main partner organization “was harassed and kicked out of the country” while “the NDI program [was] largely shut down.” So who’s left to monitor? Exactly.

As my colleague Ory Okolloh recently noted, “Kenya had thousands election observers including many NDI monitors.” So what happened? “When it came to sharing their data as far as their observations at the polling everyone balked especially the EU and IRI because it was too “political”. IRI actually released their data almost 8 months later and yet they were supposed to be the filter.”

And so, Okolloh adds, “At a time when some corroboration could have prevented bloodshed, the ‘professionals’ were nowhere to be seen, so if we are talking about verification, legitimacy, and so on … lets start there.”

Chris Spence (NDI):

  • Monitoring groups – and this kind of gets to the threshold questions about Ushahidi and some of the platforms where you’re getting a lot of interesting  information from citizens, but at the end of the day, you’ve really got to  decide, have thresholds been reached which call into question the legitimacy of  the process?  And that’s really the political question that election observers and the groups that we work with have to grapple with.

Me: An interesting comment from NDI but one that perplexes me. I don’t recall users of Ushahidi suggesting that they should be the sole source of information to qualify for threshold points. Again, the most important point to understand is that different approaches to information collection can complement each other in important ways. We need to think less in linear terms and more in terms of information ecosystems with various ecologies of information sources.

  • And there’s so much involved in that methodology that one of the concerns about  the crisis mapping or the crowdsourcing [sic] is that the public can then draw interpretations about the outcome of elections without necessarily having the  filter required.  You know, you can look at a map of some city and see four or  five or 10 or several violations of election law reported by citizens who – you  know, you have to deal with the verification problem – but is that significant in the big picture?

Me: Ok, first of all, lets not confuse “crisis mapping” and “crowdsourcing” or use the terms interchangeably. Second, individuals for the large part are not thick. The maps can clearly state that the information represented is unfiltered and unverified, hence may be misleading. Third (apologies for repeating myself), none of the groups using Ushahidi claim that the data collected is representative of the bigger picture. This gets to the issue of significance.

And fourth, (more repeating), no one I know has suggested we go with once information feed, i.e., one source of information. I’m rather surprised that Chris Spence never brings up the importance of triangulation even though he acknowledges in his opening remarks that there are projects (like Swift River) that are specifically based on triangulation mechanisms to validate crowdsourced information.

Crowdsourced information can be an important repository for triangulation. The more crowdsourced information we have, the more self-triangulation is possible and the more this data can be used as a control mechanism for officially collected information.

Yes, there are issues around verification of data and an Ushahidi powered map may not be random enough for statistical accuracy but, as my colleague Ory Okolloh notes, “the data can point to areas/issues that need further investigation, especially in real-time.”

  • [I]t’s really important that, as these tools get better –  and we like the tools; Ushahidi and the other platforms are great – but we need  to make a distinction between what can be expected out of a professional  monitoring exercise and what can be drawn from unsolicited inputs from  citizens.  And I think there are good things that can be taken from both.

Me: Excellent, I couldn’t agree more. How about organizing a full-day workshop or barcamp on the role of new technologies in contemporary election monitoring? I think this would provide an ideal opportunity to hash out the important points raised by Nathan, Daniel and Chris.

Patrick Philippe Meier

4 Comments

Posted in Digital Activism

Tagged , , , ,

Accurate Crowdsourcing for Human Rights

Posted on September 12, 2009 | 10 comments

This is a short video of the presentation I will be giving at the Leir Conference on The Next Generation of Human Rights. My talk focuses on the use of digital technologies to leverage the crowdsourcing and crowdfeeding of human rights information. I draw on Ushahidi’s Swift River initiative to describe how crowdsourced information can be auto-validated.

Here’s a copy of the agenda (PDF) along with more details. This Leir Conference aims to bring together world-leading human rights practitioners, advocates, and funders for discussions in an intimate setting. Three panels will be convened, with a focus on audience discussion with the panelists. The topics will include:

  1. Trends in Combating Human Rights Abuses;
  2. Human Rights 2.0: The Next Generation of Human Rights Organizations;
  3. Challenges and Opportunities of  Technology for Human Rights.

I will be on presenting on the third panel together with colleagues from Witness.org and The Diarna Project. For more details on the larger subject of my presentation, please see this blog post on peer-producing human rights.

The desired results of this conference are to allow participants to improve advocacy, funding, or operations through collaborative efforts and shared ideas in a natural setting.

Patrick Philippe Meier

10 Comments

Posted in Crisis Mapping

Tagged , , , ,

Moving Forward with Swift River

Posted on May 7, 2009 | 13 comments

This is an update on the latest Swift River open group meeting that took place this morning at the InSTEDD office in Palo Alto. Ushahidi colleague Kaushal Jhalla first proposed the idea behind Swift River after the terrorist attacks on Mumbai last November. Ushahidi has since taken on the initiative as a core project since the goal of Swift River is central to the group’s mission: the crowdsourcing of crisis information.

Kaushal and Chris Blow gave the first formal presentation of Swift River during our first Ushahidi strategy meeting in Orlando last March where we formally established the Swift River group, which includes Andrew Turner, Sean Gourely, Erik Hersman and myself in addition to Kaushal and Chris. Andrew has played a pivotal role in getting Swift River and Vote Report India off the ground and I highly recommend reading his blog post on the initiative.

The group now includes several new friends of Ushahidi, a number of whom kindly shared their time and insights this morning after Chris kicked off the meeting to bring everyone up to speed.  The purpose of this blog post is to outline how I hope Swift River moves forward based on this morning’s fruitful session. Please see my previous blog post for an overview of the basic methodology.

The purpose of the Swift River platform, as I proposed this morning, is to provide two core services. The first, to borrow Guarva Mishra‘s description, is to crowdsource the tagging of crisis information. The second is to triangulate the tagged information to assign reality scores to individual events. Confused? Not to worry, it’s actually really straightforward.

Crowdsourcing Tagging

Information on a developing crisis can be captured from several text-based sources such articles from online news media, Tweets and SMS, for example. Of course, video footage, pictures and satellite imagery can also provide important information, but we’re more interested in text-based data for now.

The first point to note is that information can range from being very structured to highly unstructured. The word structure is simply another way of describing how organized information is. A few examples are in order vis-a-vis text-based information.

A book is generally highly structured information. Why? Well, because the author hopefully used page numbers, chapter headings, paragraphs, punctuation, an index and table of contents. The fact that the book is structured makes it easier for the reader to find the information she is looking for. The other end of the “structure spectrum” would be a run-on sentence with nospacesandpunctuation. Not terribly helpful.

Below is a slide from a seminar I taught on disaster and conflict early warning back in 2006; ignore the (c).

ewstructure

The slide above depicts the tradeoff between control and structure. We can impose structure on data collected if we control the data entry process. Surveys are an example of a high-control process that yields high-structure. We want high structure because this allows us to find and analyze the data more easily (c.f. entropy). This has generally been the preferred approach, particularly amongst academics.

If we give up control, as one does when crowdsourcing crisis information, we open ourselves up to the possibility of having to deal with a range of structured and unstructured information. To make sense of this information typically requires data mining and natural language processing (NLP) techniques that can identify structure in said information. For example, we would want to identify nouns, verbs, places and dates in order to extra event-data.

One way to do this would be to automatically tag an article with the parameters “who, what, where and when.” A number of platforms such as Open Calais and Virtual Research Associate’s FORECITE already do this. However, these platforms are not customized for crowdsourcing of crisis information and most are entirely closed. (Note: I did consulting work for VRA many years ago).

So we need to draw (and modify) relevant algorithms that are publically available and provide and a user-friendly interface for human oversight of the automated tagging (what we also referred to as crowdsourcing the filter). Here’s a proposed interface that Chris recently designed for Swift River.

swiftriver

The idea would be to develop an algorithm that parses the text (on the left) and auto-suggests answers for the tags (on the right). The user would then confirm or correct the suggested tags and the algorithm would learn from it’s mistakes. In other words, the algorithm would become more accurate over time and the need for human oversight would decrease. In short, we’d be developing a data-driven ontology backed up by Freebase to provide semantic linkages.

VRA already does this but, (1) the data validation is carried out by one (poor) individual, (2) the articles were restricted to the headlines from Reuters and Agence France Press (AFP) newswires, and (3) the project did not draw on semantic analysis. The validation component entailed making sure that events described in the headlines were correctly coded by the parser and ensuring there were no duplicates. See VRA’s patent for the full methodology (PDF).

Triangulation and Scoring

The above tagging process would yield a highly structured event dataset like the example depicted below.

dataset

We could then use simple machine analysis to cluster the same events together and thereby do away with any duplicate event-data. The four records above would then be collapsed into one record:

datafilter2

But that’s not all. We would use a simple weighting or scoring schema to assign a reality score to determine the probability that the event reported really happened. I already described this schema in my previous post so will just give one example: An event that is reported by more than one source is more likely to have happened. This increases the reality score of the event above and pushes it higher up the list. One could also score an event by the geographical proximity of the source to the reported event, and so on. These scores could be combined to give an overall score.

Compelling Visualization

The database output above is not exactly compelling to most people. This is where we need some creative visualization techniques to render the information more intuitive and interesting. Here are a few thoughts. We could draw on Gapminder to visualize the triangulated event-data over time. We could also use the idea of a volume equalizer display.

equalize

This is not the best equalizer interface around for sure, but hopefully gets the point across. Instead of decibels on the Y-axis, we’d have probability scores that an event really happened. Instead of frequencies on the X-axis, we’d have the individual events. Since the data coming in is not static, the bars would bounce up and down as more articles/tweets get tagged and dumped into the event database.

I think this would be an elegant way to visualize the data, not least because the animation would resemble the flow or waves of a swift river but the idea of using a volume equalizer could be used as analogy to quiet the unwanted noise. For the actual Swift River interface, I’d prefer using more colors to denote different characteristics about the event and would provide the user with the option of double-clicking on a bar to drill down to the event sources and underlying text.

Patrick Philippe Meier

13 Comments

Posted in Crisis Mapping

Tagged , , ,

Video Introduction to Crisis Mapping

Posted on April 27, 2009 | 29 comments

I’ve given many presentations on crisis mapping over the past two years but these were never filmed. So I decided to create this video presentation with narration in order to share my findings more widely and hopefully get a lot of feedback in the process. The presentation is not meant to be exhaustive although the video does run to about 30 minutes.

The topics covered in this presentation include:

  • Crisis Map Sourcing – information collection;
  • Mobile Crisis Mapping – mobile technology;
  • Crisis Mapping Visualization – data visualization;
  • Crisis Mapping Analysis – spatial analysis.

The presentation references several blog posts of mine in addition to several operational projects to illustrate the main concepts behind crisis mapping. The individual blog posts featured in the presentation are listed below:

This research is the product of a 2-year grant provided by Humanity United  (HU) to the Harvard Humanitarian Initiative’s (HHI) Program on Crisis Mapping and Early Warning, where I am a doctoral fellow.

I look forward to any questions/suggestions you may have on the video primer!

Patrick Philippe Meier

29 Comments

Posted in Crisis Mapping

Tagged , , , , , , , , , , , ,

Improving Quality of Data Collected by Mobile Phones

Posted on April 22, 2009 | 6 comments

The ICTD2009 conference in Doha, Qatar, had some excellent tech demo’s. I had the opportunity to interview Kuang Chen, a PhD student with UC Berkeley’s computer science department about his work on improving data quality using dynamic forms and machine learning.

I’m particularly interested in this area of research since ensuring data quality continues to be a real challenge in the fields of conflict early warning and crisis mapping. So I always look for alternative and creative approaches that address this challenge. I include below the abstract for Kuang’s project (which includes 5 other team members) and a short 2-minute interview.

Abstract

“Organizations in developing regions want to efficiently collect digital data, but standard data gathering practices from the developed world are often inappropriate. Traditional techniques for form design and data quality are expensive and labour-intensive. We propose a new data-driven approach to form design, execution (filling) and quality assurance. We demonstrate USHER, an end-to-end system that automatically generates data entry forms that enforce and maintain data quality constraints during execution. The system features a probabilistic engine that drives form-user interactions to encourage correct answers.”

In my previous post on data quality evaluation, I pointed to a study that suggests mobile-based data entry has significantly higher error rates. The study shows that a voice call to a human operator results in superior data quality—no doubt due to the human operator double-checking the respondent’s input verbally.  USHER’s ability to dynamically adjust the user interface (form layout and data entry widgets) is one approach to provide some context-specific data-driven user feedback that is currently lacking in mobile forms, as an automated proxy of a human data entry person on the other end of the line.

Interview

This is my first video so many thanks to Erik Hersman for his tips on video editing! And many thanks to Kuang for the interview.

Patrick Philippe Meier

6 Comments

Posted in Humanitarian Technologies

Tagged , , ,

Evaluating Accuracy of Data Collection on Mobile Phones

Posted on April 18, 2009 | 4 comments

The importance of data validation is unquestioned but few empirical studies seek to assess the possible errors incurred during mobile data collection. Authors Somani Patnaik, Emma Brunskill and William Thies thus carried out what is possibly the first quantitative evaluation  (PDF) of data entry accuracy on mobile phones in resource-constrained environments. They just presented their findings at ICTD 2009.

Mobile devices have become an increasingly important tool for information collection. Hence, for example, my interest in pushing forward the idea of Mobile Crisis Mapping (MCM). While studies on data accuracy exist for personal digital assistants (PDAs), there are very few that focus on mobile phones. This new study thus evaluates three user interfaces for information collection: 1) Electronic forms; 2) SMS and 3) voice.

The results of the study indicate the following associated error rates:

  • Electronic forms = 4.2%
  • SMS = 4.5%
  • Voice = 0.45%

For compartive purposes and context, note that error rates using PDAs have generally been less than 2%. These figures represent the fraction of questions that were answered incorrectly. However, since “each patient interaction consisted of eleven questions, the probability of error somewhere in a patient report is much higher. For both electronic forms and SMS, 10 out of 26 reports (38%) contained an error; for voice, only 1 out of 20 reports (5%) contained an error (which was due to operator transcription).

I do hope that the results of this study prompt many others to carry out similar investigations.  I think we need a lot more studies like this one but with a larger survey sample (N) and across multiple sectors (this study drew on just 13 healthworkers).

The UN Threat and Risk Mapping Analysis (TRMA) project I’m working on in the Sudan right now will be doing a study on data collection accuracy using mobile phones when they roll out their program later this month. The idea is to introduce mobile phones in a number of localities and not in neighboring ones. The team will then compare the data quality of both samples.

I look forward to sharing the results.

Patrick Philippe Meier

4 Comments

Posted in Humanitarian Technologies

Tagged , , ,