Category Archives: Social Media

On Crowdsourcing, Crisis Mapping and Data Protection Standards

Posted on February 5, 2012 | 12 comments

The International Organization for Migration (IOM) just published their official Data Protection Manual. This report is hugely informative and should be required reading. At the same time, the 150-page report does not mention social media even once. This is perfectly understandable given IOM’s work, but there is no denying that disaster-affected communities are becoming more digitally-enabled—and thus increasingly the source of important, user-generated information. Moreover, it is difficult to ascertain exactly how to apply all of IOM’s Data Protection Principles to this new digital context and the work of the Standby Volunteer Task Force (SBTF).

The IOM Manual recommends that a risk-benefit assessment be conducted prior to data collection. This means weighing the probability of harm against the anticipated benefits and ensuring that the latter significantly outweigh the potential risks. But IOM explains that “the risk–benefit assessment is not a technical evaluation that is valid under all circumstances. Rather, it is a value judgement that often depends on various factors, including, inter alia, the prevailing social, cultural and religious attitudes of the target population group or individual data subject.”

The Manual also states that data collectors should always put themselves in the shoes of the data subject and consider: “How would a reasonable person, in the position of data subject, react to the data collection and data processing practices?” Again, this a value judgment rather than a technical evaluation. Applying this consistently across IOM will no doubt be a challenge.

The IOM Principles, which form the core of the manual, are as follows (keep in mind that they are obviously written with IOM’s mandate explicitly in mind):

1. Lawful & Fair Collection
2. Specified and Legitimate Purpose
3. Data quality
4. Consent
5. Transfer to Third Parties
6. Confidentiality
7. Access and Transparency
8. Data Security
9. Retention of Personal Data
10. Application of the Principles
11. Ownership of Personal Data
12. Oversight, Compliance & Internal Remedies
13. Exceptions

Take the first principle, which states that “Personal data must be obtained by lawful and fair means with the knowledge or consent of the data subject.” What does this mean when the data is self-generated and voluntarily placed in the public domain? This question also applies to a number of other principles including “Consent” and “Confidentiality”. In the section on “Consent”, the manual lists various ways that consent can be acquired. Perhaps the most a propos to our discussion is “Implicit Consent: no oral declaration or written statement is obtained, but the action or inaction of the data subjects un-equivocally indicates voluntary participation in the IOM project.”

Indeed, during the Ushahidi-Haiti Crisis Mapping Project (UHP), a renowned professor and lawyer at The Fletcher School of Law and Diplomacy was consulted to determine whether or not text messages from the disaster-affected community could be added to a public map). This professor stated there was “Implicit Consent” to map these text messages. (Incidentally, experts at Harvard’s Berkman Center were also consulted on this question at the time).

The first IOM principle further stipulates that “communication with data subjects should be encouraged at all stages of the data collection process.” But what if this communication poses a danger to the data subject? The manual further states that “Personal data should be collected in a safe and secure environment and data controllers should take all necessary steps to ensure that individual vulnerabilities and potential risks are not enhanced.” What if data subjects are not in a safe and secure environment but nevertheless voluntarily share potentially important information on social media channels?

Perhaps the only guidance provided by IOM on this question is as follows: “Data controllers should choose the most appropriate method of data collection that will enhance efficiency and protect the confidentiality of the personal data collected.” But again, what if the data subject has already volunteer information with their personal data and placed this information in the public domain?

The third principle, “Data Quality” is obviously key but the steps provided to ensure accuracy are difficult to translate within the context of crowdsourced information from the social media space. The same is true of several IOM Data Protection Principles. But some are certainly applicable with modification. Take the seventh principle on “Access and Transparency” which recommends that complaint procedures should be relatively straightforward so that data subjects can easily request to rectify or delete content previously collected from them.

“Data Security”, the eighth principle, is also directly applicable. For example, data from social media could be classified according the appropriate level of sensitivity and treated accordingly. During the response to the Haiti earthquake, for example, we kept new information on the location of orphans confidential, sharing this only with trusted colleagues in the humanitarian community. “Separating personal data from non-personal data” is another procedure that can (and has) been used in crisis mapping projects. This is for me an absolutely crucial point. Depending on the situation, we need to separate information mana-gement systems that contain data with personal identifiers from crisis mapping platforms. Obviously, the former thus need to be more secure. Encryption is also proposed for data security and applicable to crisis mapping.

The tenth IOM principle, i.e., “The Application of the Principles”, provides additional guidance on how to implement data protection and security. For example, the manual describes three appropriate methods for depersonalizing data: data-coding;  pseudonymization; and anonymization. Each of these could be applied to crisis mapping projects.

To conclude, the IOM Data Protection Manual is an important contribution and some of the principles described therein can be applied to crowdsourcing and crisis mapping. I look forward to folding these into the workflows and standard operating procedures of the SBTF (with guidance from the SBTF’s Advisory Board and other experts). There still remains a gap, however, vis-a-vis those IOM principles that are not easily customizable for the context in which the SBTF operates. There is also an issue vis-a-vis the Terms of Service of many social media platforms with respect to privacy and data protection standards.

This explains why I am actively collaborating with a major humanitarian organi-zation to explore the development of appropriate data protection standards for crowdsourcing crisis information in the context of social media. Many humanitarian organizations are struggling with these exact same issues. Yes, these organizations have long had data privacy and protection protocols in place but these were designed for a world devoid of social media. One major social media company is also looking to revisit its terms of service agreements given the increasing relevance of their platform in humanitarian response. The challenge, for all, will be to strike the right balance between innovation and regulation.

12 Comments

Posted in Crisis Mapping, Crowdsourcing, Social Media

Tagged , , ,

Some Thoughts on Real-Time Awareness for Tech@State

Posted on February 2, 2012 | 6 comments

I’ve been invited to present at Tech@State in Washington DC to share some thoughts on the future of real-time awareness. So I thought I’d use my blog to brainstorm and invite feedback from iRevolution readers. The organizers of the event have shared the following questions with me as a way to guide the conver-sation: Where is all of this headed?  What will social media look like in five to ten years and what will we do with all of the data? Knowing that the data stream can only increase in size, what can we do now to prepare and prevent being over-whelmed by the sheer volume of data?

These are big, open-ended questions, and I will only have 5 minutes to share some preliminary thoughts. I shall thus focus on how time-critical crowdsourcing can yield real-time awareness and expand from there.

Two years ago, my good friend and colleague Riley Crane won DARPA’s $40,000 Red Balloon Competition. His team at MIT found the location of 10 weather balloons hidden across the continental US in under 9 hours. The US covers more than 3.7 million square miles and the balloons were barely 8 feet wide. This was truly a needle-in-the-haystack kind of challenge. So how did they do it? They used crowdsourcing and leveraged social media—Twitter in particular—by using a “recursive incentive mechanism” to recruit thousands of volunteers to the cause. This mechanism would basically reward individual participants financially based on how important their contributions were to the location of one or more balloons. The result? Real-time, networked awareness.

Around the same time that Riley and his team celebrated their victory at MIT, another novel crowdsourcing initiative was taking place just a few miles away at The Fletcher School. Hundreds of students were busy combing through social and mainstream media channels for actionable and mappable information on Haiti following the devastating earthquake that had struck Port-au-Prince. This content was then mapped on the Ushahidi-Haiti Crisis Map, providing real-time situational awareness to first responders like the US Coast Guard and US Marine Corps. At the same time, hundreds of volunteers from the Haitian Diaspora were busy translating and geo-coding tens of thousands of text messages from disaster-affected communities in Haiti who were texting in their location & most urgent needs to a dedicated SMS short code. Fletcher School students filtered and mapped the most urgent and actionable of these text messages as well.

One year after Haiti, the United Nation’s Office for the Coordination of Humanitarian Affairs (OCHA) asked the Standby Volunteer Task Force (SBTF) , a global network of 700+ volunteers, for a real-time map of crowdsourced social media information on Libya in order to improve their own situational awareness. Thus was born the Libya Crisis Map.

The result? The Head of OCHA’s Information Services Section at the time sent an email to SBTF volunteers to commend them for their novel efforts. In this email, he wrote:

“Your efforts at tackling a difficult problem have definitely reduced the information overload; sorting through the multitude of signals on the crisis is no easy task. The Task Force has given us an output that is manageable and digestible, which in turn contributes to better situational awareness and decision making.”

These three examples from the US, Haiti and Libya demonstrate what is already possible with time-critical crowdsourcing and social media. So where is all this headed? You may have noted from each of these examples that their success relied on the individual actions of hundreds and sometimes thousands of volunteers. This is primarily because automated solutions to filter and curate the data stream are not yet available (or rather accessible) to the wider public. Indeed, these solutions tend to be proprietary, expensive and/or classified. I thus expect to see free and open source solutions crop up in the near future; solutions that will radically democratize the tools needed to gain shared, real-time awareness.

But automated natural language processing (NLP) and machine learning alone are not likely to succeed, in my opinion. The data stream is actually not a stream, it is a massive torent of non-indexed information, a 24-hour global firehose of real-time, distributed multi-media data that continues to outpace our ability to produce actionable intelligence from this torrential downpour of 0’s and 1’s. To turn this data tsunami into real-time shared awareness will require that our filtering and curation platforms become more automated and collaborative. I believe the key is thus to combine automated solutions with real-time collabora-tive crowdsourcing tools—that is, platforms that enable crowds to collaboratively filter and curate real-time information, in real-time.

Right now, when we comb through Twitter, for example, we do so on our own, sitting behind our laptop, isolated from others who may be seeking to filter the exact same type of content. We need to develop free and open source platforms that allow for the distributed-but-networked, crowdsourced filtering and curation of information in order to democratize the sense-making of the firehose. Only then will the wider public be able to win the equivalent of Red Balloon competitions without needing $40,000 or a degree from MIT.

I’d love to get feedback from readers about what other compelling cases or arguments I should bring up in my presentation tomorrow. So feel free to post some suggestions in the comments section below. Thank you!

6 Comments

Posted in Big Data, Crisis Mapping, Crowdsourcing, Social Computing, Social Media

Tagged , , , , , , , , , , , , , ,

How to Crowdsource Better Governance in Authoritarian States

Posted on February 1, 2012 | 8 comments

I was recently asked to review this World Bank publication entitled: “The Role of Crowdsourcing for Better Governance in Fragile States Contexts.” I had been looking for just this type of research on crowdsourcing for a long time and was therefore well pleased to read this publication. This blog posts focuses more on the theoretical foundations of the report, i.e., Part 1. I highly recommend reading the full study given the real-world case studies that are included.

“[The report serves] as a primer on crowdsourcing as an information resource for development, crisis response, and post-conflict recovery, with a specific focus on governance in fragile states. Inherent in the theoretical approach is that broader, unencumbered participation in governance is an objectively positive and democratic aim, and that governments’ accountability to its citizens can be increased and poor-performance corrected, through openness and empowerment of citizens. Whether for tracking aid flows, reporting on poor government performance, or helping to organize grassroots movements, crowdsourcing has potential to change the reality of civic participation in many developing countries. The objective of this paper is to outline the theoretical justifications, key features and governance structures of crowdsourcing systems, and examine several cases in which crowdsourcing has been applied to complex issues in the developing world.”

The research is grounded in the philosophy of Open-Source Governance, “which advocates an intellectual link between the principles of open-source and open-content movements, and basic democratic principles.” The report argues that “open-source governance theoretically provides more direct means to affect change than do periodic elections,” for example. According to the authors of the study, “crowdsourcing is increasingly seen as a core mechanism of a new systemic approach of governance to address the highly complex, globally interconnected and dynamic challenges of climate change, poverty, armed conflict, and other crises, in view of the frequent failures of traditional mechanisms of democracy and international diplomacy with respect to fragile state contexts.”

That said, how exactly is crowdsourcing supposed to improve governance? The authors argues that “in general, ‘transparency breeds self-correcting behavior’ among all types of actors, since neither governments nor businesses or  individuals want to be caught at doing something embarrassing and or illegal.” Furthermore, “since crowdsourcing is in its very essence based on universal participation, it is supporting the empowerment of people. Thus, in a pure democracy or in a status of anarchy or civil war (Haiti after the earthquake, or Libya since February 2011), there are few external limitations to its use, which is the reason why most examples are from democracies and situations of crisis.” On the other hand, an authoritarian regime will “tend to oppose and interfere with crowdsourcing, perceiving broad-based participation and citizen empowerment as threats to its very existence.”

So how can crowdsourcing improve governance in an authoritarian state? “Depending on the level of citizen-participation in a given state,” the authors argue that “crowdsourcing can potentially support governments’ and/or civil society’s efforts in informing, consulting, and collaborating, leading to empowerment of citizens, and encouraging decentralization and democrati-zation. By providing the means to localize, visualize, and publish complex, aggregated data, e.g. on a multi-layer map, and the increasing speed of genera-ting and sharing data up to real-time delivery, citizens and beneficiaries of government and donors become empowered to provide feedback and even become information providers in their own right.”

According to the study, this transformation can take place in three ways:

1) By sharing, debating and contributing to publicly available government, donor and other major actors’ databases, data can be distributed directly through customized web and mobile applications and made accessible and meaningful to citizens.

2) By providing independent platforms for ‘like-minded people’ to connect and collaborate, builds potential for the emergence of massive, internationally connected grassroots movements.

3) By establishing platforms that aggregate and compare data provided by the official actors such as governments, donors, and companies with crowdsourced primary data and feedback.

“The tracking of data by citizens increases transparency as well as pressure for better social accountability. Greater effectiveness of state and non-state actors can be achieved by using crowdsourced data and deliberations* to inform the provision of their services. While the increasing volume of data generated as well as the speed of transactions can be attractive even to fragile-state governments, the feature of citizen empowerment is often considered as serious threat (Sudan, Egypt, Syria,Venezuela etc.).” *The authors argue that this need to be done through “web-based deliberation platforms (e.g. DiscourseDB) that apply argumentative frameworks for issue-based argument instead of simple polling.”

The second part of the report includes a section on Crisis Mapping in which two real-world case studies are featured: the Ushahidi-Haiti Crisis Map & Mission4636 and the Libya Crisis Map. Other case studies include the UN’s Threat and Risk Mapping Analysis (TRMA) initiative in the Sudan, Participatory GIS and Community Forestry in Nepal; Election Monitoring in Guinea; Huduma and Open Data in Kenya; Avaaz and other emergent applications of crowd-sourcing for economic development and good governance. The third and final part of the study provides recommendations for donors on how to apply crowd-sourcing and interactive mapping for socio-economic recovery and development in fragile states.

8 Comments

Posted in Crowdsourcing, Social Media

Tagged , , , , , , ,

Do “Liberation Technologies” Change the Balance of Power Between Repressive Regimes and Civil Society?

Posted on November 15, 2011 | 4 comments

My dissertation is now available for download. Many thanks to my dissertation committee for their support and feedback throughout: Professor Dan Drezner, Professor Larry Diamond, Professor Carolyn Gideon and Clay Shirky. This dissertation is dedicated to Khaled Mohamed Saeed and Mohamed Bouazizi.

Abstract

Do new information and communication technologies (ICTs) empower repressive regimes at the expense of civil society, or vice versa? For example, does access to the Internet and mobile phones alter the balance of power between repressive regimes and civil society? These questions are especially pertinent today given the role that ICTs played during this year’s uprisings in Tunisia, Egypt and beyond. Indeed, as one Egyptian activist stated, “We use Facebook to schedule our protests, Twitter to coordinate and YouTube to tell the world.” But do these new ICTs—so called “liberation technologies”—really threaten repressive rule? The purpose of this dissertation is to use mixed-methods research to answer these questions.

The first half of my doctoral study comprised a large-N econometric analysis to test whether “liberation technologies” are a statistically significant predictor of anti-government protests in countries with repressive regimes. If using the Internet and mobile phones facilitates organization, mobilization and coordina-tion, then one should expect a discernible link between an increase in access to ICTs and the frequency of protests—particularly in repressive states. The results of the quantitative analysis were combined with other selection criteria to identify two country case studies for further qualitative comparative analysis: Egypt and the Sudan.

The second half of the dissertation assesses the impact of “liberation technologies” during the Egyptian Parliamentary Elections and Sudanese Presidential Elections of 2010. The analysis focused specifically on the use of Ushahidi—a platform often referred to as a “liberation technology.” Descriptive analysis, process tracing and semi-structured interviews were carried out for each case study. The results of the quantitative and qualitative analyses were mixed. An increase in mobile phone access was associated with a decrease in protests for four of the five regression models. Only in one model was an increase in Internet access associated with an increase in anti-government protests. As for Ushahidi, the Egyptian and Sudanese dictatorships were indeed threatened by the technology because it challenged the status quo. Evidence suggests that this challenge tipped the balance of power marginally in favor of civil society in Egypt, but not in the Sudan, and overall not significantly.

The main contributions and highlights of my dissertation include:

* New dataset on protests, ICTs, political and economic variables over 18 years.
* New econometric analysis and contribution to quantitative political science.
* New conceptual framework to assess impact of ICTs on social, political change.
* New operational application of conceptual framework to assess impact of ICTs.
* New datasets on independent citizen election observation in repressive states.
* New insights into role of ICTs in civil resistance against authoritarian regimes.
* New comprehensive literature on impact of ICTs on protests, activism, politics.
* New targeted policy recommendations based on data driven empirical analysis.
* New lessons learned and best practices in using the Ushahidi platform.

A PDF copy of my dissertation is available here.

4 Comments

Posted in Crisis Mapping, Crowdsourcing, Digital Activism, Social Media

Tagged , ,

My Opening Speech at CrisisMappers 2011 in Geneva

Posted on November 14, 2011 | 1 comment

Good Afternoon Crisis Mappers!

It is my great pleasure and honor to open the third International Conference of CrisisMappers. Thank you very much for being here and for contributing both your time and expertise to ICCM 2011. This past year has been a challenging and busy year for all of us in the CrisisMappers community. So the timing of this conference and its location in this quiet and scenic region of Switzerland provides the perfect opportunity to pause, take a deep breath and gently reflect on the past 12 months.

As many of you already know, the CrisisMappers Community is an informal network of members who operate at the cutting edge of crisis mapping and humanitarian technology. We are not a formal entity; we have no office, no one location, no staff, and no core funding to speak of. And yet, more than 3,000 individuals representing over 1,500 organizations in 140 countries around the world have joined this growing and thriving network.

Some of you here today were also with us in Cleveland for ICCM 2009, which is where and when, this Crisis Mappers Community was launched. We collectively founded this network for a very simple reason: to advance the study, practice and impact of crisis mapping by catalyzing information sharing and forming unique partnerships between members. A lot has happened since Cleveland, and yes, that is indeed an understatement. Take the following as just a simple proxy: shortly before ICCM 2009, I did a Google search for “crisis mapping”; this returned some 8,000 hits. Today, just two short years later, this number is well over a quarter million and growing rapidly. Much of this new content and activity is a direct result of our combined efforts, particularly in 2011.

To be sure, we have seen many new exciting developments in the field of crisis mapping and humanitarian technology in just the past 12 months. In fact, there are simply too many to highlight in these short introductory remarks, so I invite you to visit the CrisisMappers website for the full list of projects that you yourselves have ranked as most important in 2011. Over the next two days, many of these projects will be featured in Ignite Talks, demo’s and posters in the Techmology Fair and in the self-organized sessions as well.

In addition to these fine projects, a number of important and recurring themes have emerged over the past year. So I’d like to briefly touch on just five of these as a way to inform some of our conversations over the next two days.

The first is validation. We need to better assess the impact of our work. More specifically, we need independent experts who specialize in monitoring and evaluation (M&E) to critically assess our crisis mapping deployments. I thus urge our donors, many of you are here today, to make independent evaluations a requirement for all your grantees who actively deploy crisis mapping platforms. Rigorous evaluations do cost money so I strongly encourage you to make funding available in 2012 so we can validate our work.

A second theme is security. We all know that the majority of crisis mapping platforms and the technologies they integrate were not designed for highly hostile environments. At the same time, computer security is a highly specialized field and we are in serious need for security experts to lend their direct support at the coding level to resolve existing security risks. Talking is important, but coding is more important. Security experts who are members of the Crisis Mappers community already know what needs to be done. So lets get this done. What we do need to talk about is developing a clear and well defined set guidelines on how to handle Open (Social) Data that is crowdsourced from conflict zones. To be sure, we urgently need a code of conduct and one endorsed by an established and credible organization to hold ourselves accountable.

The third theme I would like to highlight is the consolidation of key partnerships between formal humanitarian organizations and informal volunteer networks. We began this conversation together exactly 12 months ago at ICCM 2010. And a considerable amount of time and energy has since gone into developing the initial scaffolding necessary to streamline these partnerships. But we still have much work to do. There is absolutely no doubt that these partnerships will continue to be critical in 2012, so we need to have these collaboration mechanisms in place earlier rather than later. To do this, we need to participate in joint crisis response simulations now to ensure that we end up with appropriate, and robust but flexible mechanisms in 2012.

A fourth recurring theme this year has been the increasing need to scale our crisis mapping efforts. This requires a change in data licensing, particularly around satellite imagery and the data derived thereof. We also need both micro-tasking platforms and automated filtering mechanisms to scale our efforts. On filtering, for example, we need natural language processing (NLP) tools to help us monitor, aggregate, triangulate and verify large volumes of social media data and text messages in real time. While these solutions already exist in the private sector and increasingly in public health, they are still not accessible or widely used by many members of the CrisisMappers community. This needs to change. The good news is that a number of colleagues who are here at ICCM have been actively working on developing micro-tasking and automated filtering solutions. I sincerely hope they’ll share their platforms more widely with the CrisisMappers community in 2012.

A fifth and final theme is of course “Mainstreaming Crisis Mapping,” the theme of this year’s international conference. Our co-hosts ICT4Peace and the JRC will discuss this theme in detail in their keynote address. So let me now turn it over to my fellow colleague and co-founder, Professor Jen Ziemke, to tell you more about our co-hosts and what to expect over the next two days…

1 Comment

Posted in Crisis Mapping, Crowdsourcing, Humanitarian Technologies, Social Media

Tagged , ,

Time-Critical Crowdsourcing for Social Mobilization and Crowd-Solving

Posted on November 6, 2011 | 15 comments

My good friend Riley Crane just co-authored a very interesting study entitled “Time-Critical Social Mobilization” in the peer-reviewed journal Science. Riley spearheaded the team at MIT that won the DARPA Red Balloon competition last year. His team found the locations of all 10 weather balloons hidden around the continental US in under 9 hours. While we were already discussing alternative approaches to crowdsourcing for social impact before the competition, the approach he designed to win the competition certainly gave us a whole lot more to talk about given the work I’d been doing on crowd sourcing crisis information and near real-time crisis mapping.

Crowd-solving non-trivial problems in quasi real-time poses two important challenges. A very large number of participants is typically required couple with extremely fast execution. Another common challenge is the need for some sort of search process. “For example, search may be conducted by members of the mobilized community for survivors after a natural disaster.” Recruiting large numbers of participants, however, requires that individuals be motivated to actually conduct the search and participate in the information diffusion. Clearly, “providing appropriate incentives is a key challenge in social mobilization.”

This explains the rationale behind DARPA decision to launch their Red Balloon Challenge: “to explore the roles the Internet and social networking play in the timely communication, wide-area team-building, and urgent mobilization required to solve broad-scope, time-critical problems.” So 10 red weather balloons were discretely placed at different locations in the continental US. A senior analyst at the National Geospatial-Intelligence Agency is said to have characterized the challenge is impossible for conventional intelligence-gathering methods. Riley’s team found all 10 balloons in 8 hours and 36 minutes. How did they do it?

Some 36 hours before the start of the challenge, the team at MIT had already recruited over 4,000 participants using a “recursive incentive mechanism.” They used the $40,000 prize money that would be awarded by the winners of the challenge as a “financial incentive structure rewarding not only the people who correctly located the balloons but also those connecting the finder [back to the MIT team].” If Riley and colleagues won:

we would allocate $4000 in prize money to each of the 10 balloons. We promised $2000 per balloon to the first person to send in the cor- rect balloon coordinates. We promised $1000 to the person who invited that balloon finder onto the team, $500 to whoever invited the in- viter, $250 to whoever invited that person, and so on. The underlying structure of the “recursive incentive” was that whenever a person received prize money for any reason, the person who in- vited them would also receive money equal to half that awarded to their invitee

In other words, the reward offers by Team MIT “scales with the size of the entire recruitment tree (because larger trees are more likely to succeed), rather than depending solely on the immediate recruited friends.” What is stunning about Riley et al.’s approach is that their “attrition rate” was almost half the rate of other comparable social network experiments. In other words, participants in the MIT recruitment tree were about twice as likely to “play the game” so-to-speak rather than give up. In addition, the number recruited by each individual followed a power law distribution, which suggests a possible tipping point dynamic.

In conclusion, the mechanism devised by the winning team “simultaneously provides incentives for participation and for recruiting more individuals to the cause.” So what insights does this study provide vis-a-vis live crisis mapping initiatives that are volunteer-based, like those spearheaded by the Standby Volunteer Task Force (SBTF) and the Humanitarian OpenStreetMap (HOT) communities? While these networks don’t have any funding to pay volunteers (this would go against the spirit of volunteerism in any case), I think a number of insights can nevertheless be drawn.

In the volunteer sector, the “currency of exchange” is credit. That is, the knowledge and acknowledgement that I participated in the Libya Crisis Map to support the UN’s humanitarian operations, for example. I recently introduced SBTF “deployment badges” to serve in part the public acknowledgment incentive. SBTF volunteers can now add badges for deployments there were engaged in, e.g., “Sudan 2011”; “New Zealand 2011”, etc.

What about using a recursive credit mechanism? For example, it would be ideal if volunteers could find out how a given report they worked on was ultimately used by a humanitarian colleague monitoring a live map. Using the Red Balloon analogy, the person who finds the balloon should be able to reward all those in her “recruitment tree” or in our case “SBTF network”. Lets say Helena works for the UN and used the Libya Crisis Map whilst in Tripoli. She finds an important report on the map and shares this with her colleagues on the Tunisian border who decide to take some kind of action as a result. Now lets say this report came from a tweet that Chrissy in the Media Monitoring Team found while volunteering on the deployment. She shared the tweet with Jess in the GPS Team who found the coordinates for the location referred to in that tweet. Melissa then added this to the live map being monitored by the UN. Wouldn’t be be ideal if each could be sent an email letting them know about Helena’s response? I realize this isn’t trivial to implement but what would have to be in place to make something like this actually happen? Any thoughts?

On the recruitment side, we haven’t really done anything explicitly to incentivize current volunteers to recruit additional volunteers. Could we incentivize this beyond giving credit? Perhaps we could design a game-like point system? Or a fun ranking system with different titles assigned according to the number of volunteers recruited? Another thought would be to simply ask existing volunteers to recruit one or two additional volunteers every year. We currently have about 700 volunteers in the SBTF, so this might be one way to increase substantially in size.

I’m not sure what type of mechanism we could devise to simultaneously provide incentives for participation and recruitment. Perhaps those incentives already exist, in the sense that the SBTF response to international crises, which perhaps serves as a sufficient draw. I’d love to hear what iRevolution readers think, especially if you have good ideas that we could realistically implement!

15 Comments

Posted in Crisis Mapping, Crowdsourcing, Digital Activism, Social Media

Tagged , , , , , , ,

Amplifying Somali Voices Using SMS and a Live Map

Posted on November 1, 2011 | 2 comments

Update: http://irevolution.net/2011/12/08/somaliaspeaks

I recently had the pleasure to meet with Al-Jazeera’s Social Media Team in Doha, Qatar. It was immediately clear that they were also interested in partnering on a joint project in Somalia when I suggested a few ideas. Several weeks later, this project is almost ready to launch. The purpose of this initiative is to let Somalis speak for themselves and to amplify those voices in the international media.

As Al-Jazeera has noted, Somalia is quickly slipping from global media attention. With Somalia out of the headline news, however, advocacy and lobbying groups will find it increasingly difficult to place pressure on policymakers and humanitarian organiza-tions to scale their intervention in this major crisis. This project therefore offers a direct and innovative way to keep Somalia in the international news. The project described below is the product of a novel collaborative effort between Al-Jazeera, Ushahidi, Souktel and Crowdflower in direct partnership with the Somali Diaspora.

The project will “interview” ordinary Somalis in Somalia and let them speak for themselves in the international media space. Interview questions drafted by Al-Jazeera will be broadcast via SMS by Souktel to 10% of their existing 50,000+ subscribers in the country. The interview questions will also invite Somalis to share in which town they are based. (Note that we are reviewing the security protocols for this). The Somali Diaspora will then translate and geolocate incoming text messages from Somali to English using a customized Crowdflower plugin. The processed messages will then be pushed (in both Somali and English) to a live Ushahidi map. Al-Jazeera will promote the live map across their main-stream and social media channels. Mapped SMS’s will each have a comments section for viewers and readers to share their thoughts. Al-Jazeera will then select the most compelling responses and text these back to the original senders in Somalia. This approach is replicable and scalable given that the partners and technologies are largely in place already.

In sum, the purpose of this project is to increase global media attention on Somalia by letting Somali voices take center stage—voices that are otherwise not heard in the international, mainstream media. If journalists are not going to speak about Somalia, then lets invite Somalis speak to the world themselves. The project will highlight these voices on a live, public map for the world to engage in a global conversation with the people of Somalia, a conversation in which Somalis and the Diaspora are themselves at the centerfold.

If you want to help out with this initiative, we’re looking for Somali-English speakers to translate and map the incoming text messages. It’s important that volunteers are familiar with the location of many cities, towns, etc., in Somalia in order to map the SMS’s. If you have the skills and time, then please add your name, email address and short bio here—would be great to have you on the team!

 

2 Comments

Posted in Crisis Mapping, Crowdsourcing, Digital Activism, Social Media

Tagged , , , , , , , , , ,

Detecting Emerging Conflicts with Web Mining and Crisis Mapping

Posted on October 10, 2011 | 8 comments

My colleague Christopher Ahlberg, CEO of Recorded Future, recently got in touch to share some exciting news. We had discussed our shared interests a while back at Harvard University. It was clear then that his ideas and existing technologies were very closely aligned to those we were pursuing with Ushahidi’s Swift River platform. I’m thrilled that he has been able to accomplish a lot since we last spoke. His exciting update is captured in this excellent co-authored study entitled “Detecting Emergent Conflicts Through Web Mining and Visualization” which is available here as a PDF.

The study combines almost all of my core interests: crisis mapping, conflict early warning, conflict analysis, digital activism, pattern recognition, natural language processing, machine learning, data visualization, etc. The study describes a semi-automatic system which automatically collects information from pre-specified sources and then applies linguistic analysis to user-specified extract events and entities, i.e., structured data for quantitative analysis.

Natural Language Processing (NLP) and event-data extraction applied to crisis monitoring and analysis is of course nothing new. Back in 2004-2005, I worked for a company that was at the cutting edge of this field vis-a-vis conflict early warning. (The company subsequently joined the Integrated Conflict Early Warning System (ICEWS) consortium supported by DARPA). Just a year later, Larry Brilliant told TED 2006 how the Global Public Health Information Net-work (GPHIN) had leveraged NLP and machine learning to detect an outbreak of SARS 3 months before the WHO. I blogged about this, Global Incident Map, European Media Monitor (EMM), Havaria, HealthMap and Crimson Hexagon back in 2008. Most recently, my colleague Kalev Leetaru showed how applying NLP to historical data could have predicted the Arab Spring. Each of these initiatives represents an important effort in leveraging NLP and machine learning for early detection of events of interest.

The RecordedFuture system works as follows. A user first selects a set of data sources (websites, RSS feeds, etc) and determines the rate at which to update the data. Next, the user chooses one or several existing “extractors” to find specific entities and events (or constructs a new type). Finally, a taxonomy is selected to specify exactly how the data is to be grouped. The data is then automatically harvested and passed through a linguistics analyzer which extracts useful information such as event types, names, dates, and places. Finally, the reports are clustered and visualized on a crisis map, in this case using an Ushahidi platform. This allows for all kinds of other datasets to be imported, compared and analyzed, such as high resolution satellite imagery and crowdsourced data.

A key feature of the RecordedFuture system is that extracts and estimates the time for the event described rather than the publication time of the newspaper article parsed, for example. As such, the harvested data can include both historic and future events.

In sum, the RecordedFuture system is composed of the following five features as described in the study:

1. Harvesting: a process in which text documents are retrieved from various sources and stored in the database. The documents are stored for long-term if permitted by terms of use and IPR legislation, otherwise they are only stored temporarily for the needed analysis.

2. Linguistic analysis: the process in which the retrieved texts are analyzed in order to extract entities, events, time and location, etc. In contrast to other components, the linguistic analysis is language dependent.

3. Refinement: additional information can be obtained in this process by synonym detection, ontology analysis, and sentiment analysis.

4. Data analysis: application of statistical and AI-based models such as Hidden Markov Models (HMMs) and Artificial Neural Networks (ANNs) to generate predictions about the future and detect anomalies in the data.

5. User experience: a web interface for ordinary users to interact with, and an API for interfacing to other systems.

The authors ran a pilot that “manually” integrated the RecordedFuture system with the Ushahidi platform. The result is depicted in the figure below. In the future, the authors plan to automate the creation of reports on the Ushahidi platform via the RecordedFuture system. Intriguingly, the authors chose to focus on protest events to demo their Ushahidi-coupled system. Why is this intriguing? Because my dissertation analyzed whether access to new information and communication technologies (ICTs) are statistically significant predictors of protest events in repressive states. Moreover, the protest data I used in my econometric analysis came from an automated NLP algorithm that parsed Reuters Newswires.

Using RecordedFuture, the authors extracted some 6,000 protest event-data for Quarter 1 of 2011. These events were identified and harvested using a “trained protest extractor” constructed using the system’s event extractor frame-work. Note that many of the 6,000 events are duplicates because they are the same events but reported by different forces. Not surprisingly, Christopher and team plan to develop a duplicate detection algorithm that will also double as a triangulation & veracity scoring feature. I would be particularly interested to see them do this kind of triangulation and validation of crowdsourced data on the fly.

Below are the protest events picked up by RecordedFuture for both Tunisia and Egypt. From these two figures, it is possible to see how the Tunisian protests preceded those in Egypt.

The authors argue that if the platform had been set up earlier this year, a user would have seen the sudden rise in the number of protests in Egypt. However, the authors acknowledge that their data is a function of media interest and attention—the same issue I had with my dissertation. One way to overcome this challenge might be by complementing the harvested reports with crowdsourced data from social media and Crowdmap.

In the future, the authors plan to have the system auto-detect major changes in trends and to add support for the analysis of media in languages beyond English. They also plan to test the reliability and accuracy of their conflict early warning algorithm by comparing their forecasts of historical data with existing conflict data sets. I have several ideas of my own about next steps and look forward to speaking with Christopher’s team about ways to collaborate.

8 Comments

Posted in Big Data, Crisis Mapping, Early Warning, Social Media

Tagged , , , , , ,

Theorizing Ushahidi: An Academic Treatise

Posted on October 2, 2011 | 10 comments

[This is an excerpt taken from Chapter 1 of my dissertation]

Activists are not only turning to social media to document unfolding events, they are increasingly mapping these events for the world to bear witness. We’ve seen this happen in Tunisia, Egypt, Libya, Syria, Yemen and beyond. My colleague Alexey Sidorenko describes this new phenomenon as a “mapping reflex.” When student activists from Khartoum got in touch earlier this year, they specifically asked for a map, one that would display their pro-democracy protests and the government crackdown. Why? They wanted the world to see that the Arab Spring extended to the Sudan.

The Ushahidi platform is increasingly used to map information generated by crowds in near-real time like the picture depicted above. Why is this important? Because live public maps can help synchronize shared awareness, an important catalyzing factor of social movements, according to Jürgen Habermas. Recall Habermas’s treatise that “those who take on the tools of open expression become a public, and the presence of a synchronized public increasingly constrains un-democratic rulers while expanding the right of that public.”

Sophisticated political maps have been around for hundreds of years. But the maps of yesteryear, like the books of old, were created and controlled by the few. While history used to be written by the victors, today, journalists like my colleague Anand Giridharadas from the New York Times are asking whether the triangulated crisis map will become the new first draft of history. In the field of geography and cartography, some refer to this new wave of democratized map-making as “neo-geography.” But this new type of geography is not only radically different from traditional approaches because it is user-generated and more par-ticipatory; the fact that today’s dynamic maps can also be updated and shared in near real-time opens up an entire new world of possibilities and responses.

Having a real time map is almost as good as having your own helicopter. A live map provides immediate situational awareness, a third dimension and additional perspective on events unfolding in time and space. Moreover, creating a map catalyzes conversations between activists, raises questions about geographic patterns or new incidents, and leads to more questions regarding the status quo in a repressive environment. To be sure, mass media alone does not change people’s minds.  Recall that political change is a two-step process, with the second—social step—being where political opinions are formed (Katz and Lazarsfeld 1955). “This is the step in which the Internet in general, and social media in particular, can make a difference” (Shirky 2010). In addition, the collaboration that takes place when creating a live map can also reinforce weak and strong ties, both of which are important for civil resistance.

The Ushahidi platform enables a form of live-mapped “sousveillance,” which refers to the recording of an activity using portable personal technologies. In many respects, however, the use of Ushahidi goes beyond sousveillance in that it generates the possibility of “dataveillance” and a possible reversal of Bentham’s panopticon. “With postmodernity, the panopticon has been informationalized; what once was organized around hierarchical observation is now organized through decoding and recoding of information” (Lyon 2006). In Seeing Like a State, James Scott argues eloquently that this process of decoding and recoding was for centuries the sole privilege of the State. In contrast, the Ushahidi platform provides a participatory digital canvas for the public decoding, recoding of information and synchronization of said information. In other words, the platform serves to democratize dataveillance by crowdsourcing what was once the exclusive realm of the “security-informational complex.”

In Domination and the Arts of Resistance: Hidden Transcripts published in 1990, James Scott distinguishes between public and hidden transcripts. The former describes the open, public interactions that take place between domina-tors and oppressed while hidden transcripts relate to the critique of power that “goes on offstage” and which the power elites cannot decode. This hidden transcript is comprised of the second step, social conversations, that Katz and Lazarsfeld (1955) argue ultimately change political behavior. Scott writes that when the oppressed classes publicize this “hidden transcript”, they become conscious of its common status. Borrowing from Habermas, the oppressed thereby become a public and more importantly a synchronized public. In many ways, the Ushahidi platform is a vehicle by which the hidden transcript is collectively published and used to create shared awareness—thereby threatening to alter the balance of power between the oppressors and oppressed.

The new dynamics that are enabled by “liberation technologies” like Ushahidi may enable a different form of democracy, one which arising from “the inability of electoral/representative politics to keep it promises [has thus] led to the development of indirect forms of democracy” (Rosanvallon 2008). More specifically, Rosanvallon indentifies three channels whereby civil society can hold the state accountable not just during elections but also between elections and independent of their results. “The first refers to the various means whereby citizens (or, more accurately, organizations of citizens) are able to monitor and publicize the behavior of elected and appointed rulers; the second to their capacity to mobilize resistance to specific policies, either before or after they have been selected; the third to the trend toward ‘juridification’ of politics  [cf. dataveillance] when individuals or social groups use the courts and, especially, jury trials to bring delinquent politicians to judgment” (Schmitter 2008, PDF).

These three phases correspond surprisingly well with the three waves of Ushahidi uses witnessed over the past three years. The first wave was reactive and documentary focused. The second was more pro-active and focused on action beyond documentation while the third seeks to capitalize on the first two to complete the rebalancing of power. Perhaps this final wave is the teleological purpose of the Ushahidi platform or What Technology Wants as per Kevin Kelly’s treatise. However, this third wave, the trend toward the “juridificaiton” of democracy bolstered by crowdsourced evidence that is live-mapped on a public Ushahidi platform, is today more a timid ripple than a tsunami of change reversing the all-seeing “panopticon”. A considerable amount of learning-by-doing remains to be done by those who wish to use the Ushahidi platform for impact beyond the first two phases of Rosanvallon’s democracy.

10 Comments

Posted in Crisis Mapping, Crowdsourcing, Digital Activism, Social Media

Tagged , , , , ,

On Synchrony, Technology and Revolutions: The Political Power of Synchronized Resistance

Posted on August 14, 2011 | 10 comments

Synchronized action is a powerful form of resistance against repressive regimes. Even if the action itself is harmless, like walking, meditation or worship, the public synchrony of that action by a number of individuals can threaten an authoritarian state. To be sure, synchronized public action demonstrates independency which may undermine state propaganda, reverse information cascades and thus the shared perception that the regime is both in control and unchallenged.

This is especially true if the numbers participating in synchrony reaches a tipping point. As Karl Marx writes in Das Kapital, “Merely quantitative differences, beyond a certain point, pass into qualitative changes.” We call this “emergent behavior” or “phase transitions” in the field of complexity science. Take a simple example from the physical world: the heating of water. A one degree increase in temperature is a quantitative change. But keep adding one degree and you’ll soon reach the boiling point of water and surprise! A physical phase transition occurs: liquid turns into gas.

In social systems, information creates friction and heat. Moreover, today’s information and communication technologies (ICTs) are perhaps the most revolutionary synchronizing tools for “creating heat” because of their scalability. Indeed, ICTs today can synchronize communities in ways that were unimaginable just a few short years ago. As one Egyptian activist proclaimed shortly before the fall of Mubarak, “We use Facebook to scheduled our protests, Twitter to coordinate, and YouTube to tell the world.” The heat is already on.

Synchrony requires that individuals be connected in order to synchronize. Well guess what? ICTs are mass, real-time connection technologies. There is conse-quently little doubt in my mind that “the advent and power of connection technologies—tools that connect people to vast amounts of information and to one another—will make the twenty-first century all about surprises;” surprises that take the form of “social phase transitions” (Schmidt and Cohen 2011). Indeed, ICTs can  dramatically increase the number of synchronized participants while sharply reducing the time it takes to reach the social boiling point. Some refer to this as “punctuated equilibria” or “reversed information cascades” in various academic literatures. Moreover, this can all happen significantly faster than ever before, and as argued in this previous blog post on digital activism, faster is indeed different.

Clay Shirky argues that “this basic hypothesis is an updated version of that outlined by Jürgen Habermas in his 1962 publication, The Structural Transformation of the Public Sphere: an Inquiry into a Category of Bourgeois Society. A group of people, so Habermas’s theory goes, who take on the tools of open expression becomes a public, and the presence of a synchronized public increasingly constrains undemocratic rulers while expanding the rights of that public […].” But to understand the inherent power of synchrony and then leverage it, we must first recognized that synchrony is a fundamental force of nature that goes well beyond social systems.

In his TED Talk from 2004, American mathematician Steven Strogatz argues that synchrony may be one of the most pervasive drivers in all of nature, extending from the subatomic scale to the farthest reaches of the cosmos. In many ways, this deep tendency towards spontaneous order is what pushes back against the second law of thermodynamics, otherwise known as entropy. 

Strogatz shares example from nature and shows a beautiful ballet of hundreds of birds flocking in unison. He explains that this display of synchrony has to do with defense. “When you’re small and vulnerable […] it helps to swarm to avoid and/or confuse predators.” When a predator strikes, however, all bets are off, and everyone disperses—but only temporarily. “The law of attraction,” says Strogatz, brings them right back together in synchrony within seconds. “There’s this constant splitting and reforming,” grouping and dispersion—swarming—which has several advantages. If you’re in a swarm, the odds of getting caught are far lower. There are also many eyes to spot the danger.

What’s spectacular about these ballets is how quickly they phase from one shape to another, dispersing and regrouping almost instantaneously even across vast distances. Individual changes in altitude, speed and direction are communicated and acted on across half-a-kilometer within just seconds. The same is true of fireflies in Borneo that synchronize their blinking across large distances along the river banks. Thousands and thousands of fireflies somehow overcoming the communication delay between the one firefly at one end of the bank and the other firefly at the furthest opposite end. How is this possible? The answer to this question may perhaps provide insights for social synchrony in the context of resistance against repressive regimes.

Strogatz and Duncan Watts eventually discovered the answer, which they published in their seminal paper entitled “Collective dynamics of small-world networks.” Published in the prestigious journal Nature,  the paper became the most highly cited article about networks for 10 years and the sixth most cited paper in all of physics. A small-world network is a type of network in which even though most nodes are not neighbors of one another, most can still be reached from other nodes by a small number of hops or steps. In the context of social systems, this type of network results in the “small world phenomenon of strangers being linked by a mutual acquaintance.”

These types of networks often arise out of preferential attachment, an inherently social dynamic. Indeed, small world networks pervade social systems. So what does this mean for synchrony as applied to civil resistance? Are smart-mobs synonymous with synchronized mobs? Do ICTs increase the prevalence of small world networks in social systems—thus increasing robustness and co-synchrony between social networks. Will meshed-communication technologies and features like check-in’s alter the topology of small world networks?

Examples of synchrony from nature clearly show that real-time communication and action across large distances don’t require mobile phones. Does that mean the same is possible in social systems? Is it possible to disseminate information instantaneously within a large crowd without using communication technologies? Is strategic synchrony possible in this sense? Can social networks engage in instantaneous dispersion and cohesion tactics to confuse the repressive regime and remain safe?

I recently spoke with a colleague who is one of the world’s leading experts on civil resistance, and was astonished when she mentioned (without my prompting) that many of the tactics around civil resistance have to do with synchronizing cohesion and dispersion. On a different note, some physicists argue that small world networks are more robust to perturbations than other network structures. Indeed, the small work structure may represent an evolutionary advantage.

But how are authoritarian networks structured? Are they too of the small world variety? If not, how do they compare in terms of robustness, flexibility and speed? In many ways, state repression is a form of synchrony itself—so is genocide. Synchrony is clearly not always a good thing. How is synchrony best interrupted or sabotaged? What kind of interference strategies are effective in this context?

10 Comments

Posted in Digital Activism, Social Media

Tagged , , , , , , , , ,