Tag Archives: Crowdflower

Using CrowdFlower to Microtask Disaster Response

Posted on February 26, 2013 | 5 comments

Cross-posted from CrowdFlower blog

A devastating earthquake struck Port-au-Prince on January 12, 2010. Two weeks later, on January 27th, a CrowdFlower was used to translate text messages from Haitian Creole to English. Tens of thousands of messages were sent by affected Haitians over the course of several months. All of these were heroically translated by hundreds of dedicated Creole-speaking volunteers based in dozens of countries across the globe. While Ushahidi took the lead by developing the initial translation platform used just days after the earthquake, the translation efforts were eventually rerouted to CrowdFlower. Why? Three simple reasons:

  1. CrowdFlower is one of the leading and most highly robust micro-tasking platforms there is;
  2. CrowdFlower’s leadership is highly committed to supporting digital humanitarian response efforts;
  3. Haitians in Haiti could now be paid for their translation work.

While the CrowdFlower project was launched 15 days after the earthquake, i.e., following the completion of search and rescue operations, every single digital humanitarian effort in Haiti was reactive. The key takeaway here was the proof of concept–namely that large-scale micro-tasking could play an important role in humanitarian information management. This was confirmed months later when devastating floods inundated much of Pakistan. CrowdFlower was once again used to translate incoming messages from the disaster affected population. While still reactive, this second use of CrowdFlower demonstrated replicability.

The most recent and perhaps most powerful use of CrowdFlower for disaster response occurred right after Typhoon Pablo devastated the Philippines in early December 2012. The UN Office for the Coordination of Humanitarian Affairs (OCHA) activated the Digital Humanitarian Network (DHN) to rapidly deliver a detailed dataset of geo-tagged pictures and video footage (posted on Twitter) depicting the damage caused by the Typhoon. The UN needed this dataset within 12 hours, which required that 20,000 tweets to be analyzed as quickly as possible. The Standby Volunteer Task Force (SBTF), a member of Digital Huma-nitarians, immediately used CrowdFlower to identify all tweets with links to pictures & video footage. SBTF volunteers subsequently analyzed those pictures and videos for damage and geographic information using other means.

This was the most rapid use of CrowdFlower following a disaster. In fact, this use of CrowdFlower was pioneering in many respects. This was the first time that a member of the Digital Humanitarian Network made use of CrowdFlower (and thus micro-tasking) for disaster response. It was also the first time that Crowd-Flower’s existing workforce was used for disaster response. In addition, this was the first time that data processed by CrowdFlower contributed to an official crisis map produced by the UN for disaster response (see above).

These three use-cases, Haiti, Pakistan and the Philippines, clearly demonstrate the added value of micro-tasking (and hence CrowdFlower) for disaster response. If CrowdFlower had not been available in Haiti, the alternative would have been to pay a handful of professional translators. The total price could have come to some $10,000 for 50,000 text messages (at 0.20 cents per word). Thanks to CrowdFlower, Haitians in Haiti were given the chance to make some of that money by translating the text messages themselves. Income generation programs are absolutely critical to rapid recovery following major disasters. In Pakistan, the use of CrowdFlower enabled Pakistani students and the Diaspora to volunteer their time and thus accelerate the translation work for free. Following Typhoon Pablo, paid CrowdFlower workers from the Philippines, India and Australia categorized several thousand tweets in just a couple hours while the volunteers from the Standby Volunteer Task Force geo-tagged the results. Had CrowdFlower not been available then, it is highly, highly unlikely that the mission would have succeeded given the very short turn-around required by the UN.

While impressive, the above use-cases were also reactive. We need to be a lot more pro-active, which is why I’m excited to be collaborating with CrowdFlower colleagues to customize a standby platform for use by the Digital Humanitarian Network. Having a platform ready-to-go within minutes is key. And while digital volunteers will be able to use this standby platform, I strongly believe that paid CrowdFlower workers also have a key role to play in the digital huma-nitarian ecosystem. Indeed, CrowdFlower’s large, multinational and multi-lingual global workforce is simply unparalleled and has the distinct advantage of being very well versed in the CrowdFlower platform.

In sum, it is high time that the digital humanitarian space move from crowd-sourcing to micro-tasking. It has been three years since the tragic earthquake in Haiti but we have yet to adopt micro-tasking more widely. CrowdFlower should thus play a key role in promoting and enabling this important shift. Their con-tinued important leadership in digital humanitarian response should also serve as a model for other private sector companies in the US and across the globe.

bio

5 Comments

Posted in Crisis Mapping, Crowdsourcing, Humanitarian Technologies

Tagged , , , , , , , ,

Digital Humanitarian Response: Moving from Crowdsourcing to Microtasking

Posted on January 20, 2013 | 22 comments

A central component of digital humanitarian response is the real-time monitor-ing, tagging and geo-location of relevant reports published on mainstream and social media. This has typically been a highly manual and time-consuming process, which explains why dozens if not hundreds of digital volunteers are often needed to power digital humanitarian response efforts. To coordinate these efforts, volunteers typically work off Google Spreadsheets which, needless to say, is hardly the most efficient, scalable or enjoyable interface to work on for digital humanitarian response.

complicated128

The challenge here is one of design. Google Spreadsheets was simply not de-signed to facilitate real-time monitoring, tagging and geo-location tasks by hundreds of digital volunteers collaborating synchronously and asynchronously across multiple time zones. The use of Google Spreadsheets not only requires up-front training of volunteers but also oversight and management. Perhaps the most problematic feature of Google Spreadsheets is the interface. Who wants to spend hours staring at cells, rows and columns? It is high time we take a more volunteer-centered design approach to digital humanitarian response. It is our responsibility to reduce the “friction” and make it as easy, pleasant and re-warding as possible for digital volunteers to share their time for the better good. While some deride the rise of “single-click activism,” we have to make it as easy as a double-click-of-the-mouse to support digital humanitarian efforts.

This explains why I have been actively collaborating with my colleagues behind the free & open-source micro-tasking platform, PyBossa. I often describe micro-tasking as “smart crowdsourcing”. Micro-tasking is simply the process of taking a large task and breaking it down into a series of smaller tasks. Take the tagging and geo-location of disaster tweets, for example. Instead of using Google Spread-sheets, tweets with designated hashtags can be imported directly into PyBossa where digital volunteers can tag and geo-locate said tweets as needed. As soon as they are processed, these tweets can be pushed to a live map or database right away for further analysis.

Screen Shot 2012-12-18 at 5.00.39 PM

The Standby Volunteer Task Force (SBTF) used PyBossa in the digital disaster response to Typhoon Pablo in the Philippines. In the above example, a volunteer goes to the PyBossa website and is presented with the next tweet. In this case: “Surigao del Sur: relief good infant needs #pabloPH [Link] #ReliefPH.” If a tweet includes location information, e.g., “Surigao del Sur,” a digital volunteer can simply copy & paste that information into the search box or  pinpoint the location in question directly on the map to generate the GPS coordinates. Click on the screenshot above to zoom in.

The PyBossa platform presents a number of important advantages when it comes to digital humanitarian response. One advantage is the user-friendly tutorial feature that introduces new volunteers to the task at hand. Furthermore, no prior experience or additional training is required and the interface itself can be made available in multiple languages. Another advantage is the built-in quality control mechanism. For example, one can very easily customize the platform such that every tweet is processed by 2 or 3 different volunteers. Why would we want to do this? To ensure consensus on what the right answers are when processing a tweet. For example, if three individual volunteers each tag a tweet as having a link that points to a picture of the damage caused by Typhoon Pablo, then we may find this to be more reliable than if only one volunteer tags a tweet as such. One additional advantage of PyBossa is that having 100 or 10,000 volunteers use the platform doesn’t require additional management and oversight—unlike the use of Google Spreadsheets.

There are many more advantages of using PyBossa, which is why my SBTF colleagues and I are collaborating with the PyBossa team with the ultimate aim of customizing a standby platform specifically for digital humanitarian response purposes. As a first step, however, we are working together to customize a PyBossa instance for the upcoming elections in Kenya since the SBTF was activated by Ushahidi to support the election monitoring efforts. The plan is to microtask the processing of reports submitted to Ushahidi in order to significantly accelerate and scale the live mapping process. Stay tuned to iRevolution for updates on this very novel initiative.

crowdflower-crowdsourcing-site

The SBTF also made use of CrowdFlower during the response to Typhoon Pablo. Like PyBossa, CrowdFlower is a micro-tasking platform but one developed by a for-profit company and hence primarily geared towards paying workers to complete tasks. While my focus vis-a-vis digital humanitarian response has chiefly been on (integrating) automated and volunteer-driven micro-tasking solutions, I believe that paid micro-tasking platforms also have a critical role to play in our evolving digital humanitarian ecosystem. Why? CrowdFlower has an unrivaled global workforce of more than 2 million contributors along with rigor-ous quality control mechanisms.

While this solution may not scale significanlty given the costs, I’m hoping that CrowdFlower will offer the Digital Humanitarian Network (DHN) generous discounts moving forward. Either way, identifying what kinds of tasks are best completed by paid workers versus motivated volunteers is a questions we must answer to improve our digital humanitarian workflows. This explains why I plan to collaborate with CrowdFlower directly to set up a standby platform for use by members of the Digital Humanitarian Network.

There’s one major catch with all microtasking platforms, however. Without well-designed gamification features, these tools are likely to have a short shelf-life. This is true of any citizen-science project and certainly relevant to digital human-itarian response as well, which explains why I’m a big, big fan of Zooniverse. If there’s a model to follow, a holy grail to seek out, then this is it. Until we master or better yet partner with the talented folks at Zooniverse, we’ll be playing catch-up for years to come. I will do my very best to make sure that doesn’t happen.

22 Comments

Posted in Crowdsourcing, Humanitarian Technologies, Social Media

Tagged , , , , , , , , , ,

Amplifying Somali Voices Using SMS and a Live Map: #SomaliaSpeaks

Posted on December 8, 2011 | 8 comments

Somalia has been steadily slipping from global media attention over the past few months. The large scale crisis is no longer making headline news, which means that advocacy and lobbying groups are finding it increasingly difficult to place pressure on policymakers and humanitarian organizations to scale their intervention in the Horn of Africa. I recently discussed this issue with Al-jazeera’s Social Media Team whilst in Doha and pitched a project to them which has just gone live this hour.

The joint project combines the efforts of multiple partners including Al-Jazeera, Ushahidi, Souktel, Crowdflower, the African Diaspora Institute and the wider Somali Diaspora. The basis of my pitch to Al-jazeera was to let ordinary Somalis speak for themselves by using SMS to crowdsource their opinions on the unfolding crisis. My colleagues at Al-jazeera liked the idea and their editorial team proposed the following question:

Al Jazeera wants to know: how has the conflict of the last few months affected your life? Please include the name of your hometown in your response. Thank you!

So I reached out to my good friend Jacob Korenblum at Souktel. He and I had been discussing different ways we might combine our respective technologies to help in Somalia. Souktel has been working in Somalia and providing various SMS based solutions to several organizations. Jacob had previously mentioned that his team had a 50,000+ member SMS subscriber list. This proved to be key. Earlier this week, the Souktel team sent out the above question in Somali to about 5,000 of their subscribers. An effort was made to try and select geographically disbursed areas.

We’ve since received well over 2,000 text message replies and counting. In order to translate and geolocate these messages, I got in touch my colleagues Vaughn Hester and Lukas Biewald at Crowdflower in San Francisco. Crowdflower uses micro-tasking solutions to process and structure data flows. They were very keen to help and thanks to their support my Ushahidi colleagues Rob Baker and Linda Kamau were able to customize this Crowdflower plugin to translate, categorize and geo-locate incoming text messages:

 

They also wrote additional software so that text messages from Souktel could be automatically forwarded to the Crowdflower plugin which would then automatically push the processed SMS’s to a live Ushahidi map hosted by Al-jazeera. While the software development was moving forward, I connected  with colleagues from the Somali American Student Association who expressed an interest in supporting this project. Thanks to them and other members of the Somali Diaspora, hundreds of Somali voices were translated and shared on Al-jazeera’s public Ushahidi map of Somalia within days. But we still need lots of help. So if you speak Somali and English, then simply follow this link.

I wanted this project to serve as a two-way conversation, however, not just a one-way information flow from Somalia to the world. Every report  that gets mapped on an Ushahidi platform is linked to public discussion forum where readers can respond and share their views on said report. So I suggested that Al-jazeera invite their viewers/readers to comment on the text messages directly. The next step will be for Al-jazeera’s editorial team to select some of the most compelling and interesting comments and to text these back to the senders of the original text messages in Somalia. This two-way flow of information can be iterated and scaled given that the technologies and workflows are already in place.

In sum, the purpose of this project is to catalyze global media attention on Somalia by letting Somali voices take center stage—voices that are otherwise not heard in the international, mainstream media. If journalists are not going to speak about Somalia, then this project  invites Somalis speak to the world themselves. The project highlights  these voices on a live, public map for the world to bear witness and engage in a global conversation with people of Somalia, a conversation in which Somalis and the Diaspora are themselves at the centerfold. It is my sincere hope that advocacy and lobby group will be able to leverage the content generated by this project to redouble their efforts in response to the escalating crisis in Somalia.

I very much hope to see this type of approach used again in Somalia and elsewhere. It is fully inline with the motivations that inspired the launch of the first Ushahidi platform almost 4 years ago today: collective witnessing. Indeed, I am often reminded of what my friend Anand Giridharadas of the New York Times wrote last year vis-a-vis Ushahidi. To paraphrase:

They used to say that history is written by the victors. But today, before the victors win, if they win, there is a chance to scream out with a text message, a text message that will not vanish, a text message that will remain immortalized on a map for the world to bear witness. What would we know about what passed between Turks and Armenians, Germans and Jews, Hutus and Tutsis, if every one of them had had the chance, before the darkness, to declare for all time:

“I was here, and this is what happened to me”?

 Use #SomaliaSpeaks to witness the project on Twitter.

I want to specifically thank the following individuals who put an incredible amount of time and effort (most pro bono) to make this project happen: Robert Baker, Linda Kamau, Michael Moszczynski, Katie Highet, Jacob Korenblum, Vaughn Hester, Mohammed Dini, Hamza Haadoow, Andrew Jawitz and of course the excellent Al Jazeera team in Doha. Thank you all for going above and beyond to make this happen. 

8 Comments

Posted in Crisis Mapping, Crowdsourcing, Digital Activism

Tagged , , , , , , ,

Amplifying Somali Voices Using SMS and a Live Map

Posted on November 1, 2011 | 2 comments

Update: http://irevolution.net/2011/12/08/somaliaspeaks

I recently had the pleasure to meet with Al-Jazeera’s Social Media Team in Doha, Qatar. It was immediately clear that they were also interested in partnering on a joint project in Somalia when I suggested a few ideas. Several weeks later, this project is almost ready to launch. The purpose of this initiative is to let Somalis speak for themselves and to amplify those voices in the international media.

As Al-Jazeera has noted, Somalia is quickly slipping from global media attention. With Somalia out of the headline news, however, advocacy and lobbying groups will find it increasingly difficult to place pressure on policymakers and humanitarian organiza-tions to scale their intervention in this major crisis. This project therefore offers a direct and innovative way to keep Somalia in the international news. The project described below is the product of a novel collaborative effort between Al-Jazeera, Ushahidi, Souktel and Crowdflower in direct partnership with the Somali Diaspora.

The project will “interview” ordinary Somalis in Somalia and let them speak for themselves in the international media space. Interview questions drafted by Al-Jazeera will be broadcast via SMS by Souktel to 10% of their existing 50,000+ subscribers in the country. The interview questions will also invite Somalis to share in which town they are based. (Note that we are reviewing the security protocols for this). The Somali Diaspora will then translate and geolocate incoming text messages from Somali to English using a customized Crowdflower plugin. The processed messages will then be pushed (in both Somali and English) to a live Ushahidi map. Al-Jazeera will promote the live map across their main-stream and social media channels. Mapped SMS’s will each have a comments section for viewers and readers to share their thoughts. Al-Jazeera will then select the most compelling responses and text these back to the original senders in Somalia. This approach is replicable and scalable given that the partners and technologies are largely in place already.

In sum, the purpose of this project is to increase global media attention on Somalia by letting Somali voices take center stage—voices that are otherwise not heard in the international, mainstream media. If journalists are not going to speak about Somalia, then lets invite Somalis speak to the world themselves. The project will highlight these voices on a live, public map for the world to engage in a global conversation with the people of Somalia, a conversation in which Somalis and the Diaspora are themselves at the centerfold.

If you want to help out with this initiative, we’re looking for Somali-English speakers to translate and map the incoming text messages. It’s important that volunteers are familiar with the location of many cities, towns, etc., in Somalia in order to map the SMS’s. If you have the skills and time, then please add your name, email address and short bio here—would be great to have you on the team!

 

2 Comments

Posted in Crisis Mapping, Crowdsourcing, Digital Activism, Social Media

Tagged , , , , , , , , , ,

Microtasking Advocacy and Humanitarian Response in Somalia

Posted on October 26, 2011 | 1 comment

I’ve been working on bridging the gap between the technology innovation sector and the humanitarian & human rights communities for years now. One area that holds great promise is the use of microtasking for advocacy and humanitarian response. So I’d like to share two projects I’m spearheading with the support of several key colleagues. I hope these pilot projects will further demonstrate the value of mainstreaming microtasking. Both initiatives are focused on Somalia.

The first pilot project plans to leverage Souktel‘s large SMS subscriber base in Somalia to render local Somali voices and opinions more visibile in the mainstream media. This initiative combines the efforts of a Somali celebrity, members of the Somali Diaspora, a major international news organization, Ushahidi and CrowdFlower. In order to translate, categorize and geolocate incoming text messages, I reached out to my colleagues at CrowdFlower, a San Francisco-based company specializing in microtasking.

I had catalyzed a partnership with Crowdflower during the PakReport deploy-ment last year and wanted to repeat this successful collaboration for Somalia. To my delight, the team at Crowdflower was equally interested in contri-buting to this initiative. So we’ve started to customize a Crowdflower plugin for Somalia. This interface will allow members of the Somali Diaspora to use a web-based platform to translate, categorize and geolocate incoming SMS’s from the Horn of Africa. The text messages processed by the Diaspora will then be published on a public Ushahidi map.

Our international media partner will help promote this initiative and invite comments in response to the content shared via SMS. The media group will then select the most compelling replies and share these (via SMS) with the authors of the original text messages in Somalia.  The purpose of this project is to catalyze more media and world attention on Somalia, which is slowly slipping from the news. We hope that the content and resulting interaction will generate the kind of near real-time information that advocacy groups and the Diaspora can leverage in their lobbying efforts.

The second pilot project is a partnership between the Standby Volunteer Task Force (SBTF), UNHCR, DigitalGlobe and Tomnod. The purpose of this project, is to build on this earlier trial run and microtask the tagging of informal shelters in a certain region of the country to identify where IDPs are located and also esti-mate the total IDP population size. The microtasking part of this project is possible thanks to the Tomnod platform, which I’ve already blogged about in the context of this recent Syria project. The project will use a more specialized rule-set and feature-key developed with UNHCR to maximize data quality.

We are also partnering with the European Commission’s Joint Research Center (JRC) on this UNCHR project. The JRC team will run their automated shelter-detection algorithms on the same set of satellite images. The goal is to compare and triangulate crowdsource methods with automated approaches to satellite imagery analysis.

There are several advantages to using microtasking solutions for advocacy and humanitarian purposes. The first is that the tasks can easily be streamlined and distributed far and wide. Secondly, this approach to microtasking is highly scalable, rapid and easily modifiable. Finally, microtasking allows for quality control via triangulation, accountability and statistical analysis. For example, only when two volunteers translate an incoming text message from Somalia in a similar way does that text message get pushed to an Ushahidi map of local Somali voices. The same kind of triangulation can be applied to the categorization and geolocation of text messages, and indeed shelters in satellite imagery.

Microtasking is no silver bullet for advocacy and humanitarian response. But it is an important new tool in the tool box that can provide substantial support in times of crisis, especially when leveraged with other traditional approaches. I really hope the two projects described above take off. In the meantime, feel free to browse through my earlier blog posts below for further information on related applications of microtasking:

·  Combining Crowdsourced Satellite Imagery Analysis with Crisis Reporting
·  OpenStreetMap’s Microtasking Platform for Satellite Imagery Tracing
·  Crowdsourcing Satellite Imagery Analysis for Somalia
· Crowdsourcing the Analysis of Satellite Imagery for Disaster Response
· Wanted for Pakistan: A Turksourcing Plugin for Crisis Mapping
· Using Massive Multiplayer Games to Turksource Crisis Information
· From Netsourcing to Crowdsourcing to Turksourcing Crisis Information
· Using Mechanical Turk to Crowdsource Humanitarian Response


1 Comment

Posted in Crisis Mapping, Crowdsourcing, Humanitarian Technologies

Tagged , , , , , ,

Using Mechanical Turk to Crowdsource Humanitarian Response

Posted on February 5, 2010 | 17 comments

I’m increasingly intrigued by the idea of applying Mechanical Turk services to humanitarian response. Mechanical Turk was first developed by Amazon to crowdsource and pay for simple tasks.

An excellent example of a Mechanical Turk service in the field of ICT for Development (ICT4D) is txteagle, a platform that enables mobile phone subscribers in developing countries to earn money and accumulate savings by completing simple SMS-based micro-tasks for large corporate clients. txteagle has been used to translate pieces of text by splitting them into individual words and sending these out by SMS. Subscribers can then reply with the translation and earn some money in the process. This automatic compensation system uses statistical machinery to automatically evaluate the value of submitted work.

In Haiti, Samasource and Crowdflower have partnered with Ushahidi and FrontlineSMS to set up a Mechanical Turk service called “Mission 4636“. The system that Ushahidi and partners originally set up uses the generosity of Haitian volunteers in the US to translate urgent SMS’s from the disaster affected population in near real-time. Mission 4636 will relocate the translation work to Haiti and become an automatic compensation system for Haitian’s in-country.

At Ushahidi, we aggregate and  categorize urgent, actionable information from multiple sources including SMS and geo-tag this information on the Ushahidi’s interactive mapping platform. In the case of Haiti, this work is carried out by volunteers in Boston, Geneva, London and Portland coordinated by the Ushahidi-Haiti Situation Room at Tufts University. Volunteer retention is often a challenge, however. I wonder whether we an automated compensation system could be used to sustain future crisis mapping efforts.

Another challenge of crowdsourcing crisis information is tracking response. We know for a fact that a number of key responders are following our near real-time mapping efforts but knowing which reports they respond to is less than automatic. We have been able to document a number of success stories and continue to receive positive feedback from responders themselves but this information is hard to come by.

In a way, by crisis mapping actionable information in near real-time and in the public domain, we are in effect trying to crowdsource response. This, by nature, is a distributed and decentralized process, hence difficult to track. The tracking challenge is further magnified when the actors in question are relief and aid organizations responding to a large disaster. As anyone who has worked in disaster response knows, communicating who is doing what, where and when is not easy. Responders don’t have the bandwidth to document which reports they’ve responded to on Ushahidi.

This is problematic for several reasons including coordination. Organizations don’t necessarily know who is responding to what and whether this response is efficient. I wonder whether a Mechanical Turk system could be set up to crowdsource discrete response tasks based on individual organizations’ mandates. Sounds a little far out and may not be feasible, but the idea nevertheless intrigues me.

The automatic compensation system could be a public way to compensate response. Incoming SMS’s could be clustered along the UN Cluster system. The Shelter Cluster, for example, would have a dedicated website to which all shelter-related SMS’s would be pushed to. Organizations working in this space would each have access to this password protected website and tag the alerts they can and want to respond to.

In order to “cash in” following a response, a picture (or text based evidence) has to be submitted as proof, by the organization in question e.g., of new shelters being built. The number of completed responses could also be made public and individuals compelled to help, could send donations via SMS to each organization to reward and further fund the responses.

The task of evaluating the evidence of responses can also be crowdsource à la Mechanical Turk and serve as a source of revenue for beneficiaries.

For example, local Haitian subscribers to the system would receive an SMS notifying them that new shelters have been set up near Jacmel. Only subscribers in the Jacmel area would receive the SMS. They would then have a look for themselves to see whether the new shelters were in fact there and text back accordingly. Dozens of individuals could send in SMS’s to describe their observations which would further help triangulate the veracity of the evaluation à la Swift River. Note that the Diaspora could also get involved in this. And like txteagle, statistical machinery could also  be used to automatically evaluate the response and dispense the micro-compensations.

I have no doubt there are a number of other important kinks to be ironed out but I wanted to throw this out there now to get some preliminary feedback. This may ultimately not be feasible or worthwhile. But I do think that a partnership between Ushahidi and Crowdflower makes sense, not only in Haiti but for future deployments as well.

See also:

  • Digital Humanitarian Response: Moving from Crowdsourcing to Microtasking [Link]

17 Comments

Posted in Crisis Mapping, Crowdsourcing, Humanitarian Technologies

Tagged , , , ,