Tag Archives: MechanicalTurk

Combining Crowdsourced Satellite Imagery Analysis with Crisis Reporting: An Update on Syria

Members of the the Standby Volunteer Task Force (SBTF) Satellite Team are currently tagging the location of hundreds of Syrian tanks and other heavy mili-tary equipment on the Tomnod micro-tasking platform using very recent high-resolution satellite imagery provided by Digital Globe.

We’re focusing our efforts on the following three key cities in Syria as per the request of Amnesty International USA’s (AI-USA) Science for Human Rights Program.

For more background information on the project, please see the following links:

To recap, the purpose of this experimental pilot project is to determine whether satellite imagery analysis can be crowdsourced and triangulated to provide data that might help AI-USA corroborate numerous reports of human rights abuses they have been collecting from a multitude of other sources over the past few months. The point is to use the satellite tagging in combination with other data, not in isolation.
To this end, I’ve recommended that we take it one step further. The Syria Tracker Crowdmap has been operations for months. Why not launch an Ushahidiplatform that combines the triangulated features from the crowdsourced satellite imagery analysis with crowdsourced crisis reports from multiple sources?

The satellite imagery analyzed by the SBTF was taken in early September. We could grab the August and September crisis data from Syria Tracker and turn the satellite imagery analysis data into layers. For example, the “Military tag” which includes large military equipment like tanks and artillery could be uploaded to Ushahidi as a KML file. This would allow AI-USA and others to cross-reference their own reports, with those on Syria Tracker and then also place that analysis into context vis-a-vis the location of military equipment, large crowds and check-points over the same time period.

The advantage of adding these layers to an Ushahidi platform is that they could be updated and compared over time. For example, we could compare the location of Syrian tanks versus on-the-ground reports of shelling for the month of August, September, October, etc. Perhaps we could even track the repositioning of  some military equipment if we repeated this crowdsourcing initiative more frequently. Incidentally, President Eisenhower proposed this idea to the UN during the Cold War, see here.

In any case, this initiative is still very much experimental and there’s lots to learn. The SBTF Tech Team headed by Nigel McNie is looking to make the above integration happen, which I’m super excited about. I’d love to see closer integration with satellite imagery analysis data in future Ushahidi deployments that crowdsource crisis reporting from the field. Incidentally, we could scale this feature tagging approach to include hundreds if not thousands of volunteers.

In other news, my SBTF colleague Shadrock Roberts and I had a very positive conference call with UNHCR this week. The SBTF will be partnering with HCR on an official project to tag the location of informal shelters in the Afgooye corridor in the near future. Unlike our trial run from several weeks ago, we will have a far more developed and detailed rule-set & feature-key thanks to some very useful information that our colleagues at HCR have just shared with us. We’ll be adding the triangulated features from the imagery analysis to a dedicated UNHCR Ushahidi platform. We hope to run this project in October and possibly again in January so HCR can do some simple change detection using Ushahidi.

In parallel, we’re hoping to partner with the Joint Research Center (JRC), which has developed automated methods for shelter detection. Comparing crowdsourced feature tagging with an automated approach would provide yet more information to UNHCR to corroborate their assessments.

Using Massive Multiplayer Games to Playsource Crisis Information

This blog sequel follows this one on Netsourcing, Crowdsourcing and Turksourcing. This new round of thinking is inspired by a recent dinner conversation (that made it to the New York Times!) with Riley Crane, Omar Wasow, Anand Giridharadas, and Jen Brea. If the ideas below seem a little off the wall, the bottle of Kirsch I brought over for the Swiss fondue is to blame. Some parts of this post were were also inspired by The Polymath Project and conversations with Kuang Chen, Abraham Flaxman and Rob Munro during the Symposium on Artificial Intelligence for Development (AI-D) at Stanford University.

I wonder how many indoor cycling bikes exist in the world, or at least in the US. The number may in part be a function of the number of gyms. In any event, to borrow the language of Douglas Adams, the number must be “vastly, hugely, mind-bogglingly big,” especially the total number of hours spent on these bikes everyday in California alone. I’ve often wondered how much energy these machines generate when used; enough to recharge my iPhone? Hundreds of iPhones?  Why do I ask?

I’ve been thinking about the number of hours that gamers spend playing computer games in the US. Hundreds of thousands hours? I’m not quite sure what the correct order of magnitude is, but I do know the number is increasing. So how does the cycling analogy come in? Simple: how can we harness the millions of hours spent playing computer games every year to turksource crisis information? Could real world information be subtly fed into these games when necessary to process Human Intelligence Tasks (HITs) that would help tag and/or geolocate crisis information? Can we think of mobile games akin to FourSquare that could generate collective action around HITs?

In other words, what is the game equivalent of reCAPTCHA for turksourcing crisis information? Remember the computer game “Where in the World is Carmen Sandiego?” Could a cross between that type of learning game, a treasure hunt and “The DaVinci Code” get gamers hooked? I’m probably thinking way too old school here. Perhaps taking the most popular games today and subtly embed some HITs is the way to go. As mentioned in my previous blog post, one of the most time consuming and human resource intensive tasks that volunteers carried out during the first weeks of Ushahidi-Haiti was the manual, near real-time geo-location of unfolding events.

So how about adding a fun gaming user interface to OpenStreetMap? Like any good game, a user would be able to specify a difficulty level. Making a mobile UI would also come in handy when tourists are abroad and want to help geolocate. You’ve heard of eco-tourism, welcome to geo-tourism. Maybe Lonely Planet or Rough Guides could partner on such a project. Or, as happened in Haiti, geo-coding was also crowdsourced thanks to the pro-active Haitian volunteers of Mission 4636 who were far quicker at tagging than student volunteers. But what happens if the only volunteers around are not country experts or familiar with satellite imagery? Is there a way to use a Mechanical Turk Service approach to greatly simplify this geocoding process?

Maybe the day will come when kids whose parents tell them to get off their computer game to do their homework will turn around and say: “But Mom, I’m learning about the geography of Mozambique, which is what my quiz is on tomorrow, and I’m playsourcing crisis information to save lives at the same time!”

Patrick Philippe Meier

From Netsourcing to Crowdsourcing to Turksourcing Crisis Information

The near real-time crisis mapping of the disasters in Haiti and Chile using Ushahidi required a substantial number of student volunteers. These volunteers were not the proverbial crowd but rather members of pre-existing, highly-connected social networks: universities. How do we move from netsourcing to crowdsourcing and on to turksourcing?

Student volunteers from Fletcher/Tufts, SIPA/Columbia and the Graduate Institute in Geneva all represent established social networks and not an anonymous crowd. They contributed over ten thousand free hours over the past 3 months to monitor hundreds of sources on the web and map actionable information on an ongoing basis. The Core Team at Fletcher spent dozens of hours training volunteers on media monitoring and mapping.

Netsourcing presents some important advantages. Pre-existing social ties can help mobilize a trusted volunteer network. I  just sent one email to the Fletcher list-serve and because the Fletcher student body is a tight community, this eventually let do hundreds of volunteers being trained and contributing to the crisis mapping of Haiti.

The first call for volunteers

At the same time, however, netsourcing is bounded crowdsourcing. In other words, netsourcing is scale-constrained. Imagine if Wikipedia contributions had been limited to professors only—that too would be bounded crowdsourcing. So how do we move from netsourcing to crowdsourcing crisis information? How do we move from having 300 volunteers connected via existing social networks to 300,000 or even 3,000,000 anonymous volunteers?

This was one of the many questions that my colleague Riley Crane, a friend of his and I discussed for almost 4 hours over dinner. (Riley recently rose to fame when he and his team at MIT that won DARPA’s Red Balloon competition). The answer, we think, is to develop a Mechanical Turk Service plug-in for Ushahidi. I’m calling this turksourcing. The two most time-consuming tasks that volunteers labored on was media monitoring and geo-location. Both processes can be disaggregated into human intelligence tasks (HITs) combined with some automation, like Swift River. And none of this would require prior training.

This is a conversation I very much look forward to continuing with Riley and one that I also plan to bring up at Nathan Eagle‘s Symposium on Artificial Intelligence for Development (AI-D) at Stanford next Monday. There is another related conversation that I’m excited to continue—namely the use of distributed, mobile gaming as an incentive catalytic for collective action, an area that Riley has spent a lot of time thinking about.

In terms of Ushahidi, If turksourcing crisis information can be combined with gaming, users could compete for altruism points, e.g., for how many HITs they contributed to a disaster response. This could be a proxy for how “good” a person is; a kind of public social ranking score for those who opt in. I imagine having individuals include their score and ranking on their blog (much like the number of Twitter followers they have). Who knows, a high altruism score could even get you more dates on Match.com.

Patrick Philippe Meier

Using Mechanical Turk to Crowdsource Humanitarian Response

I’m increasingly intrigued by the idea of applying Mechanical Turk services to humanitarian response. Mechanical Turk was first developed by Amazon to crowdsource and pay for simple tasks.

An excellent example of a Mechanical Turk service in the field of ICT for Development (ICT4D) is txteagle, a platform that enables mobile phone subscribers in developing countries to earn money and accumulate savings by completing simple SMS-based micro-tasks for large corporate clients. txteagle has been used to translate pieces of text by splitting them into individual words and sending these out by SMS. Subscribers can then reply with the translation and earn some money in the process. This automatic compensation system uses statistical machinery to automatically evaluate the value of submitted work.

In Haiti, Samasource and Crowdflower have partnered with Ushahidi and FrontlineSMS to set up a Mechanical Turk service called “Mission 4636“. The system that Ushahidi and partners originally set up uses the generosity of Haitian volunteers in the US to translate urgent SMS’s from the disaster affected population in near real-time. Mission 4636 will relocate the translation work to Haiti and become an automatic compensation system for Haitian’s in-country.

At Ushahidi, we aggregate and  categorize urgent, actionable information from multiple sources including SMS and geo-tag this information on the Ushahidi’s interactive mapping platform. In the case of Haiti, this work is carried out by volunteers in Boston, Geneva, London and Portland coordinated by the Ushahidi-Haiti Situation Room at Tufts University. Volunteer retention is often a challenge, however. I wonder whether we an automated compensation system could be used to sustain future crisis mapping efforts.

Another challenge of crowdsourcing crisis information is tracking response. We know for a fact that a number of key responders are following our near real-time mapping efforts but knowing which reports they respond to is less than automatic. We have been able to document a number of success stories and continue to receive positive feedback from responders themselves but this information is hard to come by.

In a way, by crisis mapping actionable information in near real-time and in the public domain, we are in effect trying to crowdsource response. This, by nature, is a distributed and decentralized process, hence difficult to track. The tracking challenge is further magnified when the actors in question are relief and aid organizations responding to a large disaster. As anyone who has worked in disaster response knows, communicating who is doing what, where and when is not easy. Responders don’t have the bandwidth to document which reports they’ve responded to on Ushahidi.

This is problematic for several reasons including coordination. Organizations don’t necessarily know who is responding to what and whether this response is efficient. I wonder whether a Mechanical Turk system could be set up to crowdsource discrete response tasks based on individual organizations’ mandates. Sounds a little far out and may not be feasible, but the idea nevertheless intrigues me.

The automatic compensation system could be a public way to compensate response. Incoming SMS’s could be clustered along the UN Cluster system. The Shelter Cluster, for example, would have a dedicated website to which all shelter-related SMS’s would be pushed to. Organizations working in this space would each have access to this password protected website and tag the alerts they can and want to respond to.

In order to “cash in” following a response, a picture (or text based evidence) has to be submitted as proof, by the organization in question e.g., of new shelters being built. The number of completed responses could also be made public and individuals compelled to help, could send donations via SMS to each organization to reward and further fund the responses.

The task of evaluating the evidence of responses can also be crowdsource à la Mechanical Turk and serve as a source of revenue for beneficiaries.

For example, local Haitian subscribers to the system would receive an SMS notifying them that new shelters have been set up near Jacmel. Only subscribers in the Jacmel area would receive the SMS. They would then have a look for themselves to see whether the new shelters were in fact there and text back accordingly. Dozens of individuals could send in SMS’s to describe their observations which would further help triangulate the veracity of the evaluation à la Swift River. Note that the Diaspora could also get involved in this. And like txteagle, statistical machinery could also  be used to automatically evaluate the response and dispense the micro-compensations.

I have no doubt there are a number of other important kinks to be ironed out but I wanted to throw this out there now to get some preliminary feedback. This may ultimately not be feasible or worthwhile. But I do think that a partnership between Ushahidi and Crowdflower makes sense, not only in Haiti but for future deployments as well.

See also:

  • Digital Humanitarian Response: Moving from Crowdsourcing to Microtasking [Link]