I’m increasingly intrigued by the idea of applying Mechanical Turk services to humanitarian response. Mechanical Turk was first developed by Amazon to crowdsource and pay for simple tasks.
An excellent example of a Mechanical Turk service in the field of ICT for Development (ICT4D) is txteagle, a platform that enables mobile phone subscribers in developing countries to earn money and accumulate savings by completing simple SMS-based micro-tasks for large corporate clients. txteagle has been used to translate pieces of text by splitting them into individual words and sending these out by SMS. Subscribers can then reply with the translation and earn some money in the process. This automatic compensation system uses statistical machinery to automatically evaluate the value of submitted work.
In Haiti, Samasource and Crowdflower have partnered with Ushahidi and FrontlineSMS to set up a Mechanical Turk service called “Mission 4636“. The system that Ushahidi and partners originally set up uses the generosity of Haitian volunteers in the US to translate urgent SMS’s from the disaster affected population in near real-time. Mission 4636 will relocate the translation work to Haiti and become an automatic compensation system for Haitian’s in-country.
At Ushahidi, we aggregate and categorize urgent, actionable information from multiple sources including SMS and geo-tag this information on the Ushahidi’s interactive mapping platform. In the case of Haiti, this work is carried out by volunteers in Boston, Geneva, London and Portland coordinated by the Ushahidi-Haiti Situation Room at Tufts University. Volunteer retention is often a challenge, however. I wonder whether we an automated compensation system could be used to sustain future crisis mapping efforts.
Another challenge of crowdsourcing crisis information is tracking response. We know for a fact that a number of key responders are following our near real-time mapping efforts but knowing which reports they respond to is less than automatic. We have been able to document a number of success stories and continue to receive positive feedback from responders themselves but this information is hard to come by.
In a way, by crisis mapping actionable information in near real-time and in the public domain, we are in effect trying to crowdsource response. This, by nature, is a distributed and decentralized process, hence difficult to track. The tracking challenge is further magnified when the actors in question are relief and aid organizations responding to a large disaster. As anyone who has worked in disaster response knows, communicating who is doing what, where and when is not easy. Responders don’t have the bandwidth to document which reports they’ve responded to on Ushahidi.
This is problematic for several reasons including coordination. Organizations don’t necessarily know who is responding to what and whether this response is efficient. I wonder whether a Mechanical Turk system could be set up to crowdsource discrete response tasks based on individual organizations’ mandates. Sounds a little far out and may not be feasible, but the idea nevertheless intrigues me.
The automatic compensation system could be a public way to compensate response. Incoming SMS’s could be clustered along the UN Cluster system. The Shelter Cluster, for example, would have a dedicated website to which all shelter-related SMS’s would be pushed to. Organizations working in this space would each have access to this password protected website and tag the alerts they can and want to respond to.
In order to “cash in” following a response, a picture (or text based evidence) has to be submitted as proof, by the organization in question e.g., of new shelters being built. The number of completed responses could also be made public and individuals compelled to help, could send donations via SMS to each organization to reward and further fund the responses.
The task of evaluating the evidence of responses can also be crowdsource à la Mechanical Turk and serve as a source of revenue for beneficiaries.
For example, local Haitian subscribers to the system would receive an SMS notifying them that new shelters have been set up near Jacmel. Only subscribers in the Jacmel area would receive the SMS. They would then have a look for themselves to see whether the new shelters were in fact there and text back accordingly. Dozens of individuals could send in SMS’s to describe their observations which would further help triangulate the veracity of the evaluation à la Swift River. Note that the Diaspora could also get involved in this. And like txteagle, statistical machinery could also be used to automatically evaluate the response and dispense the micro-compensations.
I have no doubt there are a number of other important kinks to be ironed out but I wanted to throw this out there now to get some preliminary feedback. This may ultimately not be feasible or worthwhile. But I do think that a partnership between Ushahidi and Crowdflower makes sense, not only in Haiti but for future deployments as well.
See also:
- Digital Humanitarian Response: Moving from Crowdsourcing to Microtasking [Link]
iRevolutions 
Crowdsourcing has been an extremely helpful tool in so far as it’s been used as a community empowerment project. Part of the success is that this Mechanical Turk instance specifically targets Creole speakers, thus the community in question. Having the ability to do more than spreading awareness and/or donating money is a sea change in how we can start considering the internet as a tool for good.
I’m looking forward to seeing how it can be utilized in the rebuilding efforts to mitigate corruption.
I have some points for discussion. Number 1:
1. What level of individual user requirement information is useful to a provider of social services (in normal circumstances or in crisis)?
Bluntly, is it useful in a practical sense for thousands of hungry people to be enabled to say that they are hungry in x using technology y?
If you were running a 24-hour hospital, would you want everybody who was sick in your area to text you?
Number 2:
On the other hand, can’t the current providers of emergency relief services find a way to do better using the rich data and feedback loop new systems might provide? A generic response of “your call is valuable to us” or “be patient” on a 21st century 911 (should we coin the “4636” message?) is surely lame at best?
The rubble of Port-au-Prince could be the ground zero of radically more accountable and responsive relief operations or social service provision.
But are there examples of service providers right now that are making use of that detail and volume of data to do a better job of delivery?
Number 3:
If a 4636-type system is established to gather urgent individual needs, what ethical and practical obligations ensue from gathering that information? What is the governance and the minimum back office which prevents it being a cruel hoax? Or is it simply a matter of reputation and survival of the most effective?
Crowdflower has done a great job of helping people crowdsource tasks. Another interesting component is crowdsourcing your publishing
In this post you mentioned that you’re aggregating responses and information. That should be built directly into the publishing system you’re using. Instead of doing it offline or manually, you should have a publishing system that just allows for people to post information and then you – the editor – can highlight and curate the best
This is what we’re doing at Grogger. We’re a new publishing system that helps people curate and crowdsource content. Let me know what you think
Pingback: Digest #23 : ::: Think Macro :::
Patrick, this is a brilliant ideation. One that I think is on the bleeding edge, so don’t get bogged down in the weeds. At this point just focus on presenting the need as clearly as possible.
I don’t want to give away too much online so I’ll save the details for an offline conversation… but suffice to say I think this is brilliant.
Thanks Sean!! Looking forward to continuing the conversation!
Hmmm – how would you comment on this view: http://www.splicetoday.com/digital/the-myriad-failures-of-crowdflower
Pingback: Microtasking Advocacy and Humanitarian Response in Somalia | iRevolution
Patrick, we did a study a couple of years back to understand the feasibility of crowdsourcing ‘for good’-work on Mechanical Turk. The platform has certainly matured since then, but I think some of the take-away findings may still be of interest.
Primarily, we found that Mechanical Turk is not at all suitable if you can’t pay people sufficiently well. Paying people any amount seems to override people’s motivation to work for a good cause. Simply put, if you give a volunteer money for their task, it will become a job and if the pay-rate is too low, the money becomes demotivating and the work quality plummets. Research in psychology has come to similar conclusions.
In addition, our not-for-pay tasks completed incredibly slowly. My guess is that one of the main reasons for this is that the Mechanical Turk platform ranks tasks largely based on how well they pay. Since it’s a competitive market, if your pay rate is not high enough, it won’t show up in the search listings and nobody will find your work. In addition, I believe a for-pay market is not the best place to recruit not-for-pay volunteers.
None of these findings imply that Mechanical Turk cannot be a good platform for crowdsourcing disaster information management. However, for longer deployments it may be prohibitively expensive to pay for the work. For volunteer-based work on the other hand, the platform does not necessarily help at all with worker recruitment and the main benefit of the market is lost.
The full paper is available here: http://hci.uma.pt/~jakob/files/Rogstadius_2011_An_Assessment_of_Intrinsic_and_Extrinsic_Motivation_on_Task_Performance_in_Crowdsourcing_Markets.pdf
Many thanks, Jakob. I’ll definitely read the paper you kindly shared.
As per our conversation just now, this post was indeed published 3 years ago. The SBTF Deployment in response to Typhoon Pablo in the Philippines (December 2012) used both Mechanical Turk (via CrowdFlower) and volunteer-based microtasking (via PyBossa). CrowdFlower worked very well for simple tasks but failed for more elaborate tasks. In contrast, the volunteer-based microtasking seemed to do well on more elaborate tasks. As mentioned in this recent blog post from last month (http://iRevolution.net/2013/01/20/digital-humanitarian-micro-tasking), I’m interested in learning reading more empirical studies that analyze this balance of paid versus non-paid. So the paper you kindly shared is perfect. Both Dan Ariely and Clay Shirky have shared interesting insights from behavioral economics that also capture this intrinsic vs extrinsic question. And both indeed note that when a price is tagged to a service, task, etc., things tend to start “breaking”.
Thanks again for reading!
These two papers offer a very interesting perspective on paid vs non-paid:
http://qje.oxfordjournals.org/content/115/3/791.abstract
http://heinonline.org/HOL/LandingPage?collection=journals&handle=hein.journals/legstud29&div=7&id=&page=
It’s especially worth noting that the payment introduced by the authors had a negative effect that lasted even after the payment was removed. Thus if you ‘break’ something by introducing a price, even for experimental purposes, it may be more difficult to recruit volunteers afterwards.
Excellent, thanks for sharing. Yes, those long term effects you describe are exactly what Dan and Clay also reported.
I know this is an old comment (on an older post), but I just came across it today.
As a current full time Mechanical Turk “worker” (as well as being involved on the requester side of things for a number of projects) and someone involved in a private community of fellow workers who treat Mechanical Turk as a professional environment (to the point that all have resigned from former positions of employment elsewhere), you are 100% correct about the pay issue.
I (and those I know) am on Mechanical Turk to work – I will not even glance at tasks that are posted “for free” or what would work out to a poor hourly rate – which, when I am not working for a certain requester on their project (which pays hourly), is not under $10/hr. I cringe at some of the results requesters have received after posting their tasks at a far lower rate – from my involvement on the requester side, it’s clear their is a direct correlation between quality of work and quality of pay. Reading forums of “regular” workers, it’s clear that those willing to accept work for low hourly rates are not taking the tasks seriously – watching TV while completing the work, telling others “tricks” to either complete HITs in a way that likely skews results and those that tell outright how to scam requesters.
Several months ago a new requester popped up – the task was something I already spend hours of work on each week on a volunteer basis. The activity was something I actually enjoy. I did two HITs. The pay was atrocious. I wrote the requester stating as such and several weeks later, looking at their website with some of the results of this task, it’s clear they got what they paid for – garbage.
I attempted to “volunteer” for the current search for the missing plane yet the bitter feelings I have over the horrific pay that was being offered for previous HITs (to work the bugs out of their system, I assume) was enough to stop me from attempting to see if they had the website working again.
Volunteer crowdsourcing is great, but doing so on a platform where people are being paid to work is not the way to go about it.
Hi C Jones, many thanks for reading and for sharing your thoughtful & insightful personal reflection, I really appreciate it.
“Volunteer crowdsourcing is great, but doing so on a platform where people are being paid to work is not the way to go about it.”
Thank you for confirming this. I wasn’t sure back in 2010 when I first came across Mechanical Turk and so experimented with similar platforms. But I have long since decided to focus on the concept of citizen science as the basis for digital humanitarian volunteering. My colleagues and I are thus using a free and open source platform used for volunteer-based citizen science and have customized this to provide relief organizations with volunteer-driven microtasking support. The platform is called MicroMappers:
http://iRevolution.net/2013/09/18/micromappers
Thanks again for sharing your experience, I deeply valued reading your insights.
Like Patrick, I really appreciate the feedback. Thank you.