Tag Archives: SBTF

Crowdsourcing Satellite Imagery Analysis for UNHCR-Somalia: Latest Results

Posted on November 9, 2011 | 10 comments


253,711

That is the total number of tags created by 168 volunteers after processing 3,909 satellite images in just five days. A quarter of a million tags in 120 hours; that’s more than 2,000 tags per hour. Wow. As mentioned in this earlier blog post, volunteers specifically tagged three different types of informal shelters to provide UNHCR with an estimate of the IDP population in the Afgooye Corridor. So what happens now?

Our colleagues at Tomnod are going to use their CrowdRank algorithm to triangulate the data. About 85% of 3,000+ images were analyzed by at least 3 volunteers. So the CrowdRank algorithm will determine which tags had the most consensus across volunteers. This built-in quality control mechanism is a distinct advantage of using micro-tasking platforms like Tomnod. The tags with the most consensus will then be pushed to a dedicated UNHCR Ushahidi platform for further analysis. This project represents an applied research & development initiative. In short, we certainly don’t have all the answers. This next phase is where the assessment and analysis begins.

In the meantime, I’ve been in touch with the EC’s Joint Research Center about running their automated shelter detection algorithm on the same set of satellite imagery. The purpose is to compare those results with the crowdsourced tags in order to improve both methodologies. Clearly, none of this would be possible without the imagery and  invaluable support from our colleagues at DigitalGlobe, so huge thanks to them.

And of course, there would be no project at all were it not for our incredible volunteers, the best “Mapsters” on the planet. Indeed, none of those 200,000+ tags would exist were it not for the combined effort between the Standby Volunteer Task Force (SBTF) and students from the American Society for Photogrammetry and Remote Sensing (ASPRS); Columbia University’s New Media Task Force (NMTF) who were joined by students from the New School; the Geography Departments at the University of Wisconsin-Madison, the University of Georgia, and George Mason University, and many other volunteers including humanitarian professionals from the United Nations and beyond.

As many already know, my colleague Shadrock Roberts played a pivotal role in this project. Shadrock is my fellow co-lead on the SBTF Satellite Team and he took the important initiative to draft the feature-key and rule-sets for this mission. He also answered numerous questions from many volunteers throughout past five days. Thank you, Shadrock!

It appears that word about this innovative project has gotten back to UNHCR’s Deputy High Commissioner, Professor Alexander Aleinikoff. Shadrock and I have just been invited to meet with him in Geneva on Monday, just before the 2011 International Conference of Crisis Mappers (ICCM 2011) kicks off. We’ll be sure to share with him how incredible this volunteer network is and we’ll definitely let all volunteers know how the meeting goes. Thanks again for being the best Mapsters around!

 

10 Comments

Posted in Crisis Mapping, Crowdsourcing, Humanitarian Technologies, Satellite Imagery

Tagged , , , , ,

Crowdsourcing Satellite Imagery Tagging to Support UNHCR in Somalia

Posted on November 1, 2011 | 10 comments

The Standby Volunteer Task Force (SBTF) recently launched a new team called the Satellite Imagery Team. This team has been activated twice within the past few months. The first was to carry out this trial run in Somalia and the second was in partnership with AI-USA for this human rights project in Syria. We’re now back in Somalia thanks to a new and promising partnership with UNHCR, DigitalGlobe, Tomnod, SBTF and Ushahidi.

The purpose of this joint project is to crowdsource the geolocation of shelters in Somalia’s Afgooye corridor. This resembles our first trial run initiative only this time we have developed formal and more specialized rule-set and feature-key in direct collaboration with our colleagues at UNHCR. As noted in this document, “Because access to the ground is difficult in Somalia, it is hard to know how many people, exactly, are affected and in what areas. By using satellite imagery to identify different types of housing/shelters, etc., we can make a better and more rapid population estimate of the number of people that live in these shelters. These estimates are important for logistics and planning purposes but are also important for understanding how the displaced population is moving and changing over time.” Hence the purpose of this project.

We’ll be tagging three different types of shelters: (1) Large permanent structures; (2) Temporary structures with a metal roof; and (3) Temporary shelters without a metal roof. Each of these shelter types is described in more details in the rule-set along with real satellite imagery examples—the feature key. The rule-set describes the shape, color, tone and clustering of the different shelter types. As per previous SBTF Satellite Team deployments, we will be using Tomnod’s excellent microtasking platform for satellite imagery analysis.

Over 100 members of the SBTF have joined the Satellite Team to support this project. One member of this team, Jamon, is an associate lecturer in the Geography Department at the University of Wisconsin-Madison. He teaches on a broad array of technologies and applications of Geographic Information Science, including GPS and  satellite imagery analysis. He got in touch today to propose offering this project for class credit to his 36 undergraduate students who he will supervise during the exercise.

In addition, my colleague and fellow Satellite Team coordinator at the SBTF, has recruited many graduate students who are members of the American Society for Photogrammetry and Remote Sensing (ASPRS) to join the SBTF team on this project. The experience that these students bring to the team will be invaluable. Shadrock has also played a pivotal role in making this project happen: thanks to his extensive expertise in remote sensing and satellite imagery, he took the lead in developing the rule-set and feature-key in collaboration with UNHCR.

The project officially launches this Friday. The triangulated results will be pushed to a dedicated UNHCR Ushahidi map for review. This will allow UNCHR to add additional contextual data to the maps for further analysis. We also hope that our colleagues at the European Commission’s Joint Research Center (JRC) will run their automated shelter tagging algorithm on the satellite imagery for comparative analysis purposes. This will help us better understand the strengths and shortcomings of both approaches and more importantly provide us with insights on how to best improve each individually and in combination.

10 Comments

Posted in Crisis Mapping, Crowdsourcing, Satellite Imagery

Tagged , , , , , ,

The Standby Volunteer Task Force: One Year On

Posted on September 25, 2011 | Leave a comment

The Standby Volunteer Task Force (SBTF) was launched exactly a year ago tomorrow and what a ride it has been! It was on September 26, 2010, that I published the blog post below to begin rallying the first volunteers to the cause.

The first blog post announcing the SBTF

Some three hundred and sixty plus days later, no fewer than 621 volunteers have joined the SBTF. These amazing individuals are based in the following sixty plus countries, including: Afghanistan, Algeria, Argentina, Armenia, Australia, Belgium, Brazil, Canada, Chile, Colombia, Czech Republic, Denmark, Egypt, Finland, France, Germany, Ghana, Greece, Guam, Guatemala, Haiti, Hungary, India, Indonesia, Iran, Ireland, Israel, Italy, Japan, Jordan, Kenya, Republic of South Korea, Lebanon, Liberia, Libya, Mexico, Morocco, Nepal, Netherlands, New Zealand, Nigeria, Pakistan, Palestine, Peru, Philippines, Poland, Portugal, Senegal, Serbia, Singapore, Slovenia, Somalia, South Africa, Spain, Sudan, Switzerland, Tajikistan, Trinidad and Tobago, Tunisia, Turkey, Uganda, United Kingdom, United States and Venezuela.

Most members have added themselves to the SBTF map below.

Between them, members of the SBTF represent several hundred organizations, including the American Red Cross, the American University in Cairo, Australia’s National University, Bertelsmann Foundation, Briceland Volunteer Fire Department, Brussels School of International Studies, Carter Center, Columbia University, Crisis Commons, Deloitte Consulting, Engineers without Borders, European Commission Joint Research Center, Fairfax County International Search & Rescue Team, Fire Department of NYC, Fletcher School, GIS Corps, Global Voices Online, Google, Government of Ontario, Grameen Development Services, Habitat for Humanity, Harvard Humanitarian Initiative, International Labor Organization, International Organization for Migration, John Carroll University, Johns Hopkins University, Lewis and Clark College, Lund University, Mercy Corps, Ministry of Agriculture and Forestry of New Zealand, Medecins Sans Frontieres, NASA, National Emergency Management Association, National Institute for Urban Search and Rescue, Nethope, New York University, OCHA, Open Geospatial Consortium, OpenStreetMap, OSCE, Pan American Health Organization, Portuguese Red Cross, Sahana Software Foundation, Save the Children, Sciences Po Paris, Skoll Foundation, School of Oriental and African Studies, Tallinn University, Tech Change, Tulane University, UC Berkeley,  UN Volunteers, UNAIDS, UNDP Bangladesh, University of Algiers, University of Colorado, University of Portsmouth, UNOPS, Ushahidi-Liberia, WHO, World Bank and Yale University.

Over the past twelve months, major SBTF deployments have included the Colombia Disaster Simulation with UN OCHA Colombia, Sudan Vote Monitor, Cyclone Yasi, Christchurch Earthquake, Libya Crisis Map and the Alabama Tornado. SBTF volunteers were also involved in other projects in Mumbai, Khartoum, Somalia and Syria with partners such as UNHCR and AI-USA. The latter two saw the establishment of a brand new SBTF team, the Satellite Imagery Team, the eleventh team to joint the SBTF Group (see figure below).  SBTF Coordinators organized and held several trainings for new members in 2011, as have our partners like the Humanitarian OpenStreetMap Team. You can learn more about all this (and join!) by visiting the SBTF blog.

We’re  grateful to have been featured in the media on several occasions over the past year, documenting how we’re changing the world, one map at a time. CNN, UK Guardian, The Economist, Fast Company, IRIN News, Washington Post, Technology Review, PBS and NPR all covered our efforts. The SBTF has also been presented at numerous conferences such as TEDxSilicon Valley, The Skoll World Forum, Re:publica, ICRC Global Communications Forum, ESRI User Conference and Share Conference. But absolutely none of this would be possible without the inspiring dedication of SBTF members and Team Coordinators.

Indeed, were it not for them, the Libya Crisis Map that we launched for UN OCHA would have looked like this (as would all the other maps):

So this digital birthday cakes goes to every SBTF member who offered their time and thereby made what this global network is today, you all know who you are and have my sincere gratitude, respect and deep admiration. SBTF Coordinators and Core Team Members deserve very special thanks and recognition for the many, many extra days and indeed weeks they have committed to the SBTF. We are also most grateful to our partners, including Ning, UN OCHA-Geneva and OCHA-Colombia for their support, camaraderie and mentorship. So a big, big thank you to all and a very happy birthday, Mapsters! I look forward to the second candle!

Leave a comment

Posted in Crisis Mapping, Crowdsourcing, Digital Activism

Tagged ,

Crowdsourcing Satellite Imagery Analysis for Somalia: Results of Trial Run

Posted on August 31, 2011 | 22 comments

We’ve just completed our very first trial run of the Standby Task Volunteer Force (SBTF) Satellite Team. As mentioned in this blog post last week, the UN approached us a couple weeks ago to explore whether basic satellite imagery analysis for Somalia could be crowdsourced using a distributed mechanical turk approach. I had actually floated the idea in this blog post during the floods in Pakistan a year earlier. In any case, a colleague at Digital Globe (DG) read my post on Somalia and said: “Lets do it.”

So I reached out to Luke Barrington at Tomnod to set up distributed micro-tasking platform for Somalia. To learn more about Tomond’s neat technology, see this previous blog post. Within just a few days we had high resolution satellite imagery from DG and a dedicated crowdsourcing platform for imagery analysis, courtesy of Tomnod . All that was missing were some willing and able “mapsters” from the SBTF to tag the location of shelters in this imagery. So I sent out an email to the group and some 50 mapsters signed up within 48 hours. We ran our pilot from August 26th to August 30th. The idea here was to see what would go wrong (and right!) and thus learn as much as we could before doing this for real in the coming weeks.

It is worth emphasizing that the purpose of this trial run (and entire exercise) is not to replicate the kind of advanced and highly-skilled satellite imagery analysis that professionals already carry out.  This is not just about Somalia over the next few weeks and months. This is about Libya, Syria, Yemen, Afghanistan, Iraq, Pakistan, North Korea, Zimbabwe, Burma, etc. Professional satellite imagery experts who have plenty of time to volunteer their skills are far and few between. Meanwhile, a staggering amount of new satellite imagery is produced  every day; millions of square kilometers’ worth according to one knowledgeable colleague.

This is a big data problem that needs mass human intervention until the software can catch up. Moreover, crowdsourcing has proven to be a workable solution in many other projects and sectors. The “crowd” can indeed scan vast volumes of satellite imagery data and tag features of interest. A number of these crowds-ourcing platforms also have built-in quality assurance mechanisms that take into account the reliability of the taggers and tags. Tomnod’s CrowdRank algorithm, for example, only validates imagery analysis if a certain number of users have tagged the same image in exactly the same way. In our case, only shelters that get tagged identically by three SBTF mapsters get their locations sent to experts for review. The point here is not to replace the experts but to take some of the easier (but time-consuming) tasks off their shoulders so they can focus on applying their skill set to the harder stuff vis-a-vis imagery interpretation and analysis.

The purpose of this initial trial run was simply to give SBTF mapsters the chance to test drive the Tomnod platform and to provide feeback both on the technology and the work flows we put together. They were asked to tag a specific type of shelter in the imagery they received via the web-based Tomnod platform:

There’s much that we would do differently in the future but that was exactly the point of the trial run. We had hoped to receive a “crash course” in satellite imagery analysis from the Satellite Sentinel Project (SSP) team but our colleagues had hardly slept in days because of some very important analysis they were doing on the Sudan. So we did the best we could on our own. We do have several satellite imagery experts on the SBTF team though, so their input throughout the process was very helpful.

Our entire work flow along with comments and feedback on the trial run is available in this open and editable Google Doc. You’ll note the pages (and pages) of comments, questions and answers. This is gold and the entire point of the trial run. We definitely welcome additional feedback on our approach from anyone with experience in satellite imagery interpretation and analysis.

The result? SBTF mapsters analyzed a whopping 3,700+ individual images and tagged more than 9,400 shelters in the green-shaded area below. Known as the “Afgooye corridor,” this area marks the road between Mogadishu and Afgooye which, due to displacement from war and famine in the past year, has become one of the largest urban areas in Somalia. [Note, all screen shots come from Tomnod].

Last year, UNHCR used “satellite imaging both to estimate how many people are living there, and to give the corridor a concrete reality. The images of the camps have led the UN’s refugee agency to estimate that the number of people living in the Afgooye Corridor is a staggering 410,000. Previous estimates, in September 2009, had put the number at 366,000” (1).

The yellow rectangles depict the 3,700+ individual images that SBTF volunteers individually analyzed for shelters: And here’s the output of 3 days’ worth of shelter tagging, 9,400+ tags:

Thanks to Tomnod’s CrowdRank algorithm, we were able to analyze consensus between mapsters and pull out the triangulated shelter locations. In total, we get 1,423 confirmed locations for the types of shelters described in our work flows. A first cursory glance at a handful (“random sample”) of these confirmed locations indicate they are spot on. As a next step, we could crowdsource (or SBTF-source, rather) the analysis of just these 1,423 images to triple check consensus. Incidentally, these 1,423 locations could easily be added to Google Earth or a password-protected Ushahidi map.

We’ve learned a lot during this trial run and Luke got really good feedback on how to improve their platform moving forward. The data collected should also help us provide targeted feedback to SBTF mapsters in the coming days so they can further refine their skills. On my end, I should have been a lot more specific and detailed on exactly what types of shelters qualified for tagging. As the Q&A section on the Google Doc shows, many mapsters weren’t exactly sure at first because my original guidelines were simply too vague. So moving forward, it’s clear that we’ll need a far more detailed “code book” with many more examples of the features to look for along with features that do not qualify. A colleague of mine suggested that we set up an interactive, online quiz that takes volunteers through a series of examples of what to tag and not to tag. Only when a volunteer answers all questions correctly do they move on to live tagging. I have no doubt whatsoever that this would significantly increase consensus in subsequent imagery analysis.

Please note: the analysis carried out in this trial run is not for humanitarian organizations or to improve situational awareness, it is simply for testing purposes only. The point was to try something new and in the process work out the kinks so when the UN is ready to provide us with official dedicated tasks we don’t have to scramble and climb the steep learning curve there and then.

In related news, the Humanitarian Open Street Map Team (HOT) provided SBTF mapsters with an introductory course on the OSM platform this past weekend. The HOT team has been working hard since the response to Haiti to develop an OSM Tasking Server that would allow them to micro-task the tracing of satellite imagery. They demo’d the platform to me last week and I’m very excited about this new tool in the OSM ecosystem. As soon as the system is ready for prime time, I’ll get access to the backend again and will write up a blog post specifically on the Tasking Server.

22 Comments

Posted in Crisis Mapping, Crowdsourcing, Humanitarian Technologies, Satellite Imagery

Tagged , , , , , , , , ,

Analyzing Satellite Imagery of the Somali Crisis Using Crowdsourcing

Posted on August 17, 2011 | 27 comments

 Update: results of satellite imagery analysis available here.

You gotta love Twitter. Just two hours after I tweeted the above—in reference to this project—a colleague of mine from the UN who just got back from the Horn of Africa called me up: “Saw your tweet, what’s going on?” The last thing I wanted to was talk about the über frustrating day I’d just had. So he said, “Hey, listen, I’ve got an idea.” He reminded me of this blog post I had written a year ago on “Crowdsourcing the Analysis of Satellite for Disaster Response” and said, “Why not try this for Somalia? We could definitely use that kind of information.” I quickly forgot about my frustrating day.

Here’s the plan. He talks to UNOSAT and Google about acquiring high-resolution satellite imagery for those geographic areas for which they need more information on. A colleague of mine in San Diego just launched his own company to develop mechanical turk & micro tasking solutions for disaster response. He takes this satellite imagery and cuts it into say 50×50 kilometers square images for micro-tasking purposes.

We then develop a web-based interface where volunteers from the Standby Volunteer Task Force (SBTF) sign in and get one high resolution 50×50 km image displayed to them at a time. For each image, they answer the question: “Are there any human shelters discernible in this picture? [Yes/No].” If yes, what would you approximate the population of that shelter to be? [1-20; 21-50; 50-100; 100+].” Additional questions could be added. Note that we’d provide them with guidelines on how to identify human shelters and estimate population figures.

No shelters discernible in this image

Each 50×50 image would get rated by at least 3 volunteers for data triangulation and quality assurance purposes. That is, if 3 volunteers each tag an image as depicting a shelter (or more than one shelter) and each of the 3 volunteers approximate the same population range, then that image would get automatically pushed to an Ushahidi map, automatically turned into a geo-tagged incident report and automatically categorized by the population estimate. One could then filter by population range on the Ushahidi map and click on those reports to see the actual image.

If satellite imagery licensing is an issue, then said images need not be pushed to the Ushahidi map. Only the report including the location of where a shelter has been spotted would be mapped along with the associated population estimate. The satellite imagery would never be released in full, only small bits and pieces of that imagery would be shared with a trusted network of SBTF volunteers. In other words, the 50×50 images could not be reconstituted and patched together because volunteers would not get contiguous 50×50 images. Moreover, volunteers would sign a code of conduct whereby they pledge not to share any of the imagery with anyone else. Because we track which volunteers see which 50×50 images, we could easily trace any leaked 50×50 image back to the volunteer responsible.

Note that for security reasons, we could make the Ushahidi map password protected and have a public version of the map with very limited spatial resolution so that the location of individual shelters would not be discernible.

I’d love to get feedback on this idea from iRevolution readers, so if you have thoughts (including constructive criticisms), please do share in the comments section below.

27 Comments

Posted in Crisis Mapping, Crowdsourcing, Humanitarian Technologies

Tagged , , , , , , ,

Passing the I’m-Not-Gaddafi Test: Authenticating Identity During Crisis Mapping Operations

Posted on June 12, 2011 | 6 comments

I’ve found myself telling this story so often in response to various questions that it really should be a blog post. The story begins with the launch of the Libya Crisis Map a few months ago at the request of the UN. After the first 10 days of deploying the live map, the UN asked us to continue for another two weeks. When I write “us” here, I mean the Standby Volunteer Task Force (SBTF), which is designed for short-term rapid crisis mapping support, not long term deploy-ments. So we needed to recruit additional volunteers to continue mapping the Libya crisis. And this is where the I’m-not-Gaddafi test comes in.

To do our live crisis mapping work, SBTF volunteers generally need password access to whatever mapping platform we happen to be using. This has typically been the Ushahidi platform. Giving out passwords to several dozen volunteers in almost as many countries requires trust. Password access means one could start sabotaging the platform, e.g., deleting reports, creating fake ones, etc. So when we began recruiting 200+ new volunteers to sustain our crisis mapping efforts in Libya, we needed a way to vet these new recruits, particularly since we were dealing with a political conflict. So we set up an I’m-not-Gaddafi test by using this Google Form:

So we placed the burden of proof on our (very patient) volunteers. Here’s a quick summary of the key items we used in our “grading” to authenticate volunteers’ identity:

Email address: Professional or academic email addresses were preferred and received a more favorable “score”.

Twitter handle: The great thing about Twitter is you can read through weeks’ worth of someone’s Twitter stream. I personally used this feature several times to determine whether any political tweets revealed a pro-Gaddafi attitude.

Facebook page: Given that posing as someone else or a fictitious person on Facebook violates their terms of service, having the link to an applicant’s Facebook page was considered a plus.

LinkedIn profile: This was a particularly useful piece of evidence given that the majority of people on LinkedIn are professionals.

Personal/Professional blog or website: This was also a great to way to authenticate an individual’s identity. We also encouraged applicants to share links to anything they had published which was available online.

For every application, we had two or more of us from the core team go through the responses. In order to sign off a new volunteer as vetted, two people had to write down “Yes” with their name. We would give priority to the most complete applications. I would say that 80% of the 200+ applications we received were able to be signed off on without requiring additional information. We did follow ups via email for the remaining 20%, the majority of whom provided us with extra info that enabled us to validate their identity. One individual even sent us a copy of his official ID. There may have been a handful who didn’t reply to our requests for additional information.

This entire vetting process appears to have worked, but it was extremely laborious and time-consuming. I personally spent hours and hours going through more than 100 applications. We definitely need to come up with a different system in the future. So I’ve been exploring some possible solutions—such as social authentication—with a number of groups and I hope to provide an update next month which will make all our lives a lot easier, not to mention give us more dedicated mapping time. There’s also the need to improve the Ushahidi platform to make it more like Wikipedia, i.e., where contributions can be tracked and logged. I think combining both approaches—identity authentication and tracking—may be the way to go.

6 Comments

Posted in Crisis Mapping, Crowdsourcing, Social Media

Tagged , , , , , , ,