Map: 24 hours of Tweets in New York

The map below depicts geo-tagged tweets posted between May 4-5, 2013 in the New York City area. Over 36,000 tweets are posted on the map (click to enlarge). Since less than 3% of all tweets are geo-tagged, the map is missing the vast majority of tweets posted in this area during those 24 hours.

New York Tweets 24 hours

Contrast the above with the 1-month worth of tweets (April-May 2013) depicted in the map below. Again, the visualization misses the vast majority of tweets since these are not geo-tagged and thus not mappable.

New York 1 Month Tweets

These visuals are screenshots of Harvard’s Tweetmap platform, which is publicly available here. My colleague Todd Mostak is one of the main drivers behind Tweetmap, so worth sending him a quick thank you tweet! Todd is working on some exciting extensions and refinements, so stay tuned as I’ll be sure to blog about them when they go live.

Bio

Data Science for 100 Resilient Cities

The Rockefeller Foundation recently launched a major international initiative called “100 Resilient Cities.” The motivation behind this global project stems from the recognition that cities are facing increasing stresses driven by the unprecedented pace urbanization. More than 75% of people expected to live in cities by 2050. The Foundation is thus rightly concerned: “As natural and man-made shocks and stresses grow in frequency, impact and scale, with the ability to ripple across systems and geographies, cities are largely unprepared to respond to, withstand, and bounce back from disasters” (1).

Resilience is the capacity to self-organize, and smart self-organization requires social capital and robust feedback loops. I’ve discussed these issues and related linkages at lengths in the posts listed below and so shan’t repeat myself here. 

  • How to Create Resilience Through Big Data [link]
  • On Technology and Building Resilient Societies [link]
  • Using Social Media to Predict Disaster Resilience [link]
  • Social Media = Social Capital = Disaster Resilience? [link]
  • Does Social Capital Drive Disaster Resilience? [link]
  • Failing Gracefully in Complex Systems: A Note on Resilience [link]

Instead, I want to make a case for community-driven “tactical resilience” aided (not controlled) by data science. I came across the term “tactical urbanism” whilst at the “The City Resilient” conference co-organized by PopTech & Rockefeller in June. Tactical urbanism refers to small and temporary projects that demonstrate what could be. We also need people-centered tactical resilience initiatives to show small-scale resilience in action and demonstrate what these could mean at scale. Data science can play an important role in formulating and implementing tactical resilience interventions and in demonstrating their resulting impact at various scales.

Ultimately, if tactical resilience projects do not increase local capacity for smart and scalable self-organization, then they may not render cities more resilient. “Smart Cities” should mean “Resilient Neighborhoods” but the former concept takes a mostly top-down approach focused on the physical layer while the latter recognizes the importance of social capital and self-organization at the neighborhood level. “Indeed, neighborhoods have an impact on a surprisingly wide variety of outcomes, including child health, high-school graduation, teen births, adult mortality, social disorder and even IQ scores” (1).

So just like IBM is driving the data science behind their Smart Cities initiatives, I believe Rockefeller’s 100 Resilient Cities grantees would benefit from similar data science support and expertise but at the tactical and neighborhood level. This explains why my team and I plan to launch a Data Science for Resilience Program at the Qatar Foundation’s Computing Research Institute (QCRI). This program will focus on providing data science support to promising “tactical resilience” projects related to Rockefeller’s 100 Resilient Cities initiative.

The initial springboard for these conversations will be the PopTech & Rockefeller Fellows Program on “Community Resilience Through Big Data and Technology”. I’m really honored and excited to have been selected as one of the PopTech and Rockefeller Fellows to explore the intersections of Big Data, Technology and Resilience. As mentioned to the organizers, one of my objectives during this two-week brainstorming session is to produce a joint set of “tactical resilience” project proposals with well articulated research questions. My plan is to select the strongest questions and make them the basis for our initial data science for resilience research at QCRI.

bio

The First Ever Spam Filter for Disaster Response

While spam filters provide additional layers of security to websites, they can also be used to process all kinds of information. Perhaps most famously, for example, the reCAPTCHA spam filter was used to transcribe the New York Times’ entire paper-based archives. See my previous blog post to learn how this was done and how spam filters can also be used to process information for disaster response. Given the positive response I received from humanitarian colleagues who read the blog post, I teamed up with my colleagues at QCRI to create the first ever spam filter for disaster response.

During international disasters, the humanitarian community (often lead by the UN’s Office for the Coordination of Humanitarian Affairs, OCHA) needs to carry out rapid damage assessments. Recently, these assessments have included the analysis of pictures shared on social media following a disaster. For example, OCHA activated the Digital Humanitarian Network (DHN) to collect and quickly tag pictures that capture evidence of damage in response to Typhoon Pablo in the Philippines (as described here and TEDx talk above). Some of these pictures, which were found on Twitter, were also geo-referenced by DHN volunteers. This enabled OCHA to create (over night) the unique damage assessment map below.

Typhon PABLO_Social_Media_Mapping-OCHA_A4_Portrait_6Dec2012

OCHA intends to activate the DHN again in future disasters to replicate this type of rapid damage assessment operation. This is where spam filters come in. The DHN often needs support to quickly tag these pictures (which may number in the tens of thousands). Adding a spam filter that requires email users to tag which image captures disaster damage not only helps OCHA and other organizations carry out a rapid damage assessment, but also increases the security of email systems at the same time. And it only takes 3 seconds to use the spam filter.

OCHA reCAPTCHA

My team and I at QCRI have thus developed a spam filter plugin that can be easily added to email login pages like OCHA’s as shown above. When the Digital Humanitarian Network requires additional hands on deck to tag pictures during disasters, this plugin can simply be switched on. My team at QCRI can easily push the images to the plugin and pull data on which images have been tagged as showing disaster damage. The process for the end user couldn’t be simpler. Enter your username and password as normal and then simply select the picture below that shows disaster damage. If there are none, then simply click on “None” and then “Login”. The spam filter uses a predictive algorithm and an existing data-base of pictures as a control mechanism to ensure that the filter cannot be gamed. On that note, feel free to test the plugin here. We’d love your feedback as we continue testing.

recpatcha2

The desired outcome? Each potential disaster picture is displayed to 3 different email account users. Only if each of the 3 users tag the same picture as capturing disaster damage does that picture get automatically forwarded to members of the Digital Humanitarian Network. To tag more pictures after logging in, users are invited to do so via MicroMappers, which launches this September in partnership with OCHA. MicroMappers enables members of the public to participate in digital disaster response efforts with a simple click of the mouse.

I would ideally like to see an innovative and forward-thinking organization like OCHA pilot the plugin for a two week feasibility test. If the results are positive and promising, then I hope OCHA and other UN agencies engaged in disaster response adopt the plugin more broadly. As mentioned in my previous blog post, the UN employs well over 40,000 people around the world. Even if “only” 10% login in one day, that’s still 4,000 images effortlessly tagged for use by OCHA and others during their disaster relief operations. Again, this plugin would only be used in response to major disasters when the most help is needed. We’ll be making the code for this plugin freely available and open source.

Please do get in touch if you’d like to invite your organization to participate in this innovative humanitarian technology project. You can support disaster response efforts around the world by simply logging into your email account, web portal, or Intranet!

bio

Crowdsourcing Life-Saving Assistance

Disaster responders cannot be everywhere at the same time, but the crowd is always there. The same is true for health care professionals such as critical care paramedics who work with an ambulance service. Paramedics cannot be posted everywhere. Can crowdsourcing help? This was the question posed to me by my colleague Mark who overseas the ambulance personnel for a major city.

graphics-ambulance-520123

Take Sudden Cardiac Arrest (SCA), for example. SCA’s account for an estimated 325,000 deaths each year in the US—one person every two minutes. Survival rates nationally are less than 8%. But Cardio-Pulmonary Resuscitation, or CPR, can sustain life until paramedics arrive by maintaining blood flow to the heart and brain. “Without oxygen-rich blood, permanent brain damage or death can occur in less than 8 minutes. After 10 minutes there is little chance of successful resuscitation. Even in modern urban settings the response times for professional rescuers commonly approach these time frames” (1). This explains why “effective bystander CPR, provided immediately after sudden cardiac arrest, can double or triple a person’s chance of survival” (2). In fact, close to 60% of adults in the US say they have taken CPR training (often due to school requirements) and 11% say they have used CPR in an actual emergency (3).

PulsePoint1

So why not develop a dedicated smartphone app to alert bystanders when someone nearby is suffering from a Sudden Cardiac Arrest? This is what Mark was getting at when we started this conversation back in April. Well it just so happens that such an app does exist. The PulsePoint mobile app “alerts CPR-trained bystanders to someone nearby having a sudden cardiac arrest that may require CPR. The app is activated by the local public safety communications center simultaneous with the dispatch of local fire and EMS resources” (4).

PulsePoint2

In sum, the purpose of the app is to increase survival rates by:

  • Reducing collapse-to-CPR times by increasing citizen awareness of cardiac events beyond a traditional “witnessed” area.
  • Reducing collapse-to-defibrillation times by increasing awareness of public access defibrillator (AED) locations through real-time mapping of nearby devices.

The PulsePoint approach is instructive to those of us applying technology to improve international humanitarian response. First, the app works within, not outside, existing institutions. When someone calls 911 to report a cardiac arrest, paramedics are still dispatched to the scene. At the same time, emergency operators use PulsePoint to alert registered bystanders in the area. Second, volunteers who receive an alert are provided with a map of nearby AEDs, i.e., additional “meta-data” important for rapid response. Third, training is key. Without CPR training, the “crowd” is not empowered to help. So Community Emergency Response Teams (CERTs) are important. Of course, not all needs require special expertise to be fulfilled, but preparedness still goes a long way.

Bio

 

Why Digital Social Capital Matters for Disaster Resilience and Response

Recent empirical studies have clearly demonstrated the importance of offline social capital for disaster resilience and response. I’ve blogged about some of this analysis here and here. Social capital is typically described as those “features of social organizations, such as networks, norms, and trust, that facilitate action and cooperation for mutual benefit.” In other words, social capital increases a group’s capacity for collective action and thus self-organization, which is a key driver of disaster resilience. What if those social organizations were virtual and the networks digital? Would these online communities “generate digital social capital”? And would this digital social capital have any impact on offline social capital, collective action and resilience?

Social Capital

A data-driven study published recently, “Social Capital and Pro-Social Behavior Online and Offline” (PDF), presents some fascinating insights. The study, carried out by Constantin M. Bosancianu, Steve Powell and Esad Bratovi, draws on their survey of 1,912 Internet users in Bosnia & Herzegovina, Croatia and Serbia. The authors specifically consider two types of social capital: bonding social capital and bridging social capital. “

“Bridging social capital is described as inclusive, fostered in networks where membership is not restricted to a particular group defined by strict racial, class, linguistic or ethnic criteria.  Regular interactions inside these networks would gradually build norms of generalized trust and reciprocity at the individual level. These relationships […] are able to offer the individual access to new information but are not very adept in providing emotional support in times of need.”

“Bonding social capital, on the other hand, is exclusive, fostered in tight-knit networks of family members and close friends. Although the degree of information redundancy in these networks is likely high (as most members occupy the same social space), they provide […] the “sociological superglue” which gets members through tough emotional stages in their lives.”

The study’s findings reveal that online and offline social capital were correlated with each other. More specifically, online bridging social capital was closely correlated with offline bridging social capital, while online binding social capital was closely correlated with offline binding social capital. Perhaps of most interest with respect to disaster resilience, the authors discovered that “offline bridging social capital can benefit from online interactions.”

bio

TEDx: Using Crowdsourcing to Verify Social Media for Disaster Response


My TEDx talk on Digital Humanitarians presented at TEDxTraverseCity. I’ve automatically forwarded the above video to the section on Big (false) Data and the use of time-critical crowdsourcing to verify social media reports shared during disasters. The talk describes the rationale behind the Verily platform that my team and I at QCRI are developing with our partners at the Masdar Institute of Technology (MIT) in Dubai. The purpose of Verily is to accelerate the process of verification by crowdsourcing evidence collection and critical thinking. See my colleague ChaTo’s excellent slide deck on Verily for more information.


bio

TEDx: Microtasking for Disaster Response

My TEDx talk on Digital Humanitarians presented at TEDxTraverseCity. I’ve automatically forwarded the above video to a short 4 minute section of the talk in which I highlight how the Digital Humanitarian Network (DHN) used micro-tasking to support the UN Office for the Coordination of Humanitarian Affairs (OCHA) in response to Typhoon Pablo in the Philippines. See this blog post to learn more about the operation. As a result of this innovative use of micro-tasking, my team and I at QCRI are collaborating with UN OCHA colleagues to launch MicroMappers—a dedicated set of microtasking apps specifically designed for disaster response. These will go live in September 2013.


bio

 

Disaster Response Plugin for Online Games

The Internet Response League (IRL) was recently launched for online gamers to participate in supporting disaster response operations. A quick introduction to IRL is available here. Humanitarian organizations are increasingly turning to online volunteers to filter through social media reports (e.g. tweets, Instagram photos) posted during disasters. Online gamers already spend millions of hours online every day and could easily volunteer some of their time to process crisis information without ever having to leave the games they’re playing.

A message like this would greet you upon logging in. (Screenshot is from World of Warcraft and has been altered)

Lets take World of Warcraft, for example. If a gamer has opted in to receive disaster alerts, they’d see screens like the one above when logging in or like the one below whilst playing a game.

In game notification should have settings so as to not annoy players. (Screenshot is from World of Warcraft and has been altered)

If a gamer accepts the invitation to join the Internet Response League, they’d see the “Disaster Tagging” screen below. There they’d tag as many pictures as wish by clicking on the level of disaster damage they see in each photo. Naturally, gamers can exit the disaster tagging area at any time to return directly to their game.

A rough concept of what the tagging screen may look like. (Screenshot is from World of Warcraft and has been altered)

Each picture would be tagged by at least 3 gamers in order to ensure the accuracy of the tagging. That is, if 3 volunteers tag the same image as “Severe”, then we can be reasonably assured that the picture does indeed show infrastructure damage. These pictures would then be sent back to IRL and shared with humanitarian organizations for rapid damage assessment analysis. There are already precedents for this type of disaster response tagging. Last year, the UN asked volunteers to tag images shared on Twitter after a devastating Typhoon hit the Philippines. More specifically, they asked them to tag images that captured the damage caused by the Typhoon. You can learn more about this humanitarian response operation here.

IRL is now looking to develop a disaster response plugin like the one described above. This way, gaming companies will have an easily embeddable plugin that they can insert into their gaming environments. For more on this plugin and the latest updates on IRL, please visit the IRL website here. We’re actively looking for feedback and welcome collaborators and partnerships.

Bio

Acknowledgements: Screenshots created by my colleague Peter Mosur who is the co-founder of the IRL.

Using Social Media to Predict Disaster Resilience (Updated)

Social media is used to monitor and predict all kinds of social, economic, political and health-related behaviors these days. Could social media also help identify more disaster resilient communities? Recent empirical research reveals that social capital is the most important driver of disaster resilience; more so than economic and material resources. To this end, might a community’s social media footprint indicate how resilience it is to disasters? After all, “when extreme events at the scale of Hurricane Sandy happen, they leave an unquestionable mark on social media activity” (1). Could that mark be one of resilience?

Twitter Heatmap Hurricane

Sentiment analysis map of tweets posted during Hurricane Sandy.
Click on image to learn more.

In the immediate aftermath of a disaster, “social ties can serve as informal insurance, providing victims with information, financial help and physical assistance” (2). This informal insurance, “or mutual assistance involves friends and neighbors providing each other with information, tools, living space, and other help” (3). At the same time, social media platforms like Twitter are increasingly used to communicate during crises. In fact, data driven research on tweets posted during disasters reveal that many tweets provide victims with information, help, tools, living space, assistance and other more. Recent studies argue that “such interactions are not necessarily of inferior quality compared to simultaneous, face-to-face interactions” (4). What’s more, “In addition to the preservation and possible improvement of existing ties, interaction through social media can foster the creation of new relations” (5). Meanwhile, and “contrary to prevailing assumptions, there is evidence that the boom in social media that connects users globally may have simultaneously increased local connections” (6).

A recent study of 5 billion tweets found that Japan, Canada, Indonesia and South Korea have highest percentage of reciprocity on Twitter (6). This is important because “Network reciprocity tells us about the degree of cohesion, trust and social capital in sociology” (7). In terms of network density, “the highest values correspond to South Korea, Netherlands and Australia.” The findings further reveal that “communities which tend to be less hierarchical and more reciprocal, also displays happier language in their content updates. In this sense countries with high conversation levels … display higher levels of happiness too” (8).

A related study found that the language used in tweets can be used to predict the subjective well-being of those users (9). The same analysis revealed that the level of happiness expressed by Twitter users in a community are correlated with members of that same community who are not on social media. Data-driven studies on happiness also show that social bonds and social activities are more conducive to happiness than financial capital (10). Social media also includes blogs. A new study analyzed more than 18.5 million blog posts found that “bloggers with lower social capital have fewer positive moods and more negative moods [as revealed by their posts] than those with higher social capital” (11).

Collectivism vs Individualism countries

Finally, another recent study analyzed more than 2.3 million twitter users and found that users in collectivist countries engage with others more than those in individualistic countries (12). “In high collectivist cultures, users tend to focus more on the community to which they belong,” while  people in individualistic countries are “in a more loosely knit social network,” and so typically “look after themselves or only after immediate family members” (13). The map above displays collectivist and individualistic countries; with the former represented by lighter shades and the latter darker colors.

In sum, one should be able to measure “digital social capital” and thus disaster resilience by analyzing social media networks before, during and after disasters. “These disaster responses may determine survival, and we can measure the likelihood of them happening” via digital social capital dynamics reflected on social media (14). One could also combine social network analysis with sentiment analysis to formulate various indexes. Anyone interested in pursuing this line of research?

bio

Why the Share Economy is Important for Disaster Response and Resilience

A unique and detailed survey funded by the Rockefeller Foundation confirms the important role that social and community bonds play vis-à-vis disaster resilience. The new study, which focuses on resilience and social capital in the wake of Hurricane Sandy, reveals how disaster-affected communities self-organized, “with reports of many people sharing access to power, food and water, and providing shelter.” This mutual aid was primarily coordinated face-to-face. This may not always be possible, however. So the “Share Economy” can also play an important role in coordinating self-help during disasters.

In a share economy, “asset owners use digital clearinghouses to capitalize the unused capacity of things they already have, and consumers rent from their peers rather than rent or buy from a company” (1). During disasters, these asset owners can use the same digital clearinghouses to offer what they have at no cost. For example, over 1,400 kindhearted New Yorkers offered free housing to people heavily affected by the hurricane. They did this using AirBnB, as shown in the short video above. Meanwhile, on the West Coast, the City of San Francisco has just lunched a partnership with BayShare, a sharing economy advocacy group in the Bay Area. The partnership’s goal is to “harness the power of sharing to ensure the best response to future disasters in San Francisco” (2).

fon wifi sharing

While share economy platforms like AirBnB are still relatively new, many believe that “the share economy is a real trend and not some small blip (3). So it may be worth taking an inventory of share platforms out there that are likely to be useful for disaster response. Here’s a short list:

  • AirBnBA global travel rental platform with accommodations in 192 countries. This service has already been used for disaster response as described above.
  • FonEnables people to share some of their home Wi-Fi  in exchange for getting free Wi-Fi from 8 million people in Fon’s network. Access to information is always key during & after disasters. The map above  displays a subset of all Fon users in that part of Europe.
  • LendingClub: A cheaper service than credit cards for borrowers. Also provides better interest rates than savings accounts for investors. Access to liquidity is often necessary after a disaster.
  • LiquidSpaceProvides high quality temporary workspaces and office rentals. These can be rented by the hour and by the day.  Dedicated spaces are key for coordinating disaster response.
  • Lyft: An is on-demand ride-sharing smartphone app for cheaper, safer rides. This service could be used to transport people and supplies following a disaster. Similar to Sidecar.
  • RelayRides:  A car sharing marketplace where participants can rent out their own cars. Like Lyft, RelayRides could be used to transport goods and people. Similar to Getaround. Also, ParkingPanda is the parking equivalent.
  • TaskRabbit: Get your deliveries and errands completed easily & quickly by trusted individuals in your neighborhood. This service could be used to run quick errands following disasters. Similar to Zaarly, a marketplace that helps you discover and hire local services. 
  • Yerdle: An “eBay” for sharing items with your friends. This could be used to provide basic supplies to disaster-affected neighborhoods. Similar to SnapGood, which also allows for temporary sharing.

Feel free to add more examples via the comments section below if you know of other sharing economy platforms that could be helpful during disasters.

While these share tools don’t necessary reinforce bonding social capital since face-to-face interactions are not required, they do stand to increase levels of bridging social capital. The former refers to social capital within existing social networks while the latter refers to “cooperative connections with people from different walks of life,” and is often considered “more valuable than ‘bonding social capital'” (3). Bridging social capital is “closely related to thin trust, as opposed to the bonding social capital of thick trust” (4). Platforms that facilitate the sharing economy provide reassurance vis-à-vis the thin trust since they tend to vet participants. This extra reassurance can go a long way during disasters and may thus facilitate mutual-aid at a distance.

 bio