With every new tweeted disaster comes the same old question: what is the added value of tweets for disaster response? Only a handful of data-driven studies actually bother to move the debate beyond anecdotes. It is thus high time that a meta-level empirical analysis of the existing evidence be conducted. Only then can we move towards a less superficial debate on the use of social media for disaster response and emergency management.
In her doctoral research Dr. Sarah Vieweg found that between 8% and 24% of disaster tweets she studied “contain information that provides tactical, action-able information that can aid people in making decisions, advise others on how to obtain specific information from various sources, or offer immediate post-impact help to those affected by the mass emergency.” Two of the disaster datasets that Vieweg analyzed were the Red River Floods of 2009 and 2010. The tweets from the 2010 disaster resulted in a small increase of actionable tweets (from ~8% to ~9%). Perhaps Twitter users are becoming more adept at using Twitter during crises? The lowest number of actionable tweets came from the Red River Floods of 2009, whereas the highest came from the Haiti Earthquake of 2010. Again, there is variation—this time over space.
In this separate study, over 64,000 tweets generated during Thailand’s major floods in 2011 were analyzed. The results indicate that about 39% of these tweets belonged to the “Situational Awareness and Alerts” category. “Twitter messages in this category include up-to-date situational and location-based information related to the flood such as water levels, traffic conditions and road conditions in certain areas. In addition, emergency warnings from authorities advising citizens to evacuate areas, seek shelter or take other protective measures are also included.” About 8% of all tweets (over 5,000 unique tweets) were “Requests for Assistance,” while 5% were “Requests for Information Categories.”
In this more recent study, researchers mapped flood-related tweets and found a close match between that resulting map and the official government flood map. In the map below, tweets were normalized, such that values greater than one mean more tweets than would be expected in normal Twitter traffic. “Unlike many maps of online phenomena, careful analysis and mapping of Twitter data does NOT simply mirror population densities. Instead con-centration of twitter activity (in this case tweets containing the keyword flood) seem to closely reflect the actual locations of floods and flood alerts even when we simply look at the total counts.” This also implies that a relatively high number of flood-related tweets must have contained accurate information.
Shifting from floods to fires, this earlier research analyzed some 1,700 tweets generated during Australia’s worst bushfire in history. About 65% of the tweets had “factual details,” i.e., “more than three of every five tweets had useful infor-mation.” In addition, “Almost 22% of the tweets had geographical data thus identifying location of the incident which is critical in crisis reporting.” Around 7% of the tweets were seeking information, help or answers. Finally, close to 5% (about 80 tweets) were considered “directly actionable.”
Preliminary findings from applied research that I am carrying out with my Crisis Computing team at QCRI also reveal variation in value. In one disaster dataset we studied, up to 56% of the tweets were found to be informative. But in two other datasets, we found the number of informative tweets to be very low. Meanwhile, a recent Pew Research study found that 34% of tweets during Hurricane Sandy “involved news organizations providing content, government sources offering information, people sharing their own eyewitness accounts and still more passing along information posted by others.” In addition, “fully 25% [of tweets] involved people sharing photos and videos,” thus indicating “the degree to which visuals have become a more common element of this realm.”
Finally, this recent study analyzed over 35 million tweets posted by ~8 million users based on current trending topics. From this data, the authors identified 14 major events reflected in the tweets. These included the UK riots, Libya crisis, Virginia earthquake and Hurricane Irene, for example. The authors found that “on average, 30% of the content about an event, provides situational awareness information about the event, while 14% was spam.”
So what can we conclude from these few studies? Simply that the value of tweets for disaster response can vary considerably over time and space. The debate should thus not center around whether tweets yield added value for disaster response but rather what drives this variation in value. Identifying these drivers may enable those with influence to incentivize high-value tweets.
This interesting study, “Do All Birds Tweet the Same? Characterizing Twitter Around the World,” reveals some very interesting drivers. The social network analysis (SNA) of some 5 million users and 5 billion tweets across 10 countries reveals that “users in the US give Twitter a more informative purpose, which is reflected in more globalized communities, which are more hierarchical.” The study is available here (PDF). This American penchant for posting “informative” tweets is obviously not universal. To this end, studying network typologies on Twitter may yield further insights on how certain networks can be induced—at a structural level—to post more informative tweets following major disasters.
Regardless of network typology, however, policy still has an important role to play in incentivizing high-value tweets. To be sure, if demand for such tweets is not encouraged, why would supply follow? Take the forward-thinking approach by the Government of the Philippines, for example. The government actively en-couraged users to use specific hashtags for disaster tweets days before Typhoon Pablo made landfall. To make this kind of disaster reporting via twitter more actionable, the Government could also encourage the posting of pictures and the use of a structured reporting syntax—perhaps a simplified version of the Tweak the Tweet approach. Doing so would not only provide the government with greater situational awareness, it would also facilitate self-organized disaster response initiatives.
In closing, perhaps we ought to keep in mind that even if only, say, 0.001% of the 20 million+ tweets generated during the first five days of Hurricane Sandy were actionable and only half of these were accurate, this would still mean over a thousand informative, real-time tweets, or about 15,000 words, or 25 pages of single-space, relevant, actionable and timely disaster information.
PS. While the credibility and veracity of tweets is an important and related topic of conversation, I have already written at length about this.
iRevolutions 
Patrick this is great information!. Many thanks. Can you share more reports and research about the impact or outcomes of what was deemed “actionable”, “important” and hence inferred as valuable during disasters? Because information and whether it’s informative and actionable can look quite different to different groups (large agencies/gov’t vs. NGO vs. affected communities) It would be great to have more conversations along these lines. As well as what it takes to have it moved from valuable to understanding if decisions were changed on account of it. Thanks again Patrick for these great posts!
Hi Jennifer, thanks for your kind words and for reading. I really like your point about perceptions of actionability varying based on which group is doing the perceiving. I would very much like to see more research on this as well and hope you’ll bring your expertise to bear on this question. Thanks again for reading.
Jennifer, on your impact question:
As you (more than many others) already know, there has been very little to no rigorous research on the question you pose re impact. The field of social media for emergency management is still very new–not to mention the data-driven academic research in this space. As you also know, research on impact of information vis-a-vis decision making is challenging even when said information comes from traditional humanitarian sources. For example, what exactly is the impact of the Humanitarian Dashboard on decision-making? The Dashboard has been around for years. So where are all the independent, rigorous M&E studies? Point is, evidence-based decision making is still little more than an ideal in many circles. This fact often has little to do with the value of the underlying information. See for example:
Campbell, Susanna and Patrick Meier. 2007. “Deciding to Prevent Violent Conflict: Early Warning and Decision-Making at the United Nations.” Paper prepared for the 48th Annual Convention of the International Studies Association (ISA) in Chicago. Available online. http://iRevolution.files.wordpress.com/2011/07/campbell-meier-isa-2007.pdf
My current research focuses on better understanding the value of social media for disaster response and emergency management by taking a data-driven approach. Impact is a separate question and one that I would rather leave to those who have actual professional background & experience in monitoring and evaluation (M&E), which as you well know is a field of expertise in and of itself. As I noted in my opening remarks at CrisisMappers 2012, one of the major problems with this new space (like all new fields) is the very thin evidence-base that exists. Doing rigorous, independent M&E is not only time consuming but also expensive. Hence my ask to donors for greater commitment to funding M&E projects vis-a-vis crisis mapping and humanitarian technology.
What I find amusing (for the lack of a better word) is that until very recently one couldn’t even bring up social media in the context of disaster response without being laughed at or without having that conversation be a complete non-starter. Now that we are starting to empirically demonstrate that social media can be a source of actionable information, people immediately want to see impact of said information on decision-making vis-a-vis disaster response. As Einstein once said, “Time exists so that everything doesn’t happen at the same time.” Point being it will take time (and money) to set up rigorous M&E projects to address the impact question and learn from the results. And again, we don’t even know what the impact of established information products (like the Humanitarian Dashboard) have been even though these are hardly new.
But again, one thing at a time. We first have to empirically demonstrate the added value of social media. Second, we have to develop protocols, procedures, platforms, etc, to have actionable content from social media be included in decision-making processes, and thence to evaluate how/if this new content made any difference, and if so what kind of difference. This is a challenge that will obviously take more than one person and his blog to address & resolve. So again, I really hope that you will bring your expertise and fund-raising abilities to this challenge since you already know that is a challenge.
Thanks again for reading! I look forward to your contributions.
Shared by Joyce Monsees:
The uses of Twitter during the 2010 Pakistan floods
Click to access uses-of-twitter-during-the-2010-pakistan-floods.pdf
Patrick, thanks for diving into more of the details regarding the challenges we all face in understanding decision-making and impact. I agree with so much of what you’ve said. Your work (and also the hard work of many others in our ecosystem) has helped many people take an idea forward, open up more discussions on its value, believe and measure the first stages of value and now… push forward more questions about outcomes and impact. And these hard but growing questions are a testament to the success of your work and others. I absolutely agree that understanding impact and engaging in traditional forms of evaluation face a multitude of challenges. I would like to suggest that it needs a re-envisioning of its role in learning, methodology and design especially for new information flows/communication like twitter, SMS, and (even off-line communication especially with disaster affected communities). And this is perhaps where we may depart in approach (but I hope to have more conversations about this!) Perhaps rather than one step at a time, I hope that we can leverage the ecosystem that you helped create and take these steps together. OMG it will be challenging 🙂 — time is limited, demands are high, funding is limited, and it may even get some laughs and surely a few non starters. But I believe that we will all better understand how to measure outcomes and eventually impact in this space if we begin to learn how to do it now and together. I see your expertise and that of others as essential in redesigning learning/evaluation/impact. This may break the norm of external evaluations and potentially challenge existing methodologies that define our current understanding of rigor. But I believe we need to explore this to make sure that what we are measuring, how we are analyzing, and how we attribute impact is reflecting the intended purpose or identifying intended/unintended outcomes. And most importantly help groups with different perspectives learn in real-time how to improve during the next crisis. Thanks Patrick for such great conversations!
Maybe I’m just prejudiced by my dislike of Twitter, but looking at your closing paragraph “In closing, perhaps we ought to keep in mind that even if only, say, 0.001% of the 20 million+ tweets generated during the first five days of Hurricane Sandy were actionable and only half of these were accurate, this would still mean over a thousand informative, real-time tweets,” I focus on the false positives. Let’s say we achieve a 0.1% false positive rate for identifying relevant tweets (a pretty good rate for any sort of automated process for natural language understanding/classification.) We’ll still have 200 misclassified messages for every correctly classified one with accurate information. That’s not a very encouraging signal-to-noise ratio. My enthusiasm is much stronger for approaches like the one you cited in which the Philippines gov’t encouraged a specific set of hashtags. Seems to be much more practical.
Hi David, thanks for your input. One could consider false positives as inaccurate tweets: “… and only half of these were accurate…” So assuming that 50% of all actionable tweets are either false or false positives, the remaining 50% still means “a thousand informative, real-time tweets, or about 15,000 words, or 25 pages of single-space, relevant, actionable and timely disaster information.” But I completely agree with you re the need for more structured reporting. This is already happening with the launch of the 911 SMS service in Virginia:
http://www.vagazette.com/news/dp-nws-york-911-texting-20121218-16,0,3853809.story
“When texting 911, users should include the location of the emergency in the first message and should not use text abbreviations or slang.”
Thanks again
Pingback: Social Media = Social Capital = Disaster Resilience? | iRevolution
Thank you for your analysis Patrick. In your previous writings, you’ve mentioned the importance of trust networks. Are you thinking that one way to efficiently distill the signal from the noise is to add greater weight to data from those who are trusted? I wonder if there is a way in which people who are in a trust network can designate other people whom they trust. Ideally, the network would grow and would maintain its integrity.
Hi Joe, many thanks for your comment and for reading. Yes, you’re absolutely right, a bounded crowdsourcing approach could be a huge asset:
http://iRevolution.net/2011/12/07/why-bounded-crowdsourcing
Thanks again!
Patrick, thanks for sharing your thoughts on this. I agree that empirical analysis of existing evidence be conducted, but I also feel it is important to collect data from online users, too.
In fact, I am gathering data now for my masters final project from people who used American Red Cross online social accounts during disaster response (self-selected, online survey) – Take the survey! (http://ow.ly/fC8j1). I feel you are correct in saying that social media in disaster response was, until recently, often laughed at. I have been on the receiving end of same!
While use of social media in disaster is still relatively new, the use of it gives clear indicators of how powerful a communications tool that it is.
The data coming from the survey so far seems to indicate a significant interest in citizen participation through digital reporting. It seems obvious, but the world is likely ready for campaign calling digital contributors to action; reporting from the field.
I realize this is somewhat old, but the American Red Cross held an Emergency Social Data Summit in August 2010, (http://www.scribd.com/doc/40080608/The-Path-Forward-ARC-Crisis-Data-Summit-Wrap-Up) and some key questions arose from the meeting that led to suggestions on the path forward, which was:
“The need to reach out to communities to train, build expectations, identify possible solutions, and find answers (**if possible**) in advance of a disaster.” Network-friendly code like “e911” to help route and triage reporting across diverse social networks.
I feel if varying groups could connect the dots as a group, we could do great things.
Thanks, Patrick, for helping to inspire this thinking.
Hi Julie, many thanks for your input, I really appreciate it. Very much looking forward to reading the report of your final MA project, so please do let me know when it is available as I’m sure it will be full of important insights.
That Emergency Social Data Summit was really well timed and I learned a lot from participating:
http://iRevolution.net/2010/08/14/crowd-is-always-there
Thanks again for your comment, Julie, really looking forward to learning more from you and the results of your final project. All the best!
@Joe, trust, ranking and penalties for frivolous use of a hashtag are all easy to implement, thus negating most of David’s concerns.
@Patrick, one conclusion I drew from the data you provide is that humans actively communicate their state of being and are capable of providing high value, actionable information that would have been otherwise unavailable to decision makers.
Twitter is a means to exchange this information that happened by accident. What study has gone into those situations where a public information campaign included a free SMS # that was tied to an EOC ? Of the few examples I have seen, an explicit and supported system always has favorable signal to noise ratio.
Is it time for a compilation of the best examples of designed information flow for civil benefit?
Another study worth reading (had hoped to summarize in a blog post but ran out of time):
Emergency Management, Twitter,and Social Media Evangelism
http://www.academia.edu/1071890/Emergency_Management_Twitter_and_Social_Media_Evangelism
Interesting study. My question is more practical: How valuable is Twitter to handle crisis actionable information when compared with other mediums of communication? Should governments and NGOs invest resources in parsing tweets and decide based on them, or is it more wise to invest in more capacity to answer emergency phone lines? Can mobile Internet be better used for disaster response and is Twitter a good bet?
Very good point, Osvaldo. A comparative analysis is certainly much needed. Thanks for sharing.
Pingback: Comparing the Quality of Crisis Tweets Versus 911 Emergency Calls | iRevolution
Pingback: How to Create Resilience Through Big Data | iRevolution
Pingback: Why the Public Uses Social Media During Disasters (and Why Some Don’t) | iRevolution
Pingback: A Research Framework for Next Generation Humanitarian Technology and Innovation | iRevolution
Pingback: Automatically Extracting Disaster-Relevant Information from Social Media | iRevolution
Pingback: Social Media for Emergency Management: Question of Supply and Demand | iRevolution
Pingback: How Online Gamers Can Support Disaster Response | iRevolution
Pingback: Analyzing 2 Million Disaster Tweets from Oklahoma Tornado | iRevolution
Pingback: Using Social Media to Predict Disaster Resilience | iRevolution
Pingback: AIDR: Artificial Intelligence for Disaster Response | iRevolution