How to Create Resilience Through Big Data

Revised! I have edited this article several dozen times since posting the initial draft. I have also made a number of substantial changes to the flow of the article after discovering new connections, synergies and insights. In addition, I  have greatly benefited from reader feedback as well as the very rich conversa-tions that took place during the PopTech & Rockefeller workshop—a warm thank you to all participants for their important questions and feedback!


I’ve been invited by PopTech and the Rockefeller Foundation to give the opening remarks at an upcoming event on interdisciplinary dimensions of resilience, which is  being hosted at Georgetown University. This event is connected to their new program focus on “Creating Resilience Through Big Data.” I’m absolutely de-lighted to be involved and am very much looking forward to the conversations. The purpose of this blog post is to summarize the presentation I intend to give and to solicit feedback from readers. So please feel free to use the comments section below to share your thoughts. My focus is primarily on disaster resilience. Why? Because understanding how to bolster resilience to extreme events will provide insights on how to also manage less extreme events, while the converse may not be true.

Big Data Resilience


One of the guiding questions for the meeting is this: “How do you understand resilience conceptually at present?” First, discourse matters.  The term resilience is important because it focuses not on us, the development and disaster response community, but rather on local at-risk communities. While “vulnerability” and “fragility” were used in past discourse, these terms focus on the negative and seem to invoke the need for external protection, overlooking the fact that many local coping mechanisms do exist. From the perspective of this top-down approach, international organizations are the rescuers and aid does not arrive until these institutions mobilize.

In contrast, the term resilience suggests radical self-sufficiency, and self-sufficiency implies a degree of autonomy; self-dependence rather than depen-dence on an external entity that may or may not arrive, that may or may not be effective, and that may or may not stay the course. The term “antifragile” just recently introduced by Nassim Taleb also appeals to me. Antifragile sys-tems thrive on disruption. But lets stick with the term resilience as anti-fragility will be the subject of a future blog post, i.e., I first need to finish reading Nassim’s book! I personally subscribe to the following definition of resilience: the capacity for self-organization; and shall expand on this shortly.

(See the Epilogue at the end of this blog post on political versus technical defini-tions of resilience and the role of the so-called “expert”. And keep in mind that poverty, cancer, terrorism etc., are also resilient systems. Hint: we have much to learn from pernicious resilience and the organizational & collective action models that render those systems so resilient. In their book on resilience, Andrew Zolli and Ann Marie Healy note the strong similarities between Al-Qaeda & tuber-culosis, one of which are the two systems’ ability to regulate their metabolism).

Hazards vs Disasters

In the meantime, I first began to study the notion of resilience from the context of complex systems and in particular the field of ecology, which defines resilience as “the capacity of an ecosystem to respond to a perturbation or disturbance by resisting damage and recovering quickly.” Now lets unpack this notion of perturbation. There is a subtle but fundamental difference between disasters (processes) and hazards (events); a distinction that Jean-Jacques Rousseau first articulated in 1755 when Portugal was shaken by an earthquake. In a letter to Voltaire one year later, Rousseau notes that, “nature had not built [process] the houses which collapsed and suggested that Lisbon’s high population density [process] contributed to the toll” (1). In other words, natural events are hazards and exogenous while disas-ters are the result of endogenous social processes. As Rousseau added in his note to Voltaire, “an earthquake occurring in wilderness would not be important to society” (2). That is, a hazard need not turn to disaster since the latter is strictly a product or calculus of social processes (structural violence).

And so, while disasters were traditionally perceived as “sudden and short lived events, there is now a tendency to look upon disasters in African countries in particular, as continuous processes of gradual deterioration and growing vulnerability,” which has important “implications on the way the response to disasters ought to be made” (3). (Strictly speaking, the technical difference between events and processes is one of scale, both temporal and spatial, but that need not distract us here). This shift towards disasters as processes is particularly profound for the creation of resilience, not least through Big Data. To under-stand why requires a basic introduction to complex systems.

complex systems

All complex systems tend to veer towards critical change. This is explained by the process of Self-Organized Criticality (SEO). Over time, non-equilibrium systems with extended degrees of freedom and a high level of nonlinearity become in-creasingly vulnerable to collapse. Social, economic and political systems certainly qualify as complex systems. As my “alma mater” the Santa Fe Institute (SFI) notes, “The archetype of a self-organized critical system is a sand pile. Sand is slowly dropped onto a surface, forming a pile. As the pile grows, avalanches occur which carry sand from the top to the bottom of the pile” (4). That is, the sand pile becomes increasingly unstable over time.

Consider an hourglass or sand clock as an illustration of self-organized criticality. Grains of sand sifting through the narrowest point of the hourglass represent individual events or natural hazards. Over time a sand pile starts to form. How this process unfolds depends on how society chooses to manage risk. A laisser-faire attitude will result in a steeper pile. And grain of sand falling on an in-creasingly steeper pile will eventually trigger an avalanche. Disaster ensues.

Why does the avalanche occur? One might ascribe the cause of the avalanche to that one grain of sand, i.e., a single event. On the other hand, a complex systems approach to resilience would associate the avalanche with the pile’s increasing slope, a historical process which renders the structure increasingly vulnerable to falling grains. From this perspective, “all disasters are slow onset when realisti-cally and locally related to conditions of susceptibility”. A hazard event might be rapid-onset, but the disaster, requiring much more than a hazard, is a long-term process, not a one-off event. The resilience of a given system is therefore not simply dependent on the outcome of future events. Resilience is the complex product of past social, political, economic and even cultural processes.

dealing with avalanches

Scholars like Thomas Homer-Dixon argue that we are becoming increasingly prone to domino effects or cascading changes across systems, thus increasing the likelihood of total synchronous failure. “A long view of human history reveals not regular change but spasmodic, catastrophic disruptions followed by long periods of reinvention and development.” We must therefore “reduce as much as we can the force of the underlying tectonic stresses in order to lower the risk of synchro-nous failure—that is, of catastrophic collapse that cascades across boundaries between technological, social and ecological systems” (5).

Unlike the clock’s lifeless grains of sand, human beings can adapt and maximize their resilience to exogenous shocks through disaster preparedness, mitigation and adaptation—which all require political will. As a colleague of mine recently noted, “I wish it were widely spread amongst society  how important being a grain of sand can be.” Individuals can “flatten” the structure of the sand pile into a less hierarchical but more resilience system, thereby distributing and diffusing the risk and size of an avalanche. Call it distributed adaptation.

operationalizing resilience

As already, the field of ecology defines  resilience as “the capacity of an ecosystem to respond to a perturbation or disturbance by resisting damage and recovering quickly.” Using this understanding of resilience, there are at least 2 ways create more resilient “social ecosystems”:

  1. Resist damage by absorbing and dampening the perturbation.
  2. Recover quickly by bouncing back or rather forward.

Resisting Damage

So how does a society resist damage from a disaster? As hinted earlier, there is no such thing as a “natural” disaster. There are natural hazards and there are social systems. If social systems are not sufficiently resilient to absorb the impact of a natural hazard such as an earthquake, then disaster unfolds. In other words, hazards are exogenous while disasters are the result of endogenous political, economic, social and cultural processes. Indeed, “it is generally accepted among environmental geographers that there is no such thing as a natural disaster. In every phase and aspect of a disaster—causes, vulnerability, preparedness, results and response, and reconstruction—the contours of disaster and the difference between who lives and dies is to a greater or lesser extent a social calculus” (6).

So how do we apply this understanding of disasters and build more resilient communities? Focusing on people-centered early warning systems is one way to do this. In 2006, the UN’s International Strategy for Disaster Reduction (ISDR) recognized that top-down early warning systems for disaster response were increasingly ineffective. They thus called for a more bottom-up approach in the form of people-centered early warning systems. The UN ISDR’s Global Survey of Early Warning Systems (PDF), defines the purpose of people-centered early warning systems as follows:

“… to empower individuals and communities threatened by hazards to act in sufficient time and in an appropriate manner so as to reduce the possibility of personal injury, loss of life, damage to property and the environment, and loss of livelihoods.”

Information plays a central role here. Acting in sufficient time requires having timely information about (1) the hazard/s, (2) our resilience and (3) how to respond. This is where information and communication technologies (ICTs), social media and Big Data play an important role. Take the latter, for example. One reason for the considerable interest in Big Data is prediction and anomaly detection. Weather and climatic sensors provide meteorologists with the copious amounts of data necessary for the timely prediction of weather patterns and  early detection of atmospheric hazards. In other words, Big Data Analytics can be used to anticipate the falling grains of sand.

Now, predictions are often not correct. But the analysis of Big Data can also help us characterize the sand pile itself, i.e., our resilience, along with the associated trends towards self-organized criticality. Recall that complex systems tend towards instability over time (think of the hourglass above). Thanks to ICTs, social media and Big Data, we now have the opportunity to better characterize in real-time the social, economic and political processes driving our sand pile. Now, this doesn’t mean that we have a perfect picture of the road to collapse; simply that our picture is clearer than ever before in human history. In other words, we can better measure our own resilience. Think of it as the Quantified Self move-ment applied to an entirely different scale, that of societies and cities. The point is that Big Data can provide us with more real-time feedback loops than ever before. And as scholars of complex systems know, feedback loops are critical for adaptation and change. Thanks to social media, these loops also include peer-to-peer feedback loops.

An example of monitoring resilience in real-time (and potentially anticipating future changes in resilience) is the UN Global Pulse’s project on food security in Indonesia. They partnered with Crimson Hexagon to forecast food prices in Indonesia by analyzing tweets referring to the price of rice. They found an inter-esting relationship between said tweets and government statistics on food price inflation. Some have described the rise of social media as a new nervous system for the planet, capturing the pulse of our social systems. My colleagues and I at QCRI are therefore in the process of appling this approach to the study of the Arabic Twittersphere. Incidentally, this is yet another critical reason why Open Data is so important (check out the work of OpenDRI, Open Data for Resilience Initiative. See also this post on Demo-cratizing ICT for Development with DIY Innovation and Open Data). More on open data and data philanthropy in the conclusion.

Finally, new technologies can also provide guidance on how to respond. Think of Foursquare but applied to disaster response. Instead of “Break Glass in Case of Emergency,” how about “Check-In in Case of Emergency”? Numerous smart-phone apps such as Waze already provide this kind of at-a-glance, real-time situational awareness. It is only a matter of time until humanitarian organiza-tions develop disaster response apps that will enable disaster-affected commu-nities to check-in for real time guidance on what to do given their current location and level of resilience. Several disaster preparedness apps already exist. Social computing and Big Data Analytics can power these apps in real-time.

Quick Recovery

As already noted, there are at least two ways create more resilient “social eco-systems”. We just discussed the first: resisting damage by absorbing and dam-pening the perturbation.  The second way to grow more resilient societies is by enabling them to rapidly recover following a disaster.

As Manyena writes, “increasing attention is now paid to the capacity of disaster-affected communities to ‘bounce back’ or to recover with little or no external assistance following a disaster.” So what factors accelerate recovery in eco-systems in general? In ecological terms, how quickly the damaged part of an ecosystem can repair itself depends on how many feedback loops it has to the non- (or less-) damaged parts of the ecosystem(s). These feedback loops are what enable adaptation and recovery. In social ecosystems, these feedback loops can be comprised of information in addition to the transfer of tangible resources.  As some scholars have argued, a disaster is first of all “a crisis in communicating within a community—that is, a difficulty for someone to get informed and to inform other people” (7).

Improving ways for local communities to communicate internally and externally is thus an important part of building more resilient societies. Indeed, as Homer-Dixon notes, “the part of the system that has been damaged recovers by drawing resources and information from undamaged parts.” Identifying needs following a disaster and matching them to available resources is an important part of the process. Indeed, accelerating the rate of (1) identification; (2) matching and, (3) allocation, are important ways to speed up overall recovery.

This explains why ICTs, social media and Big Data are central to growing more resilient societies. They can accelerate impact evaluations and needs assessments at the local level. Population displacement following disasters poses a serious public health risk. So rapidly identifying these risks can help affected populations recover more quickly. Take the work carried out by my colleagues at Flowminder, for example. They  empirically demonstrated that mobile phone data (Big Data!) can be used to predict population displacement after major disasters. Take also this study which analyzed call dynamics to demonstrate that telecommunications data could be used to rapidly assess the impact of earthquakes. A related study showed similar results when analyzing SMS’s and building damage Haiti after the 2010 earthquake.


Resilience as Self-Organization and Emergence

Connection technologies such as mobile phones allow individual “grains of sand” in our societal “sand pile” to make necessary connections and decisions to self-organize and rapidly recover from disasters. With appropriate incentives, pre-paredness measures and policies, these local decisions can render a complex system more resilient. At the core here is behavior change and thus the importance of understanding behavior change models. Recall  also Thomas Schelling’s observation that micro-motives can lead to macro-behavior. To be sure, as Thomas Homer-Dixon rightly notes, “Resilience is an emergent property of a system—it’s not a result of any one of the system’s parts but of the synergy between all of its parts.  So as a rough and ready rule, boosting the ability of each part to take care of itself in a crisis boosts overall resilience.” (For complexity science readers, the notions of transforma-tion through phase transitions is relevant to this discussion).

In other words, “Resilience is the capacity of the affected community to self-organize, learn from and vigorously recover from adverse situations stronger than it was before” (8). This link between resilience and capacity for self-organization is very important, which explains why a recent and major evaluation of the 2010 Haiti Earthquake disaster response promotes the “attainment of self-sufficiency, rather than the ongoing dependency on standard humanitarian assistance.” Indeed, “focus groups indicated that solutions to help people help themselves were desired.”

The fact of the matter is that we are not all affected in the same way during a disaster. (Recall the distinction between hazards and disasters discussed earlier). Those of use who are less affected almost always want to help those in need. Herein lies the critical role of peer-to-peer feedback loops. To be sure, the speed at which the damaged part of an ecosystem can repair itself depends on how many feedback loops it has to the non- (or less-) damaged parts of the eco-system(s). These feedback loops are what enable adaptation and recovery.

Lastly, disaster response professionals cannot be every where at the same time. But the crowd is always there. Moreover, the vast majority of survivals following major disasters cannot be attributed to external aid. One study estimates that at most 10% of external aid contributes to saving lives. Why? Because the real first responders are the disaster-affected communities themselves, the local popula-tion. That is, the real first feedback loops are always local. This dynamic of mutual-aid facilitated by social media is certainly not new, however. My colleagues in Russia did this back in 2010 during the major forest fires that ravaged their country.

While I do have a bias towards people-centered interventions, this does not mean that I discount the importance of feedback loops to external actors such as traditional institutions and humanitarian organizations. I also don’t mean to romanticize the notion of “indigenous technical knowledge” or local coping mechanism. Some violate my own definition of human rights, for example. However, my bias stems from the fact that I am particularly interested in disaster resilience within the context of areas of limited statehood where said institutions and organizations are either absent are ineffective. But I certainly recognize the importance of scale jumping, particularly within the context of social capital and social media.


Information-based feedback loops general social capital, and the latter has been shown to improve disaster resilience and recovery. In his recent book entitled “Building Resilience: Social Capital in Post-Disaster Recovery,” Daniel Aldrich draws on both qualitative and quantitative evidence to demonstrate that “social resources, at least as much as material ones, prove to be the foundation for resilience and recovery.” His case studies suggest that social capital is more important for disaster resilience than physical and financial capital, and more important than conventional explanations. So the question that naturally follows given our interest in resilience & technology is this: can social media (which is not restricted by geography) influence social capital?

Social Capital

Building on Daniel’s research and my own direct experience in digital humani-tarian response, I argue that social media does indeed nurture social capital during disasters. “By providing norms, information, and trust, denser social networks can implement a faster recovery.” Such norms also evolve on Twitter, as does information sharing and trust building. Indeed, “social ties can serve as informal insurance, providing victims with information, financial help and physical assistance.” This informal insurance, “or mutual assistance involves friends and neighbors providing each other with information, tools, living space, and other help.” Again, this bonding is not limited to offline dynamics but occurs also within and across online social networks. Recall the sand pile analogy. Social capital facilitates the transformation of the sand pile away (temporarily) from self-organized criticality. On a related note vis-a-vis open source software, “the least important part of open source software is the code.” Indeed, more important than the code is the fact that open source fosters social ties, networks, communities and thus social capital.

(Incidentally, social capital generated during disasters is social capital that can subsequently be used to facilitate self-organization for non-violent civil resistance and vice versa).

RESILIENCE through big data

My empirical research on tweets posted during disasters clearly shows that while many use twitter (and social media more generally) to post needs during a crisis, those who are less affected in the social ecosystem will often post offers to help. So where does Big Data fit into this particular equation? When disaster strikes, access to information is equally important as access to food and water. This link between information, disaster response and aid was officially recognized by the Secretary General of the International Federation of Red Cross & Red Crescent Societies in the World Disasters Report published in 2005. Since then, disaster-affected populations have become increasingly digital thanks to the very rapid and widespread adoption of mobile technologies. Indeed, as a result of these mobile technologies, affected populations are increasingly able to source, share and generate a vast amount of information, which is completely transforming disaster response.

In other words, disaster-affected communities are increasingly becoming the source of Big (Crisis) Data during and following major disasters. There were over 20 million tweets posted during Hurricane Sandy. And when the major earth-quake and Tsunami hit Japan in early 2011, over 5,000 tweets were being posted every secondThat is 1.5 million tweets every 5 minutes. So how can Big Data Analytics create more resilience in this respect? More specifically, how can Big Data Analytics accelerate disaster recovery? Manually monitoring millions of tweets per minute is hardly feasible. This explains why I often “joke” that we need a local for rapid disaster recovery. Thanks to social computing, artifi-cial intelligence, machine learning and Big Data Analytics, we can absolutely develop a “” for rapid recovery. In fact, I’m working on just such a project with my colleagues at QCRI. We are also developing algorithms to auto-matically identify informative and actionable information shared on Twitter, for example. (Incidentally, a by-product of developing a robust for disaster response could very well be an increase in social capital).

There are several other ways that advanced computing can create disaster resilience using Big Data. One major challenge is digital humanitarian response is the verification of crowdsourced, user-generated content. Indeed, misinforma-tion and rumors can be highly damaging. If access to information is tantamount to food access as noted by the Red Cross, then misinformation is like poisoned food. But Big Data Analytics has already shed some light on how to develop potential solutions. As it turns out, non-credible disaster information shared on Twitter propagates differently than credible information, which means that the credibility of tweets could be predicted automatically.


In sum, “resilience is the critical link between disaster and development; monitoring it [in real-time] will ensure that relief efforts are supporting, and not eroding […] community capabilities” (9). While the focus of this blog post has been on disaster resilience, I believe the insights provided are equally informa-tive for less extreme events.  So I’d like to end on two major points. The first has to do with data philanthropy while the second emphasizes the critical importance of failing gracefully.

Big Data is Closed and Centralized

A considerable amount of “Big Data” is Big Closed and Centralized Data. Flow-minder’s study mentioned above draws on highly proprietary telecommunica-tions data. Facebook data, which has immense potential for humanitarian response, is also closed. The same is true of Twitter data, unless you have millions of dollars to pay for access to the full Firehose, or even Decahose. While access to the Twitter API is free, the number of tweets that can be downloaded and analyzed is limited to several thousand a day. Contrast this with the 5,000 tweets per second posted after the earthquake and Tsunami in Japan. We therefore need some serious political will from the corporate sector to engage in “data philanthropy”. Data philanthropy involves companies sharing proprietary datasets for social good. Call it Corporate Social Responsibility (CRS) for digital humanitarian response. More here on how this would work.

Failing Gracefully

Lastly, on failure. As noted, complex systems tend towards instability, i.e., self-organized criticality, which is why Homer-Dixon introduces the notion of failing gracefully. “Somehow we have to find the middle ground between dangerous rigidity and catastrophic collapse.” He adds that:

“In our organizations, social and political systems, and individual lives, we need to create the possibility for what computer programmers and disaster planners call ‘graceful’ failure. When a system fails gracefully, damage is limited, and options for recovery are preserved. Also, the part of the system that has been damaged recovers by drawing resources and information from undamaged parts.” Homer-Dixon explains that “breakdown is something that human social systems must go through to adapt successfully to changing conditions over the long term. But if we want to have any control over our direction in breakdown’s aftermath, we must keep breakdown constrained. Reducing as much as we can the force of underlying tectonic stresses helps, as does making our societies more resilient. We have to do other things too, and advance planning for breakdown is undoubtedly the most important.”

As Louis Pasteur famously noted, “Chance favors the prepared mind.” Preparing for breakdown is not defeatist or passive. Quite on the contrary, it is wise and pro-active. Our hubris—including our current infatuation with Bid Data—all too often clouds our better judgment. Like Macbeth, rarely do we seriously ask our-selves what we would do “if we should fail.” The answer “then we fail” is an option. But are we truly prepared to live with the devastating consequences of total synchronous failure?

In closing, some lingering (less rhetorical) questions:

  • How can resilience can be measured? Is there a lowest common denominator? What is the “atom” of resilience?
  • What are the triggers of resilience, creative capacity, local improvisation, regenerative capacity? Can these be monitored?
  • Where do the concepts of “lived reality” and “positive deviance” enter the conversation on resilience?
  • Is resiliency a right? Do we bear a responsibility to render systems more resilient? If so, recalling that resilience is the capacity to self-organize, do local communities have the right to self-organize? And how does this differ from democratic ideals and freedoms?
  • Recent research in social-psychology has demonstrated that mindfulness is an amplifier of resilience for individuals? How can be scaled up? Do cultures and religions play a role here?
  • Collective memory influences resilience. How can this be leveraged to catalyze more regenerative social systems?


Epilogue: Some colleagues have rightfully pointed out that resilience is ultima-tely political. I certainly share that view, which is why this point came up in recent conversations with my PopTech colleagues Andrew Zolli & Leetha Filderman. Readers of my post will also have noted my emphasis on distinguishing between hazards and disasters; that the latter are the product of social, economic and political processes. As noted in my blog post, there are no natural disastersTo this end, some academics rightly warn that “Resilience is a very technical, neutral, apolitical term. It was initially designed to characterize systems, and it doesn’t address power, equity or agency…  Also, strengthening resilience is not free—you can have some winners and some losers.”

As it turns out, I have a lot say about the political versus technical argument. First of all, this is hardly a new or original argument but nevertheless an important one. Amartya Senn discussed this issue within the context of famines decades ago, noting that famines do not take place in democracies. In 1997, Alex de Waal published his seminal book, “Famine Crimes: Politics and the Disaster Relief In-dustry in Africa.” As he rightly notes, “Fighting famine is both a technical and political challenge.” Unfortunately, “one universal tendency stands out: technical solutions are promoted at the expense of political ones.” There is also a tendency to overlook the politics of technical actions, muddle or cover political actions with technical ones, or worse, to use technical measures as an excuse not to undertake needed political action.

De Waal argues that the use of the term “governance” was “an attempt to avoid making the political critique too explicit, and to enable a focus on specific technical aspects of government.” In some evaluations of development and humanitarian projects, “a caveat is sometimes inserted stating that politics lies beyond the scope of this study.” To this end, “there is often a weak call for ‘political will’ to bridge the gap between knowledge of technical measures and action to implement them.” As de Waal rightly notes, “the problem is not a ‘missing link’ but rather an entire political tradition, one manifestation of which is contemporary international humanitarianism.” In sum, “technical ‘solutions’ must be seen in the political context, and politics itself in the light of the domi-nance of a technocratic approach to problems such as famine.”

From a paper I presented back in 2007: “the technological approach almost always serves those who seek control from a distance.” As a result of this technological drive for pole position, a related “concern exists due to the separation of risk evaluation and risk reduction between science and political decision” so that which is inherently politically complex becomes depoliticized and mechanized. In Toward a Rational Society (1970), the German philosopher Jürgen Habermas describes “the colonization of the public sphere through the use of instrumental technical rationality. In this sphere, complex social problems are reduced to technical questions, effectively removing the plurality of contending perspectives.”

To be sure, Western science tends to pose the question “How?” as opposed to “Why?”What happens then is that “early warning systems tend to be largely conceived as hazard-focused, linear, topdown, expert driven systems, with little or no engagement of end-users or their representatives.” As De Waal rightly notes, “the technical sophistication of early warning systems is offset by a major flaw: response cannot be enforced by the populace. The early warning information is not normally made public.”  In other words, disaster prevention requires “not merely identifying causes and testing policy instruments but building a [social and] political movement” since “the framework for response is inherently political, and the task of advocacy for such response cannot be separated from the analytical tasks of warning.”

Recall my emphasis on people-centered early warning above and the definition of resilience as capacity for self-organization. Self-organization is political. Hence my efforts to promote greater linkages between the fields of nonviolent action and early warning years ago. I have a paper (dated 2008) specifically on this topic should anyone care to read. Anyone who has read my doctoral dissertation will also know that I have long been interested in the impact of technology on the balance of power in political contexts. A relevant summary is available here. Now, why did I not include all this in the main body of my blog post? Because this updated section already runs over 1,000 words.

In closing, I disagree with the over-used criticism that resilience is reactive and about returning to initial conditions. Why would we want to be reactive or return to initial conditions if the latter state contributed to the subsequent disaster we are recovering from? When my colleague Andrew Zolli talks about resilience, he talks about “bouncing forward”, not bouncing back. This is also true of Nassim Taleb’s term antifragility, the ability to thrive on disruption. As Homer-Dixon also notes, preparing to fail gracefully is hardly reactive either.

35 responses to “How to Create Resilience Through Big Data

  1. Conceptually, what you seem to be considering is something analogous to a stress test. Institutions and communities are stressed by unexpected and significant (possibly conceived in different ways) events. Even slowly evolving processes, such as global warming, rise to greater consciousness by events, such as superstores. Latent dangers, such as the vulnerability of nuclear plants to a Tsunami is always there, but not felt until an event crystalizes the danger. Humans and human institutions are, in this respect, rather obtuse. The political science literature has a deep and long-term interest in this apparent reality. Cobb and Elder way back in the 1970s wrote of triggering events. A closely related literature speaks of focusing events. And Baumgartner and Jones have even offered a model of policy change that emphasizes the central role of stasis until an “unexpected” event offers a point of disruption. They call it a punctuated Equilibrium model. See Bryan Jones and Frank Baumgartner, Agendas and Instability in American Politics. On focusing events, see Birkland, But at another level political scientists have called into question the very conceptual category “crisis.” Murray Edelman raised this point in Constructing the Political Spectacle. His point is quite interesting and a bit subversive. I can’t do it justice here, but in short he asks us to think about when a crisis is so understood. Not all social conditions that could be understood as a crisis actually achieve that status. On the other hand, some crisis (such as WMD in Iraq in 2002-03) had no empirical footing, and we knew that at the time. Reality, in this respect, is socially constructed. Some things are kept off the crisis agenda while other things are included, without respect to the actual harm done by the social condition at hand. Problems and crisis have the status they do according to whose interests are served, and not necessarily tied to actual harm. Stepping back for a minute, focusing events can disrupt this. Three Mile Island was a focusing event that disrupted the stasis surrounding nuclear energy.

    It seems to me that the power of ubiquitous or crowdsourced platforms is seen in the new found ability to call out focusing events that would, in Edelman’s view and the view of Baumgartner and Jones, be left out of consciousness according to the interests of dominant actors in a particular system bound by stasis. There are fewer — or at least different — gatekeepers.

    What does this have to do with resilience? Maybe nothing. But if it does it probably has a lot to do with a rethinking of what it means to be resilient. Do we want resilient systems if they are dis-functional and reflective of a particular moment in a policy triangle, as Baumgartner and Jones discuss? There is a danger in fixating on resilience if it is confused with a resistance to change.

    Secondly, your topic is not dissimilar to our conversations about governance in areas of limited statehood. This is an entirely separate point from the one I’ve just tried to make. Consolidated states are, by definition, resilient. State institutions in areas of limited statehood are, by definition, not resilient. This is a service delivery model of resilient. But as we also said in our Berlin meetings, even consolidated states can experience momentary lapses of governance capacity. We saw this famously with Katrina, and to a lesser extent with Sandy. And we saw it with the earthquake in Japan. The US and Japan are, for the most part, consolidated states. But both were overwhelmed by the enormity of the circumstances at hand.

    Two thought, random and flow of consciousness as they are. I hope it triggers something for you. I’ve be happy to discuss more over a beer.

    • Wow, many thanks for your detailed feedback, Steven, really appreciate it!

      Yes, I read up on punctuated equilibria models while at the Santa Fe Institute and have seen them applied in a number of systems, ie, not just physical but social systems, etc.

      And certainly agreed re conceptual category of crisis. Many humanitarians also challenge this. Andrew Zolli and I were just talking about the problems of binary thinking re crises and resilience. I’ll definitely have to look into Edelman’s work, many thanks for the tip. The categorization of a situation as a crisis is a political decision, right? Hence the interesting impact of social media, which has been likened to a new “nervous system” of our planet; sousveillance, hidden/public transcripts, Seeing Like a State, and so on. So yes, totally totally agreed with what you write: “It seems to me that the power of ubiquitous or crowdsourced platforms is seen in the new found ability to call out focusing events that would, in Edelman’s view and the view of Baumgartner and Jones, be left out of consciousness according to the interests of dominant actors in a particular system bound by stasis. There are fewer — or at least different — gatekeepers.” Excellent stuff.

      Yes, many parallels to our conversations in Berlin about areas of limited statehood!

      Thanks again for taking the time to share your feedback, Steven. Very much looking forward to continuing our conversations in person in coming months!

  2. Pingback: Disaster Resiliency in a Complex World « DISASTER GESTALT

  3. Hi Patrick: Nice job on this posting. I did not seen any mention of the recent work of the National Academy of Sciences on resilience.
    Their report is titled Disaster Resilience; a National Imperative.

    • Thanks very much, Claire. I have been meaning to read this report for a while. It is a tad on the long side but will do my best to at least browse through in coming weeks. Thanks for the reminder!

  4. There is much to consider and comment upon in your well-articulated post. Two things stick in my mind. First, is that there is I think much to say for the idea that a major component of resilience is as much about the planning for failure as it is about trying to prevent failure.. For I believe, if you accept self-organized criticality (SOC), so long as complexity continues to grow (and it will), failure at some point is inevitable. And in a set of nested SOC systems (and how many different systems can we imagine are interacting with one another as we move from the level of neighborhoods, to towns, to cities, to states, to regions, to countries, to international relations??) the disaster is both result of the grain of sand falling (to use the classic SOC metaphor), it becomes a grain of sand falling into another SOC system. And the problem is that once all the systems reach criticality, it is almost impossible to predict which grain of sand becomes a disaster of limited scope and which grain causes the cascading. “synchronous failure” of all the connected systems that brings down economies, governments, and even nations. It can even bring conflict and war. By planning and preparing for what will be done when the worst happens, rather than ignore the possibility or believing that our solutions will prevent the worst, such plans might be an essential element in limiting the possible cascading effects of a disaster.

    Regarding the political will and relationship of politics to resiliency, I wonder if it is possible that in some places these political systems are reaching their critical state, if they have not already? I am beginning to wonder if this isn’t the case with the U.S.. It seems the ability of government to come to some productive agreement on complex problems is rapidly diminishing; and increasingly cities have become reliant on states, and the states have become reliant on the federal government to plan for, respond to, and recover from, disasters. This is not a way to build resiliency. The lessons of Katrina may not have really been learned. If something does not change in the future, there is another grain of sand out there with our country’s name on it, and it might bring a small avalanche, or it might bring down the whole sand pile.

    • Hi Joseph,

      Many thanks for your feedback, I really appreciate it. Yes, I completely agree: planning for failure & prevention. But the notion of failing gracefully is a preventive action, it is a way to manage failure so as to mitigate the worst impact. On nested SOC’s, yes absolutely. That is what Homer-Dixon refers to as synchronous failure. In sum, yes, preparedness, preparedness, preparedness.

      Very interesting point re political systems reaching their critical state. I think that is spot on. Political systems are no longer (if they ever were) able to manage the growing complexity of the social, economic, environmental, etc systems. Complexity science is particularly insightful in this respect, as is organizational theory and the study of networks versus hierarchies. As these political systems veer towards instability, I’m hoping that bottom-up, grassroots initiatives will increasingly fill the void and provide an alternative to top-down institutional control. On this note, I highly recommend reading an upcoming book called “Areas of Limited Statehood” to be published by Oxford University Press this year.

      Thanks again!

  5. Hi Patrick, I’m still unpacking your long and well thought out post, but just a couple of thoughts: I don’t know that I buy the criticism of “fragile” over “at risk”, although I agree that invokes the need for external protection in perhaps a condecending say. I think both fragile and at risk communities pretty much imply the same thing – lack of resiliency is clearly a characteristic.

    Regarding the use of complex systems as a lens, you might want to dig a little deeper on that and instead focus on developing negative feedback systems. If you wanted to use an ecological lens for that, Howard Odum, I believe was the first to really dig into this from a systems thinking perspective. In any event, negative feedback systems negate change from an initial goal state – purturbations are eventually brought back into “homeostatis”. The question is whether the disaster, which it hits at risk or fragile communities, exceeds the threshold of the negative feedback system, which will cause the sytem functions to break down (the river dies from too much pollution in an ecology analogy). In this case, you get positive feedback loops (increasing change from an initial state – the disaster), which cause the community or society to spin out of control into a destabilized situation.

    So I would pose that the question really is how to create more resiliant negative feedback structures that can withstand risks brought on by disasters. The evidence for a realistic planning process would probably involve both systems dynamic models to test assumptions on what a robust negative feedback system would be as related to a flood, earthquake, or whatnot, and a risk assessment (probability vs impact type matrices) backed up by montecarlo simulations or some such to get a good sense of the potential event.

    Regarding the big data aspect, I’m still wrapping my head around this in terms of how it fits in, but clearly at a minimum this may drive both the situational awareness and real-time response mechanisms.

  6. Apologies – in first paragraph I mean to add “vulnerable”, as in, “…although I agree that vulnerable invokes the need for external protection…”

  7. Patrick, this post is a treasure chest of ideas!  Thanks!

    I’m getting through the Antifragile as well- and one concept I am trying to wrap my head around is that all things antifragile actually benefit from stress and shocks. This is different than resilience, as a definition of resilience (oversimplifying here) is that business operations are maintained in face of shocks. With antifragility, you not only maintain operations but benefit (post traumatic stress vs post traumatic growth). From the development perspective, what I am trying to figure out is how to translate this definition into something concrete that we can DO with the communities that we work with. I’ll give you a concrete example. I am currently on vacation in a town in the northern Montenegro that lived through the biggest snow storm in the country’s history last year- entire country was paralyzed (literally) for 7 days, while this place ended up under snow for weeks after the rest of the country continued life as usual. What I noticed this year is that the houses look differently… the architecture has changed (in ONE year), people are building add ons on their roofs, their windows and garages, in anticipation of the similar mayham this year. Over coffees, people are quick to admit that if they were to build their home today (as opposed to years ago) they would do it completely differently. So one idea that I am playing with in my head, in order to make this place antifragile, is to maybe pair communities with architecture guys who could help with some low hanging fruits/low cost improvements in terms of remodeling their homes and barns, which on one side could help stimulate some economic activity and on the other help make the next storm less of a disaster.

    In any case, I found your post fantastic. Hope your presentation will be filmed so those of us who wont attend the event could watch it!  best, millie

    • Hi Millie, many thanks for your kind feedback!

      I completely share your struggle in operationalizing antifragility in the context of at-risk communities. Your observations about norther Montenegro are fascinating, particularly the insightful link you make with architecture. This too has been in my head quite a bit, i.e., the connection with urban planning. If you have time, I highly recommend Jeff Speck’s “Walkable Cities”. I was actually going to cite some of that book in this blog post but “ran out of space”. In any case, I’d love to continue thinking through these issues with you. We must find a way to finally do this in person soon!

      Thanks again for your feedback! And Happy Holidays!

  8. Patrick, thanks for the book recommendation! 🙂 Will definitely look it up,and i agree- one of these days, will have to do this face to face!

  9. patrick — maybe the discussion here has run its course and my comment is too off-track, but i’d like to note two puzzlements anyway:

    regarding resilience: a decade or two ago, the buzz about building strong networks was that they should be robust and resilient, in part by having redundancies built in. despite overlaps, these were three different concepts. since then, resilience has become the main buzz. the other two concepts — robustness and redundancy — have either slipped aside, or been subordinated to resilience. at the same time, many writings i see about enhancing resilience nowadays are less about adding robustness than about finding innovative ways to assure multi-use adaptations and substitutabilities. that’s a broader concept of resilience, and it may well be preferable. yet, I remain puzzled at the evolution of this language.

    regarding self-organization: what is said to be self-organized is often really other-organized in some critical way. as you note, it’s frequently said that “The archetype of a self-organized critical system is a sand pile.” but I continue to doubt this, for sand piles are organized mainly by gravity — an outside force. and something similar applies to many groups that are said to be self-organized or self-organizing. they may well be so, to a degree. yet their postures and actions may also be shaped critically by the gravity of their situation, the context in which they are embedded, etc. — again outside factors that spell other-organization. it always sounds good to be self-organizing, but it may also mislead analysis. and why sand piles are deemed such an archetype remains a puzzle to me (but perhaps I’m missing something?).

    anyway, another interesting post. – onward, david

  10. Pingback: Big Data for Development: From Information to Knowledge Societies? | iRevolution

  11. Pingback: Crisis Mapping, Neogeography and the Delusion of Democratization | iRevolution

  12. Pingback: Resilience = Anarchism = Resilience? | iRevolution

  13. Pingback: Humanitarianism in the Network Age: Groundbreaking Study | iRevolution

  14. Pingback: Big Data for Conflict Prevention | iRevolution

  15. Pingback: Dependance a factor of weakening | Marcus' s Space

  16. Terry Considine

    For me, resilience is the ‘stored potential’ which is intrinsic to that which aims to withstand [individual/community/system/process etc], respond positively to crises, having the capacity and capability to adapt and realign if and when a hazard or threat impacts. A key word here is capacity. As has been discussed, the more and more tightly coupled systems become, the less redundancy lies within thereby, removing or reducing the ‘stored potential’ required to be resilient.

  17. Pingback: Using Waze, Uber, AirBnB and SeeClickFix for Disaster Response | iRevolution

  18. Pingback: Data Science for 100 Resilient Cities | iRevolution

  19. Thank you very much for sharing this post and all the links that you highlighted.
    I am doing a PhD in resilience of complex adaptive systems, and I think you put too much effort to define resilience.
    Why should we care what resilience mean? One might say that because we want to build resilient systems. This is a valid point. But I don’t think by defining something explicitly we have more chance of building resilient communities or systems.
    In your post: resilience is seen from the emergence point of view. Well, from complexity science, emergence is emergence, no way to measure it. It emerges. You can see it but you cannot measure it and tell exactly because point A and Point B has the capacity to absorb change, then the system was resilient.
    Take for example a person who has suffered from a traumatic event: Can you ask him/her what was the single atom of his experience when he/she was experiencing the “impact” of the traumatic event that led him toward recovery?
    From my reading and analysis, resilience cannot be measured in a linear way. In your post you have mentioned about the idea of This is a good way of seeing resilience for as you said there are “at least two ways of increasing resilience”. Which one matches your system, in a particular time and space and context? If resilience is about self-sufficiency then it should be political too. So, in reality the point is about the “self” part of self-organisation rather than the organisation part. In natural systems self-organisation is a form of defensive strategy. However, when it comes to criticality, for social systems, the question is how much is critical in order for change to happen and whether that change is in favour of the political system of a system and its connected nodes. Given that we have the ability to learn and adapt, then one can modify the flow of information and knowledge to control “criticality” hence delay or disrupt internal activation of processes in systems that are important for manifestation of resilience. Also, what about the capacity of social system to handle particular form of information in time of need. I while think people-centric processes and bottom approaches are essential for resilience to happen, still think that we are in exploration phase of the true capabilities of information technology. And finally, sand pile model is an an archetype that helps understanding of the process of SOC not the SOC itself. I don’t think the sand pile analogy is a good way of describing SOC and Self-organisation in human systems for a simple existence of our brain as the most complex system in the universe. If a grain of sand had the opportunity of having a human brain, would the pile still exist or the grain simple could make conscious decisions about its desired positioning based on its experience of being surrounded by other grains of sands?
    Finally, resilience seems more like a movement rather than a thing to measure. It is a form of mental shift toward living a happier life in harmony with nature and utilise our enormous capacity for creativity, compassion and simply for the unique experience of being human.

  20. I agree with those who have appreciated how well articulated the initial thoughts are both describing resiliency conceptually and relating to complex , adaptive systems. However, a few comments:
    1) Sand piles are not self-organizing so the analogy breaks down when considering the tipping point – many times self-organizing systems can anticipate the shift and move to adjust so that avalanches don’t happen – they also learn. But then, that is your point isn’t it?
    2) Crowd-sourcing for resiliency can also lead to over-reaction, which can lead to the well-known phenomena in systems of overshoot and collapse. This is brought on when actors are reacting to only locally available information. Information/knowledge of system-level states take longer to integrate and process data to develop a holistic viewpoint. What emerges may not be resilience but overshoot behaviors that drive the system into an undesirable, less optimal state if not panic, collapse, or chaos. Some type of collective wisdom must emerge – which often, but not always, requires a centralized mechanism of integration. Other sources of system level behaviors that threaten resilience are the coupling of delays in negative feedback loops (that cause the overshoot) with positive reinforcing loops (that amplify the oscillations). These are not discussed.
    3) Transformation as a component of resilience is left out of the discussion. The inherent ability of a system to reconfigure/reinvent itself is an important concept that is not well understood. It involves both flexibility and connectivity – as does adaptive and “bounce back” concepts of resilience – but goes further. Adaptations allow systems to keep functioning under changed conditions but essentially keep playing the same game; transformation allows the system to change the game to provide more functionality at the system level. Punctuated equilibrium (as one comment mentioned) is one mechanism for transformation as a concept of resiliency – the ability to flee what has become a sub-optimal basin of attraction due to exogenous shocks. Other mechanisms should be explored.

    • Thanks for reading and for your feedback, Nancy.

      1) “But then, that is your point isn’t it?” –> Yes it is
      2) “… Systems of overshoot and collapse.” –> Indeed, hence my advocating for gentle action. Crowdsourcing is in now way incompatible with gentle action.
      3) Indeed, eg, anti-fragility.

  21. Pingback: The Best of iRevolution in 2013 | iRevolution

  22. Pingback: Sosyal medya toplumun sinir sistemi mi? « Cem S. Sütcü

  23. According to you which data would qualify as ‘big data’? Can structural system responses recorded (spatial-temporal) on existing structures subsequently used for condition assessment of the structure be called ‘big data’?

  24. Pingback: Pueden los Datos evitar tragedias como el 11M? | BigData4Success

  25. Pingback: Latest Findings on Disaster Resilience: From Burma to California via the Rockefeller Foundation | iRevolution

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s