One of the inherent concerns about crowdsourced crisis information is that the data is not statistically representative and hence “useless” for any serious kind of statistical analysis. But my colleague Christina Corbane and her team at the European Commission’s Joint Research Center (JRC) have come up with some interesting findings that prove otherwise. They used the reports mapped on the Ushahidi-Haiti platform to show that this crowdsourced data can help predict the spatial distribution of structural damage in Port-au-Prince. The results were presented at this year’s Crisis Mapping Conference (ICCM 2010).
The data on structural damage was obtained using very high resolution aerial imagery. Some 600 experts from 23 different countries joined the World Bank-UNOSAT-JRC team to assess the damage based on this imagery. This massive effort took two months to complete. In contrast, the crowdsourced reports on Ushahidi-Haiti were mapped in near-real time and could “hence represent an invaluable early indicator on the distribution and on the intensity of building damage.”
Corbane and her co-authors “focused on the area of Port-au-Prince (approximately 9 by 9 km) where a total of 1,645 messages have been reported and 161,281 individual buildings have been identified, each classified into one of the 5 different damage grades.” Since the focus of the study is the relationship between crowdsourced reports and the intensity of structural damage, only grades 4 and 5 (structures beyond repair) were taken into account. The result is a bivariate point pattern consisting of two variables: 1,645 crowdsourced reports and 33,800 damaged buildings (grades 4 and 5 combined).
The above graphic simply serves as an illustrative example of the possible relationships between simulated distributions of SMS and damaged buildings. The two figures below represent the actual spatial distribution of crowdsourced reports and damaged buildings according to the data. The figures show that both crowdsourced data and damage patterns are clustered even though the latter is more pronounced. This suggests that some kind of correlation exists between the two distributions.
Corbane and colleagues therefore used spatial point pattern process statistics to better understand and characterize the spatial structures of crowdsourced reports and building damage patterns. They used the Ripley’s K-function which is often considered “the most suitable and functional characteristic for analyzing point processes.” The results clearly demonstrate the existence of statistically significant correlation between the spatial patterns of crowdsourced data and building damages at “distances ranging between 1 and 3 to 4 km.”
The co-authors then used the marked Gibbs point process model to “derive the conditional intensity of building damage based on the pairwise interactions between SMS [crowdsourced reports] and building damages.” The resulting model was then used to compute the predicted damage intensity values, which is depicted below with the observed damage intensity.
The figures clearly show that the similarity between the patterns exhibited by the predictive model and the actual damage pattern is particularly strong. This visual inspection is confirmed by the computed correlation between the observed and predicted damage patterns shown below.
In sum, the results of this empirical study demonstrates the existence of a spatial dependence between crowdsourced data and damaged buildings. The results of the analysis also show how statistical interactions between the patterns of crowdsourced data and building damage can be used for modeling the intensity of structural damage to buildings.
These findings are rather stunning. Data collected using unbounded crowdsourcing (non-representative sampling) largely in the form of SMS from the disaster affected population in Port-au-Prince can predict, with surprisingly high accuracy and statistical significance, the location and extent of structural damage post-earthquake.
The World Bank-UNOSAT-JRC damage assessment took 600 experts 66 days to complete. The cost probably figured in the hundreds of millions of dollars. In contrast, Mission 4636 and Ushahidi-Haiti were both ad-hoc, volunteer-based projects and virtually all the crowdsourced reports used in the study were collected within 14 days of the earthquake (most within 10 days).
But what does this say about the quality/reliability of crowdsourced data? The authors don’t make this connection but I find the implications particularly interesting since the actual content of the 1,645 crowdsourced reports were not factored into the analysis, simply the GPS coordinates, i.e., the meta-data.
iRevolutions 
Nice review of interesting work. A quibble on your estimate of how much the UNOSAT work cost.
As a rough estimate (I have no information beyond your post):
600 experts * 66 days/expert * 8 hours/day * $100/hour = $32 million
This was, in truth, larger than I expected, but far shy of your estimate of “hundreds of millions”.
Yes, you’re right, that definitely is a quibble given the point/findings of the study. In any case, you’re forgetting to include the cost of the very high resolution aerial imagery for both pre- and post disaster. But even if you ignore the financial cost incurred in acquiring that data, are we seriously going to argue when comparing $32 million to close to $0? Not to mention the 66 days versus 14 days? Isn’t that missing the entire point of the study? You’re right though, that certainly qualifies as a quibble.
Hey Christoph, sorry, just realized i may have come across as somewhat terse in my reply. I wasn’t quite sure how to calculate the approximate cost of the PDNA and yes, my figure may certainly be off.
No worries.
Do Corbane et al. provide any suggestions on how crowdsourced input should be filtered or adjusted – or even if it needs to be filtered/adjusted? You’ve written about that component of “crowd reliability” before: https://irevolution.wordpress.com/2010/09/22/911-system/
Would you have a link to the Corbane et al study? I have not been able to locate it and would be very interested to read it.
The study is not formally out yet
A link to the study would be great. This is really fascinating. I would be interested to know if similar predictions could be made about disease outbreaks with crowdsourced data.
Will let you know when the study comes out
I am one of the “et al”. Just to put things in perspective, those 600 experts contributed mostly by digitising damage artefacts in tiles of the aerial imagery. I guess 3 days full-time on average per expert is a more reliable estimate of that effort. Inside the UNOSAT/WB/JRC team, I’d say a total of 12 experts worked full time for 6 weeks on this. That will lower your cost by a factor 20. WB told me their imagery costs US$350,000 (not sure if this also included the LIDAR flight). Add Google and NOAA airborne to that, and you get to US$ 1,000,000 for imagery only.
Beware of the context, however. The image analysis was done to come up with the damage, losses and needs in the housing sector. That was estimated at 6.5 BUS$, if I remember correctly. Our main motivation for the study was to see if the SMS patterns would have allowed for a better structuring of the damage assessment effort, eventually reaching a better accuracy, at lower cost, but always to make sure that the impact estimate was as close as possible to reality.
Hope this helps.
That definitely helps, thanks Guido! And sorry I missed you at the JRC earlier this week. It was great to participate and present at ValGeo 2010.
Patrick –
Great post and really interesting. Can you tell us more about how the experts modeled the data statistically, and what kind of distributional assumptions were made about the data. Also, were any other covariates collected to test joint spatial dependence or thrown is as controls on the models that determine the ‘significance’ of the crowd-sourced data? Lastly, any thoughts on how access to cell-phones in Haiti makes this study generalizable or have less external validity?
Cheers,
Grant
Best to ask Guido directly who just commented on the post above since I didn’t do the study
Patrick really…? Your numbers a orders of magnitude off. The WB-JRC-UNOSAT assessment had several phases one of which was a crowd sourcing phase that engaged 600+ scientists and engineers from over 23 countries who volunteered their time to do a building by building analysis of the the damage. This was achieved in less than a week. Have another look at Galen’s ignite presentation at the recent ICCM conference 🙂
LOL, I love it. Everyone is focusing on the cost of the assessment instead of the actual results and focus of the study, thereby missing the point, wouldn’t you say? I wish I could have paid more attention to Galen’s Ignite Talk at ICCM 2010 but was too busy dealing with conference logistics via Skype because we weren’t able to hire proper conference organizers. As for how long it took to release the PDNA, the JRC study noted 2 months, not a week. But again, the whole point of the study was to test whether any correlation existed between damaged building and text messages. How about talking about that bit? 🙂
ps. I have crossed out my numbers since they were orders of magnitude off.
Looks great Patrick & Guido! where can we read more about the process undertaken and correction factors? Those are key IMO.
I was trying to reconcile the precision of position reports:
1)people reported neighborhood-level positions mostly (what is the average size of neighborhoods in the area? 1 km across?)
2) with
““focused on the area of Port-au-Prince (approximately ***9 by 9 km***) where a total of 1,645 messages have been reported and 161,281 individual buildings have been identified, each classified into one of the 5 different damage grades.”
2) with
“The results clearly demonstrate the existence of statistically significant correlation between the spatial patterns of crowdsourced data and building damages at ***distances ranging between 1 and 3 to 4 km.***”
Thinking about this geographic resolution, also made me wonder how the process corrects for population/traffic density overall.
e.g. if you get a heatmap of density of cellphone subscriber transit in the period of the day when reports were sent; as a reference for correction; how much does it differ from the SMS data reports?
That could be the difference between the research showing “Crowdsourced data predicting crisis impact” …and “Research finds there are more people where there are more buildings” and that “more buildings are damaged where there are more buildings”, no? (thus links to help understand the correction factors are key)
Thanks!
–digress
We could use e.g. cell subscriber densities at different times of day as a way of mapping urban vulnerability. Makes me think of what an Urban-level GDACS alert of sorts would look like; and improvements we have to do to in all our systems for the sorts of spatial resolutions we need for urban work
IMO the above seems to confirm what we in MapAction have felt about the potential value of Ushahidi in sudden-onset events – that the value lies more in the aggregate distribution of incidents than in the individual report contents.
This is because a recurring problem in the field, especially in EQ disasters, is establishing the extent of human impact. If crowd data can be a reliable proxy for this, it will be useful. However it MUST be available quickly – within the first 48 hours or so – and is just one of a number of data sources that will be triangulated (government reports, conventional media, known areas of community vulnerability, overfly reconnaissance etc) before they can be used to set geographical priorities for field-level assessment.
Just one other thing. Properly surveyed (rather than inferred) building damage assessment is very important for early recovery actions. In Haiti it would have been worth spending a lot of money (although never the ‘hundreds of millions’ – come on!) as a key evidence source for the PDNA which supported the appeal for $11.5bn of relief and reconstruction assistance.
However, remote sensed (whether by imagery or SMS cloud) building damage data does not always, of itself, have overwhelming operational usefulness during the early stages of response. USAR teams are usually tasked based on structural triage which requires on-ground ‘sizeup’ and assessment.
Pingback: Analyzing Call Dynamics to Assess the Impact of Earthquakes | iRevolution
Pingback: Top 10 Posts of 2010 | iRevolution
A/I don’t understand something. could someone help me.
1/ “The results clearly demonstrate the existence of statistically significant correlation between the spatial patterns of crowdsourced data and building damages at “distances ranging between 1 and 3 to 4 km.”
=> so the granularity of the correlation is about several kilometers, isn’t it?
2/ “The co-authors then used the marked Gibbs point process model to “derive the conditional intensity of building damage based on the pairwise interactions between SMS [crowdsourced reports] and building damages.”
The figures clearly show that the similarity between the patterns exhibited by the predictive model and the actual damage pattern is particularly strong. ”
=> using this predictive model, the 2 figures seem to show a granularity of correlation much higher than several kilometers (if the squares represent the 9×9 km area of port de prince)
so I am not sure to understand: Are the results in a first stage a “raw” correlation and in a second stage , an improvment of the correlation regarding a predictive model? Could someone clarify this points. Thanks
B/Of course the next question: How to use such results? What is the consistency of such model for other earthquakes/crisis? One use case really cannot show something.
=> for the moment if I were picky I would say such predictive model doesn’t demonstrate any cost cuts because it has been designed “a posteri” i.e.using the costly generated accurate damage assessment data (and It is always easier to design a posteri a predictive model on past events with all the right data than predicting the real future with a lack and a noisy data)
So next step is to use this resulting haiti predictive model and apply it into another ushahidi platform used for another earthquake/crisis and check the confidence of the results. It’s only at that stage that the author could start to show the confidence and thus cost effective property of the method. This would be very interesting. I hope the authors will continue such research and not move to another topic as usual.
Best,
Nicolas
Pingback: An Open Letter to the Good People at Benetech | iRevolution
Pingback: Tracking Population Movements using Mobile Phones and Crisis Mapping: A Post-Earthquake Geospatial Study in Haiti | iRevolution
Pingback: The Best of iRevolution: Four Years of Blogging | iRevolution
Pingback: Big Data for Development: Challenges and Opportunities | iRevolution
Pingback: How to Create Resilience Through Big Data | iRevolution
Pingback: Tracking Population Movements using Mobile Phones and Crisis Mapping: A Post-Earthquake Geospatial Study in Haiti | iRevolution