My colleague Nils Weidmann just published this co-authored piece with Harvard Professor Monica Toft. The paper deserves serious attention. Weidmann and Toft review this article on the spatial prediction of ethnic conflict that was authored by Lim, Metzler and Bar-Yam (LMB) and published in the prestigious journal Science.
I reviewed the article myself earlier this year and while I was highly suspicious of the findings—correlations of 0.9 (!) and above—I did not dig deeper. But Weidmann and Toft have done just this and their findings are worth reading.
The authors clearly show that the analysis by LMB “suffers from a biased selection of groups and regions, an inadequate null hypothesis and unit of analysis.” This really begs the following question: how did the LMB paper ever make it through the peer-review process?
The authors’ case selection is seriously biased as it “seems to adjust the group map as to better fit the model predictions,” for example. The isolationist policy recommendations that LMB put forward are thus founded on misleading methods and ought to be entirely dismissed.
Better yet, Science should retract the LMB paper or at least publish the commentary by Weidmann and Toft. Indeed, another question that follows from the conclusion reached by Weidmann and Toft is this: how many other below-par papers have been accepted and published by Science?
In sum, not only are the methods used by LMB questionable, but as Weidmann and Toft conclude, “the model provides little advance on prior research” in the field of crisis mapping.
On the plus side, the fact that there is push back on early articles in the field of crisis mapping is also a good sign and evidence that the field is becoming more formalized. In addition, the general approach taken by LMB still holds much promise for crisis mapping—it simply needs to be done with a lot more care and transparency. Indeed, combining agent-based models with real world empirical data and a sound understanding of ethnic conflict could become a winning strategy for crisis mapping analytics.
In closing, I look forward to following Nils Weidmann’s work at Princeton and have no doubt that he will continue to play an important role in the development of the field, and will do so with integrity and rigorous scholarship.
iRevolutions 
I thought that I must have missed something when I first read the LMB article. If I achieved a model fit of 0.98 for any of my social science work then the first thing I would think is that I had done something terribly wrong such as over-fitting my model. I am glad that Weidmann and Toft performed this analysis because it sheds light on LMB’s findings. I look forward to reading LMB’s response to Weidmann and Toft.
Thanks for reading, John.
Pingback: The Best of iRevolution: Four Years of Blogging | iRevolution