A brief analysis of the Global Terrorism Database
Julian Hatwell
April 1st, 2016
Introduction
This post is inspired by a recent article in the Huffington Post which contained a barchart (see appendices) of fatalities caused by terrorism in Western Europe over the years since 1970.
The original source of the data is the Global Terrorism Database, which is freely available to download. It’s fairly trivial to reconstruct the graph from this base data, as I show here. The original is in the appendices:
On initial visual assessment, the eye is drawn to the mountain of bars representing the 70’s and 80’s and the relative sparsity from the mid-90’s onwards. After 2001 there are only the 4 notable peaks in 2004 (Madrid), 2005 (London), 2011 (Norway) and 2015 (Paris). One can immediately gain a sense that the danger from terrorism from 1970 to the end of the 1990’s was much greater than it is now.
In my opinion, the graph is a fairly successful attempt to demonstrate that our current perception and fear of global terrorism is caused by something other than factual measures such as the number of deaths (or number of incidents). Our collective perception may be more a symptom of the tactics of terror or simply the dominant socio-political narrative. This unease we feel causes us to find rationales for ignoring the facts of our recent history and to allow ourselves to believe that no time is as dangerous as now.
My reposting of the original graphic on Facebook drew the attention of one particular friend who asserted that the graph wasn’t making a valid comparison to current Islamist terrorism because it “includes Northern Ireland in the 1970s and ETA in the 1980s which is not quite the same.”
He went on to say “ETA and IRA conflicts were localised not global, secondly, it could be argued that they were liberation movements in search of a political solution and as a result they had the sympathy of parties on the Left. Corbyn and Livingstone supported Sinn Fein but not Al Qaeda or ISIS.”
Essentially these comments serve well to emphasise my point. Can we say that people killed in Ireland by the IRA are less dead than people killed in Paris by Islamic extremists? Or were bombs planted by the IRA less dangerous because Jeremy Corbyn liaised with Gerry Adams when he was otherwise shut out of the political discourse? I think not and neither does the Global Terrorism Database make any such distinction.
This inspired me to do my own analysis of the Global Terrorism Database to see what (objective) stories the data has to tell and help to cut through some of the political noise and cognitive bias that surrounds such an emotive topic.
Exploring the data
To examine more closely the different trends in localised terrorism compared to international extremism it’s necessary to determine which are the local and international terror groups in the subset of the data pertaining to Western Europe and incidents with 1 or more fatality.
The Global Terrorism Database does not make this distinction explicit. I had to go through the list myself and label all the groups that I could be sure about (e.g. Basque separatists and Irish Republicans, compared to Secret Organization of al Qa’ida in Europe). I share my categories in the appendices.
My first question, of all the deaths from terrorism recorded in the database since 1970 (plus 130 from Paris 2015), how many are from International terrorism?
Judging by the numbers here, 4.7 times as many deaths were caused by localised terrorism between 1970 and 2015.
Given many of these localised groups have been much quieter since September 11th bombings of 2001 and we’ve had the Good Friday Agreement (GFA) working well since December 1999, it might help to look at the balance since the GFA:
The balance has shifted to 1.5 times as many deaths from international terrorists as localised terrorism. The international threat must feel more current, but is still altogether out of balance given the totals. Take a look at the proportions:
The dashed outlines show where numbers are lower than expected given the overall proportions. In short, the weight of fatalities from international terrorist attacks falls disproportionately after the GFA. This is no surprise.
What’s important is this; Based on an assumption that things are more dangerous now, in the 15 years after the GFA you’d expect to see at least half as many fatalities as in the 30 previous years, a ratio of 2:1 (Pre:Post). The observed ratio is 9:1. In other words, 90% of all these fatalities occurred before the beginning of 2001, the year of the September 11th attacks.
Here’s the first graph with the local vs international split highlighted:
One should conclude that those 378 deaths caused by global terrorists in the years since the GFA are almost entirely accounted for in Madrid and London bombings and the Paris attacks. Clearly these are very deadly incidents. However, it is now easy to pick out some equally large events in the data going back to the 1970’s and 1980’s. It would be useful see if the lethality of the attacks has changed over time.
Looking at average death toll per incident seems to indicate that when they strike, the international terrorists are deadlier. They appear to have become vastly more deadly per incident in the last 15 years. However, this simplistic explanation does not tell the whole story, as we shall see.
The following two graphs restate the same data as counts of numbers of incidents that result in fatalities:
Here, the localised terror groups absolutely dominate the graph. It’s also obvious that there was a far greater, more constant and continuous threat in the past. All incidents are rapidly diminishing in number to the present day.
Looking at the international incidents alone, we can see that they follow the same trend, decreasing in number since the 1970’s-80’s:
This is the clearest evidence that not only are global terror attacks are much rarer than localised groups over period 1970-2015, they have also become far more rare since the 1990’s.
So back to the question of whether the attacks have become deadlier in recent years. If that is the case then surely we do have as much to fear from larger scale attacks, even if they are less frequent.
Firstly, a look at the localised groups:
The points are aggregated such that August 1980 shows up as 137 fatalities. In fact the most lethal non-International terrorist attack in Western Europe was the “Strage di Bologna” which killed 85, allegedly carried out by the neo-fascist Nuclei Armati Rivoluzionari. A mark has been added to indicate where this would lie on the plot as a standalone point. Note how this compares to the second single most lethal attack, carried out by Anders Breivik in Norway (also indicated). The official explanation is that he was an individual unlikely to have been linked to any group.
For the localised terrorist attacks, the steady downward trend is obvious and this echoes what we know to be true from history; In Western Europe, terrorism from local sources (Nationalism, Republicanism, Revolutionary Communism, Fascism) was very prevalent in the 1970’s and 1980’s. From this peak it has steadily declined and all but disappeared today.
I continue along the same lines for the international extremist terror threat:
Here we see a shallow rise in the in the 1970’s to 90’s followed by a steady decline to the present day with very low fatality rates.
There is a majority of low level attacks with a small number of very serious incidents involving aeroplane bombings in the 70’s and 80’s, the public transport suicide bombings in Madrid and London (’04 and ’05) and the death cell that attacked Paris last year.
The four catastrophic flight bombings easily rival the death tolls of more recent attacks. So, if the fatalities are trending downwards for both groups, is it reasonable to see the apparent 10-fold increase in the average number of fatalities per incident as a sign of greater danger?
The problem is that it’s a somewhat crude measure. It takes no account of all the months where there were no fatal attacks. Consider instead the fatalities per month over the whole period. This acts as a weighting against very rare events:
Accounting for all the months when there were no incidents, it’s clear that the localised terrorists were the more deadly group in the period 1970-2000, by the sheer number of incidents. Even though international terrorists carried out a number of airplane bombing atrocities, these were less than a handful in two decades.
The threat from international terrorists, if measured in a rate of fatalities over a given time period, has remained constant and low and has never come close to the danger we faced in the period 1970 - 2000 from localised terrorism representing a mix of sectarian, separatist, revolutionary communist and fascist ideologies in different countries at the time.
This analysis deals with the situation in Western Europe and widely available, highly credible data. The only conclusion one can draw from this is that we’re much safer now than we were in the period 1970-2000.
Of course, there is no way to count the attacks that have been covertly thwarted before they take place. These represent an unknowable, unrealised danger only hinted at by government and media sources. To consider this at all takes us full circle to the question of why we feel so threatened in the first place.
Appendices
This was purely a theoretical exercise not intended to be taken as a prediction or forecast. There are many volatile factors that this picture could change dramatically and upredictably at any time. I’m looking at trends over time and not taking any influential factors in to consideration.
The analysis is done. If anyone is interested in the workings, read on.
Data Corrections
After downloading the freely available Global Terrorism Database I found that it contains data only up to 2014. Huffington Post’s statistical consultant probably added 2015-16 from other sources, although they don’t tell us that. I’ve added a value of 130 fatalities in France for 2015.
There is an entry for the Red Brigades 1974 hostage taking in Genoa with 1 killed which has month set as zero. I have updated this to April
Categorisation of the terrorist groups
The catogories applied to the various terror groups. Lockerbie was listed under “Unknown” but was easy to single out and label. Paris Attacks are also individually marked as this point has been added in. Any advice on making it more accurate is most welcome:
## tergroup International.Extremism
## 1 X1.May No
## 3 X2nd.of.June.Movement No
## 6 Abu.Hafs.al.Masri.Brigades Yes
## 7 Abu.Nidal.Organization..ANO. Yes
## 19 Action.Directe No
## 23 Action.Group.for.th.Dstrctn.f.th.Plc.Stt No
## 24 al.Fatah Yes
## 25 Al.Qa.ida.in.Iraq Yes
## 26 Al.Sa.iqa Yes
## 27 Al.Borkan.Liberation.Organization Yes
## 28 Anti.American.Arab.Liberation.Front Yes
## 29 Anti.Capitalist.Brigades No
## 30 Anti.Marxist.Commando No
## 31 Anti.terrorism.ETA..ATE. No
## 32 Anti.terrorist.Liberation.Group..GAL. No
## 44 Arab.Liberation.Army Yes
## 45 Arab.Revolutionary.Cells Yes
## 46 Arab.Unionist.Nationalist.Organization Yes
## 47 Arabian.Peninsula.Freemen Yes
## 48 Argentine.Anticommunist.Alliance..AAA. Yes
## 52 Armed.Groups.of.Organised.Proletariat No
## 53 Armed.Islamic.Group..GIA. Yes
## 55 Armed.Nuclei.for.Territorial.Counterpowr No
## 56 Armed.Nucleus.for.Popular.Autonomy No
## 57 Armed.Proletarian.Nuclei..NAP. No
## 63 Armed.Proletarian.Power No
## 64 Armed.Revolutionary.Nuclei..NAR. No
## 74 Armenian.Secret.Army.for.th.Lbrtn.f.Armn Yes
## 86 Autonomen No
## 87 Autonomous.Anti.Capitalist.Commnds..CAA. No
## 88 Autonomous.Revolutionary.Brigade No
## 89 Baader.Meinhof.Group No
## 95 Basque.Country.Autonomous.Self.Defns.Grp No
## 96 Basque.Fatherland.and.Freedom..ETA. No
## 327 Basque.guerrillas No
## 328 Basque.Separatists No
## 330 Basque.terrorists No
## 332 Black.December No
## 333 Black.Lebanon Yes
## 334 Black.Order No
## 337 Black.September No
## 343 Black.September.II No
## 344 Blue.Archer.Group No
## 345 Breton.Liberation.Front..FLB. No
## 348 Catalan.Liberation.Front..FAC. No
## 349 Catholic.Reaction.Force No
## 350 Combat.Brigades No
## 351 Commando.Heinz.Neumann No
## 352 Committee.for.the.Sfgrd.f.th.Islmc.Rvltn Yes
## 353 Cmmtt.f.Sldrty.wth.Arb.n.M.E.P.P..CSPPA. Yes
## 355 Communist.Combattant.Cells..CCC...Belgm. No
## 357 Communist.Fighting.Unit No
## 358 Communist.Nuclear No
## 359 Communist.Territorial.Nuclei No
## 360 Continuity.Irish.Republican.Army..CIRA. No
## 364 Corsican.National.Liberation.Frnt..FLNC. No
## 374 Corsican.Separatists No
## 375 Crazy.Brabant.Killers No
## 376 Croatian.Nationalists No
## 378 Cuncolta.Naziunalista No
## 379 Delta.Group No
## 381 Dev.Sol No
## 382 Direct.Action.Against.Drugs..DADD. No
## 384 Eagles.of.the.Palestinian.Revolution Yes
## 385 Frst.f.Octbr.Antfscst.Rsstnc.Grp..GRAPO. No
## 423 Force.17 No
## 425 Free.Galician.People.s.Guerrilla.Army No
## 427 French.National No
## 428 Front.for.the.Liberatn.f.Lbnn.frm.Frgnrs Yes
## 429 Front.for.the.Libertn.f.th.Frnch.Sml.Cst Yes
## 430 Fuerza.Nueva No
## 431 Greek.National.Socialist.Organization No
## 432 Grey.Wolves No
## 433 Guadeloupe.Liberation.Army No
## 434 Guardians.of.the.Islamic.Revolution Yes
## 435 Guardsmen.of.Islam Yes
## 436 Guerrillas No
## 438 Hizballah Yes
## 444 Hofstad.Network No
## 445 Honour.of.the.Police No
## 446 Iberian.Anarchist.Federation No
## 447 Iberian.Liberation.Movement..MIL. No
## 448 Individual No
## 466 International.Anti.Commnst.Intllgnc.Srvc No
## 467 Internationl.Jstc.Grp..Gm..l.Adl.l.Almy. Yes
## 468 Iparretarrak..IK. No
## 469 Iranians Yes
## 471 Iraqi.Mujahideen Yes
## 472 Irish.National.Liberation.Army..INLA. No
## 535 Irish.People.s.Liberation.Orgnztn..IPLO. No
## 549 Irish.Republican.Army..IRA. No
## 828 Irish.Republican.Extremists No
## 868 Italian.Social.Movement..MSI. No
## 875 Jammu.and.Kashmir.Liberation.Front Yes
## 876 January.22 No
## 877 Japanese.Red.Army..JRA. Yes
## 878 Jewish.Armed.Resistance Yes
## 879 Justice.Commandos.for.the.Armenian.Gencd Yes
## 889 Kurdistan.Workers..Party..PKK. Yes
## 891 Lebanese.Armed.Revolutionary.Fctn..LARF. Yes
## 894 Left.Wing.Militants No
## 895 Left.Wing.Terrorists No
## 896 Libyan.Students Yes
## 897 Libyan.Students..Sponsored.by.Qadff.Rgm. Yes
## 898 Loyalist.Volunteer.Forces..LVF. No
## 903 Mafia No
## 908 May.15.Organization.for.th.Lbrtn.f.Plstn Yes
## 910 Militant.People.s.Revolutionary.Forces No
## 911 Moslem.Liberation.Front Yes
## 912 Movement.for.Rebuilding.Fatah Yes
## 913 Movement.for.Self.Determination No
## 914 Movement.of.the.Third.Reich No
## 915 National.Front.for.the.Liberation.f.Blgm No
## 916 National.Front.for.the.Salvation.of.Liby Yes
## 917 National.Movement.Against.th.Mhgrb.Invsn Yes
## 918 National.Organiztn.f.Cyprt.Fghtrs..EOKA. No
## 919 National.Socialist.Underground No
## 926 Neo.Fascists No
## 929 Neo.Nazi.Group No
## 942 New.Armenian.Resistance Yes
## 943 New.Order No
## 944 November.17.Revolutionry.Orgnztn..N17RO. No
## 962 Official.Irish.Republican.Army..OIRA. No
## 988 Oglaigh.na.hEireann No
## 989 Organization.of.Mauritanian.Nationalists Yes
## 990 Organization.of.Mujahadin.of.Islam Yes
## 991 Organized.Comrads.for.Communism No
## 992 Orly.Organization No
## 993 Palestine.Liberation.Organization..PLO. Yes
## 995 Palestinians Yes
## 997 People.s.Liberation.Army..Northrn.Irlnd. No
## 998 People.s.Resistance.Organized.Army No
## 999 Polisario.Front No
## 1000 Popular.Forces.of.April.25 No
## 1007 Popular.Front.fr.th.Lbrtn.f.Plstn..PFLP. Yes
## 1014 Pplr.Frnt.fr.th.Lbrtn.f.P..G.C..PFLP.GC. Yes
## 1016 Portuguese.Anti.Communist.Movement No
## 1017 Prima.Linea No
## 1028 Pro.Libyan Yes
## 1029 Proletarian.Patrols No
## 1030 Protestant.Action.Group No
## 1033 Protestant.Extremists No
## 1156 Qaddaffis Yes
## 1159 Raul.Sendic.International.Brigade No
## 1160 Real.Irish.Republican.Army..RIRA. No
## 1165 Rebellious.Group.Lambros.Foundas No
## 1166 Red.Army.Faction..RAF. No
## 1174 Red.Army.for.the.Liberation.of.Catalonia No
## 1175 Red.Brigades No
## 1216 Red.Brigades.Fghtng.Cmmnst.Prty..BR.PCC. No
## 1219 Red.Brigades.Fightng.Cmmnst.Unn..BR.UCC. No
## 1221 Red.Commandos No
## 1222 Red.Flag No
## 1223 Red.Flying.Squad No
## 1224 Red.Hand.Commandos No
## 1233 Red.Hand.Defenders..RHD. No
## 1240 Republican.Action.Force No
## 1245 Revolutionary.Cells No
## 1246 Revolutionary.Nuclei No
## 1247 Revolutionary.Organization.f.Sclst.Mslms Yes
## 1248 Revolutionry.Ptrtc.Ant.Fscst.Frnt..FRAP. No
## 1252 Revolutionary.People.s.Struggle..ELA. No
## 1254 Revolutionary.Solidarity No
## 1255 Revolutionary.Struggle No
## 1256 Right.Wing.Extremists No
## 1260 Right.wing.Terrorists No
## 1261 S..O..S..France No
## 1262 Saor.Eire..Irish.Republican.Group. No
## 1264 Scheutzen.group No
## 1265 Secret.Organization.f.l.Q.U.0092.d.n.Erp Yes
## 1266 Sect.of.Revolutionaries..Greece. No
## 1268 Shiite.Muslims Yes
## 1269 Socialist.Patients..Collective..SPK. No
## 1270 Soldiers.of.Truth No
## 1272 South.Moluccans No
## 1277 Spanish.Armed.Group No
## 1278 Spanish.Basque.Battalion..BBE...rightst. No
## 1285 Spanish.Falange No
## 1286 Spanish.National.Action No
## 1288 Syrian.Mujahideen Yes
## 1289 Terra.Lliure No
## 1290 The.New.Irish.Republican.Army No
## 1292 Turkish.Revenge.Brigade Yes
## 1293 Turks Yes
## 1294 Ulster.Freedom.Fighters..UFF. No
## 1403 Ulster.Volunteer.Force..UVF. No
## 1530 Unknown No
## 1554 Lockerbie Yes
## 1712 Waltraud.Boock.Group No
## 1713 White.Extremists No
## 1714 White.Wolves..UK. No
## 1715 Wild.Cats No
## 1716 Workers.Brigade.for.Communism No
## 1717 Paris.Attacks YesThe plot trend lines
The main trend lines were added using a zero inflated model. Traditional linear models would be dragged downwards on the Y axis by the prevalence of zeros in the data, giving too low an estimate among other undesirable characteristics. The zero inflated model is robust to this type of data.
In the non-international plot, a secondary line is created from a basis function with knots that follow the non-linear ups and downs in historical trends.
To make the fairest possible comparison in the international groups plot, the same basis function above was considered. However, it was found to be far too sensitive to outliers. The model used here is a natural spline with 3 degrees of freedom which is a lot smoother and linear at the ends. Still, it is less stable than the hurdle model because of the spread between all the months with zero fatalities recorded and the number of points with very high recorded fatalities. This is reflected in the very large confidence bands and the uplift at the right extremity where it tries to head towards the influential outliers.
Finding the best fit
For the scatter plots, finding the right models to add trend lines was not a traditional linear regression task as the data are not iid.
I’m really looking at a time series where it is as certain as can be that near neighbours in the data will be correlated. Whatever are the factors that lead to terrorism one month are probably just as true in the previous and next months, but may vary more from year to year, and most as the decades go by.
There are also many months where the number of incidents, and consequently fatalities, is zero. It’s not a smooth distribution.
The challenge was to find a robust smoothing line that reveals trends in the count data where there is serious non-linearity and dramatic zero inflation.
I started with the assumption that number of fatalities in a given month follows standard models for count data, but initially struggled with it as can be seen from the first attempt to perform a poisson goodness of fit:
What’s happening here? The data are very skewed and my obvious next step is to operate on the log transformed data (+1 for the zeros). When thinking about why that is, it might make more sense to think of the incidents themselves as the count data, while the number of deaths for each incident is really some level of intensity. This might even be considered to be quantitative, and certainly leaves me thinking that I’m actually trying to model a multivariate response of frequency and intensity. That’s going to take quite a bit of brain power and will require a lot more explanatory variables to solve it. I put this problem aside for another day. I just need something good enough for a trend line.
I try with the log transform to tame the high skew and dispersion.
Here in the poisson model I can see evidence of zero inflation. Note that the integer valued bars aggregate a number of values by the log transform, and how the previous graph has a very pronounced wave shape starting way below the line at zero and the travelling well above the line for many values. This means there are more times when zero deaths are counted in a given month than the standard poisson model allows for.
These are a little better with less evidence of zero inflation. However, there’s still a real lack of fit at \(e^2\) and above. I want to find a model suitable for this situation.
I go through the same process with the fatalities data from international terror incidents.
This is a disasterous fit. There is huge zero inflation compared to what the model expects followed by intense overdispersion at higher values. Of course this stands to reason as there are many months out of the time period when there are no fatal incidents.
Again, the negative binomial is a far better, albeit rather imperfect fit. This can be corrected by means of a zero inflated or hurdle model:
Here is the non-international data fitted with linear negative binomial, hurdle and zero-inflated poisson models:
Here is the international data fitted with linear negative binomial, hurdle and zero-inflated poisson models:
There’s not much to choose between the latter two in each case. Vuong’s non-nested hypothesis test is useful here:
## Generic hurdle and zero infl fits to the data. Non-international, then international## Vuong Non-Nested Hypothesis Test-Statistic:
## (test-statistic is asymptotically distributed N(0,1) under the
## null that the models are indistinguishible)
## -------------------------------------------------------------
## Vuong z-statistic H_A p-value
## Raw -0.8600498 model2 > model1 0.19488
## AIC-corrected -0.8600498 model2 > model1 0.19488
## BIC-corrected -0.8600498 model2 > model1 0.19488## Vuong Non-Nested Hypothesis Test-Statistic:
## (test-statistic is asymptotically distributed N(0,1) under the
## null that the models are indistinguishible)
## -------------------------------------------------------------
## Vuong z-statistic H_A p-value
## Raw -2.200632 model2 > model1 0.013881
## AIC-corrected -2.200632 model2 > model1 0.013881
## BIC-corrected -2.200632 model2 > model1 0.013881## Final hurdle and zero infl fits to the data, using parameters discovered from model fitting process. Non-international, then international## Vuong Non-Nested Hypothesis Test-Statistic:
## (test-statistic is asymptotically distributed N(0,1) under the
## null that the models are indistinguishible)
## -------------------------------------------------------------
## Vuong z-statistic H_A p-value
## Raw -0.2960719 model2 > model1 0.38359
## AIC-corrected -0.2960719 model2 > model1 0.38359
## BIC-corrected -0.2960719 model2 > model1 0.38359## Vuong Non-Nested Hypothesis Test-Statistic:
## (test-statistic is asymptotically distributed N(0,1) under the
## null that the models are indistinguishible)
## -------------------------------------------------------------
## Vuong z-statistic H_A p-value
## Raw -1.40573 model2 > model1 0.079902
## AIC-corrected -1.40573 model2 > model1 0.079902
## BIC-corrected -1.40573 model2 > model1 0.079902Negative values favour model 2, in the sequence. My test for the non-international doesn’t give a significant result and for International groups favours model 2, zero inflated rather than hurdle.