What's the meaning of 'highly likely'?
Featured image from Comedy Central via Imgur.
The good days to bury bad news continue, as the information vortex continues to swirl around Belgium. I'm sure everyone may be happier there when the cameras and those with agendas pick up their soap boxes and go home. Counter terrorism is hard, and incidents certainly scary - but people are getting used to it. As the Financial Times pointed out this morning, after 9/11, when the U.S. indexes finally reopened (Sept 17th), the NYSE fell 7.1% and continued falling for the rest of the week. On Tuesday, most European markets shook off a 'wobble' and had recovered by the afternoon.
Today however, I wish to discuss something regarding the opinions and policy decisions being made in the rest of Europe, such as the greater deployment of police in most European capitals. One of the reasons this is justified, as Michael Fallon was discussing on Channel 4 news yesterday, is the use of threat levels by domestic intelligence agencies. At the moment, the threat level from International Terrorism the UK* is Severe, which it has sat at since August 29th, 2014. You can find out more about how these levels are set from MI5's website, and also the history of level changes. The only time the levels have hit critical in the UK were related to two quite specific incidents; one of them is the reason that liquids are banned on flights (2007), and the other was the falling-at-the-last-hurdle car bombs outside Tiger Tiger and the Glasgow airport incident, which surely IS should have remembered the attitude that locals take to people trying to disrupt their travel plans.
So far so what? The threat levels are designed to give a broad indication of the likelihood of an incident, so in the last 572 days, an attack has been 'highly likely'. But what does 'highly likely' actually mean? Academics in the field of risk management know that verbal expressions of risk are of limited use because of the huge range of interpretation that people can make from a single expression. This is valid for lay people (as most people probably would expect) and for experts, as was observed by Ruth Beyth-Marom in 1982. Beyth-Marom experimented with political forecasters, and found that the range of variability in some expressions (at 80% ranges, i.e. the most extreme 20% of responses were discarded) was not consistent among these experts, who all worked for the same organisation. 'High chance' for example, represented a probability likelihood range between 71 and 91; 'Close to certain' meanwhile, represented a probability range between 58 and 97. Beyth-Marom's categorisation of expressions even splits 'high chance' (70-90) and 'very high chance' (70-100), despite a 20 point overlap. Her experiment also demonstrated that these interpretations of numbers depend on the context of numbers given. Beyth-Marom's paper has some good graphics demonstrating these numbers but they do not copy well, so look it up if you have access to Wiley online's academic journals (Journal of Forecasting, Issue 1). This effect has been observed in a number of other studies, such as Budescu et al on IPCC reports (2009), and Teigen and Brun (1999) on directionality (positivity and negativity of phrases).
So what does this mean for data and public policy? Taking Beyth-Marom's stricter interquartile range (discarding 50% of the most extreme answers) of the phrase 'high chance', if there was a 75% - 87% chance of a terrorist incident on the underground, or your city's bus network, tomorrow, would you use it?
After being told of that risk (75% - 87%) for 572 days, with only one occurrence (the somewhat tendentious stabbing in Leytonstone), how much attention would you actually pay to this warning? The security services require the public to be 'alert but not alarmed', but does such a use of the phrase 'highly likely' risk the public tuning out the warnings?
*Note: there are actually three levels, the other two of which refer to activity in Northern Ireland. Obviously for now, we'll just leave that where it is, but there were two incidents in Northern Ireland over the covered period.