Study author Colin Clarke says he set out to assess New Zealand’s mandatory helmet laws and came to his conclusion by pooling injury and behavioural data from various sources.
But correlation does not imply causation – and this is one reason Clarke’s analysis falls down.
Randomised controlled trials (RCT) are the gold standard when trying to establish a causal link between factor A and outcome B. However, not every potential causal link can be studied this way.
In Clarke’s analysis, enacting a mandatory helmet law is factor A and negative events such as premature deaths constitute outcome B. But his paper fails to meet any of the necessary criteria to establish a causal relationship. This generally requires at least four stages of analysis:
1) strength of relationship 2) correct temporal ordering 3) presence of dose response 4) elimination of potential confounders.
Let’s see how Clarke’s analysis shapes up against this criteria.
1) Strength of relationship
The first step in establishing a causal link is to measure the association between factor A (helmet laws) and outcome B (cyclist injury or death).
For a trend analysis such as this one, it is important to estimate and compare the changing rates of injury among one or more cyclist groups. Then, when introducing an intervention such as mandatory helmet laws, you need to effectively estimate the injury rate before and after the intervention, to assess whether it had an impact on the outcome, or whether it is part of a larger trend.
Clark fails to mention that rates of commuter cycling were declining from around 1986, long before mandatory helmet laws were introduced in 1994. And long before helmet use began to increase, from 1990.
2) Correct temporal ordering
For factor A (helmet laws) to have caused outcome B (cyclist injury and death), A must precede B in time.
Clarke does compare data from before and after the helmet laws were introduced. But the data is from 1988 to 1991 (three years before the helmet laws) and 2003 to 2007 (13 years later). Clarke presents no data from around the time the laws were introduced, with the exception of fatalities.
This is an important distinction because adjacent dates are more correlated than dates further apart. On the flip side, fatalities, injuries and cycling rates nearer in time to the introduction of mandatory helmet laws are more likely to be influenced by this change.
Clarke links a 51% reduction in average hours cycled per person by comparing years 1989 to 1990 with 2006 to 2009. Again, neither period is near the introduction of mandatory helmet laws.
The article does present data for 1996 to 1999, a period more relevant to mandatory helmet laws. When this period is compared with the equivalent pre-helmet law period of 1988 to 1991, there are substantial drops in cyclist injuries overall (down 47%) and serious injuries (down 53%).
This drop remains present even when adjusting for the amount of cycling in those years (down 17% and 53% respectively). The drop in serious cycling injuries (adjusted for million hours spent travelling) when comparing these two periods is significantly more than that for pedestrians, suggesting that mandatory helmet laws protected cyclists, rather than put them at risk.
Additionally, there was a 23% decline in bicycle-related fatalities in the immediate three years after the helmet laws were introduced (1994 to 1996) compared with the preceding three years (1991 to 1993). Clarke does not mention this fact in his analysis and discussion.
3) Presence of a dose response
In a causal relationship, increases or decreases in a factor leads to an increase or decrease in an outcome. This is known as a dose response.
Helmet laws were introduced to increase helmet wearing and mitigate bicycle-related head injuries. In this analysis, the rate of helmet wearing is the dose. From 1990 to 1998, helmet wearing increased to about 95% and remained steady after the helmet laws were introduced, with the largest increase among adults.
The fatality and injury data presented in Clarke’s paper shows a decline in cyclist fatalities, cyclist injuries and the ratio of cyclists and pedestrian injuries (per million hours of activity) during this time when helmet wearing increased rapidly. This shows a dose effect for helmet wearing and declining fatalities or injuries.
Since that time, helmet wearing has remained very high in New Zealand and injuries have risen. This shows there is no dose effect. The implication is that other factors have influenced the rate of bicycle-related injuries and fatalities over the last 18 years beyond mandatory helmet laws.
4) Elimination of potential confounders
When the above three criteria have been met, it is then important to rule out other variables besides factor A that could contribute to outcome B.
Clarke makes no attempt to address confounding factors and attributes all declines in cycling rates and increases in cycling fatalities and injuries to the helmet law.
But, if you look to Clarke’s source for bicycle injuries, Sandar Tin Tin’s 2010 study, you see several potential reasons outside of mandatory helmet laws for declines in cycling rates and increases in injuries. These include the lack of a cycling focus in the New Zealand road safety agenda, an increase in kids being driven to school (due to parental concerns of safety) and the pre-helmet-law decline in cycling rates.
It is important to question long-standing public health policies. But there are too many weaknesses in Clarke’s analysis and choice of data – particularly the four year absence of data around the time helmet laws were introduced – to support his conclusions that mandatory helmet laws halved the number of cyclists and contributed to 53 deaths each year.
Further, there is no indication from Clarke’s study if the rate of head injuries or fatalities increased or decreased after helmet laws were introduced, as the data presented are for all bicycle related injuries and fatalities.
Past research shows a significant decline in traumatic brain injuries between the pre- and post-mandatory helmet periods. But the scientists involved did not rush to declare that the mandatory helmet laws caused the decline. Instead, they delivered a perfectly weighted comment in light of the data and methodological limitations.
In the future, I would hope Mr Clarke follows their lead.
Dr Jake Olivier is a senior lecturer in biostatistics in the Prince of Wales Clinical School at UNSW.
This opinion piece first appeared in The Conversation.