Back to News
July 14, 2017
New Study Proves Statistical Acrobatics Can’t Hide Shoddy Methodology
A recent study published by the National Bureau of Economic Research is making the news for “proving” one of the most popular anti-gun fallacies out there: that more guns means more crime.
Of course, a simple analysis demonstrates a downward trend in crime rates, even as the number of firearms on the market has increased dramatically over the past 20 years, clearly rendering the argument ridiculous. Even without a PhD, the trends are clear. So how does the gun control crowd fight against reality? By devising complicated statistical analyses that poorly attempt to show the opposite is true. This new study is no different.
Building on a deeply-flawed study the authors published in 2014, they revisit the topic of the supposed effects of Right-to-Carry (RTC) laws on the violent crime rate. One of the premier researchers who was attacked in the study, John R. Lott, has published his own rebuttal (worth the read here) that tackles many of the cherry-picking strategies of the authors. There are far too many additional problems with the methodology of the study to cover in a blog post, but here are three of the more egregious problems.
“Violent Crime” is used as a variable
They use “Violent Crime” as a variable. Unlike prior studies, including the authors’ own previous studies, this report is entirely based on supposed effects of RTC laws on “the violent crime rate,” a miscellaneous measure that lumps together the radically different crimes of murder, rape, robbery, and aggravated assault. The only specific violent crime type addressed is murder, and the authors’ statistical results (as distinct from the verbal spin they put on them) indicated no significant (at the conventional 5% level) effect of RTC laws on murder rates.
Perhaps the analysis was limited to violent crime because the authors’ 2014 study, which used a virtually identical set of data, found no significant effect on murder rates, rape rates or robbery rates. The 2014 study did find borderline significant (5-10% significance) “effects” on aggravated assault rates, though only under some methodological conditions. Because the vast majority of reported violent crimes are aggravated assaults, most of what the authors are referring to as “violent crime” in the 2017 report is aggravated assault.
As far as the reader can determine, the 2017 study, like the 2014 study, found no effect of RTC laws on rates of murder, rape, or robbery, and possibly detrimental effects of RTC laws limited to aggravated assault, but obscured the weak and mixed character of their results by needlessly lumping together very different types of violent crime into a measure dominated by aggravated assaults.
If the laws really caused increases in aggravated assaults, they should, at minimum, specifically increase aggravated assaults committed with guns. The results reported in their 2014 study however, indicated that there was no significant impact of RTC laws on gun-related aggravated assault in 6 of 8 models. The authors choose not to mention this inconvenient finding in the report of their 2017 study.
Aggravated assault data is unreliable
The FBI’s Uniform Crime Reporting (UCR) data for aggravated assault is unreliable. It has long been recognized that police-based crime statistics are subject to serious error, except those pertaining to murder, because most other crime types are not reported to the police. Thus, the rate of reported aggravated assault or other violent crime type can appear to increase even if the actual rate did not increase at all. Likewise, the rate of reported crime can appear higher in one county or state than in another even if there is no actual difference.
How do we test the data? One key test of the validity of the UCR crime rates is checking to see how closely these city crime rates are correlated with crime rates based on victim surveys. The worst correlations are for aggravated assault. In fact, the correlation is significantly negative for this crime type – cities that had higher aggravated assault rates according to victim surveys had lower aggravated assault rates according to UCR police-based data.
Study results are highly unstable
As the authors know, the results of these types of studies are highly unstable. Studies done by the authors, the study done by the National Research Council that they cite, the one done by Marvell and Moody that they cite, and those done by others all agree on one point – the results of panel studies of state- and county-level crime statistics are highly unstable, and vary radically depending on which exact set of methodological procedures are used. The logical inference should be that the authors’ results cannot be relied upon, because they are likely to be reversed as soon as some future researcher introduces yet another methodological variant.
To summarize, the authors’ current and prior research has indicated – (1) an apparent crime-elevating effect of RTC for rates of a very poorly measured type of crime, aggravated assault, and (2) no significant effect on rates of murder, a very accurately measured type of crime.
This suggests that the authors’ aggravated assault results, and thus their results regarding the amorphous “violent crime rate” are an artifact of measurement error in UCR police-based crime data rather than a reflection of actual effects of RTC laws.
Any crime-increasing effect of RTC laws would presumably have to involve some RTC-produced increase in gun availability. Although the authors had data on gun prevalence, they never tested the effect of RTC laws on gun prevalence (they only test whether the effect of RTC laws on crime interacts with gun prevalence). Failing to carry out such an obvious test is an odd omission, which the authors do not explain.
Categories: BP Item, Featured, Government Relations, Top Stories, WTS
25650
