Debate over impact of lowered prosecutions on city homicides; additionally bigger questions on artificial management strategies in causal inference.
Andy Wheeler writes:
I believe this forwards and backwards could also be of curiosity to you and your readers.
There was a broadcast paper attributing very giant will increase in homicides in Philadelphia to the insurance policies by progressive prosecutor Larry Krasner (+70 homicides a yr!). A bunch of researchers then revealed an intensive critique, going by means of completely different potential variants of knowledge and fashions, exhibiting that fairly a couple of affordable variants estimate lowered homicides (with commonplace errors typically masking 0):
– Hogan original paper,
– Kaplan et al critique
– Hogan response
– my writeupI do know these posts are loads of weeds to dig into, however they contact on fairly a couple of matters which are recurring themes to your weblog—many researcher levels of freedom in artificial management designs, revealed papers getting extra deference (the Kaplan critique was rejected by the identical journal), a researcher not sharing information/code and utilizing that obfuscation as a defend in response to critics (e.g. your replication information is unhealthy so your critique is invalid).
I took a glance, and . . . I believe this use of artificial management evaluation isn’t good. I just about agree with Wheeler, besides that I’d go additional than he does in my criticism. He says the artificial management evaluation within the examine in query has information points and issues with forking paths; I’d say that even with none points of knowledge and forking paths (for instance, had the evaluation been preregistered), I nonetheless wouldn’t prefer it.
Overview
Earlier than attending to the statistical particulars, let’s overview the substantive context. From the unique article by Hogan:
De-prosecution is a coverage to not prosecute sure legal offenses, no matter whether or not the crimes had been dedicated. The analysis query right here is whether or not the appliance of a de-prosecution coverage has an impact on the variety of homicides for giant cities in america. Philadelphia presents a pure experiment to look at this query. Throughout 2010–2014, the Philadelphia District Legal professional’s Workplace maintained a constant and sturdy variety of prosecutions and sentencings. Throughout 2015–2019, the workplace engaged in a scientific coverage of de-prosecution for each felony and misdemeanor instances. . . . Philadelphia skilled a concurrent and traditionally giant enhance in homicides.
I’d phrase this barely otherwise. Reasonably than saying, “Right here’s a common analysis query, and we now have a pure experiment to study it,” I’d desire the formulation, “Right here’s one thing fascinating that occurred, and let’s attempt to perceive it.”
It’s tough. On one hand, sure, one of many main causes for arguing concerning the impact of Philadelphia’s coverage on Philadelphia is to get a way of the impact of comparable insurance policies there and elsewhere sooner or later. Alternatively, Hogan’s paper may be very a lot centered on Philadelphia between 2015 and 2019. It’s not constructed as an observational examine of any common query about insurance policies. Sure, he pulls out another cities that he characterizes as having completely different common insurance policies, however there’s no try to totally contain these different cities within the evaluation; they’re simply used as comparisons to Philadelphia. So in the end it’s an N=1 evaluation—a quantitative case examine—and I believe the title of the paper ought to respect that.
Following our “Why ask why” framework, the Philadelphia story is an fascinating information level motivating a extra systematic examine of the impact of prosecution insurance policies and crime. For now we now have this comparability of the remedy case of Philadelphia to the management of 100 different U.S. cities.
Listed below are a few of the information. From Wheeler (2023), right here’s a comparability of tendencies in murder charges in Philadelphia to 3 different cities:
Wheeler chooses these specific three comparability cities as a result of they had been those that had been picked by the algorithm utilized by Hogan (2022). Hogan’s evaluation compares Philadelphia from 2015-2019 to a weighted common of Detroit, New Orleans, and New York throughout these years, with these cities chosen as a result of their weighted common lined as much as that of Philadelphia through the years 2010-2014. From Hogan:
As Wheeler says, it’s kinda goofy for Hogan to line these up utilizing murder rely quite than murder charges . . . I’ll have extra to say in a bit concerning this use of artificial management evaluation. For now, let me simply observe that the overall sample in Wheeler’s longer time collection graph is according to Hogan’s story: Philadelphia’s murder fee moved up and down over the many years, in vaguely related methods to the opposite cities (growing all through the Sixties, barely declining within the mid-Seventies, rising once more within the late-Eighties, then steadily declining since 1990), however then steadily growing from 2014 onward. I’d wish to see extra cities on this graph (pure comparisons to Philadelphia could be different Rust Belt cities equivalent to Baltimore and Cleveland. Additionally, hey, why not present a mixture of different giant cities equivalent to LA, Chicago, Houston, Miami, and so on.) however that is what I’ve bought right here. Additionally it’s annoying that the above graphs cease in 2019. Hogan does have this graph only for Philadelphia that goes to 2021, although:
As you’ll be able to see, the rise in homicides in Philadelphia continued, which is once more according to Hogan’s story. Why solely use information as much as 2019 within the analyses? Hogan writes:
The years 2020–2021 have been deliberately excluded from the evaluation for 2 causes. First, the AOPC and Sentencing Fee information for 2020 and 2021 weren’t but obtainable as of the writing of this text. Second, the 2020–2021 information could also be considered as aberrational due to the coronavirus pandemic and civil unrest associated to the homicide of George Floyd in Minnesota.
I’d nonetheless wish to see the evaluation together with 2020 and 2021. The primary evaluation is the comparability of time collection of murder charges, and, for that, the AOPC and Sentencing Fee information wouldn’t be wanted, proper?
In any case, primarily based on the graphs above, my overview is that, yeah, homicides went up loads in Philadelphia since 2014, a rise that coincided with lowered prosecutions and which didn’t appear to be taking place in different cities throughout this era. At the least, so I believe. I’d wish to see the time collection for the charges within the different 96 cities within the information as nicely, going from, say, 2000, all the best way to 2021 (or to 2022 if murder information from that yr at the moment are obtainable).
I don’t have these 96 cities, however I did discover this graph going as much as 2000 from a distinct Wheeler post:
Ignore the shaded intervals; what I care about right here is the information. (And, yeah, the graph ought to embody zero, because it’s in the neighborhood.) There was a nationwide enhance in homicides since 2014. Sadly, from this nationwide development line alone I can’t separate out Philadelphia and some other cities which may have instituted a de-prosecution technique throughout this era.
So, my abstract, primarily based on studying all of the articles and discussions linked above, is . . . I simply can’t say! Philadelphia’s murder fee went up since 2014 throughout the identical interval that it decreased prosecutions, and this was a part of a nationwide development of elevated homicides—however there’s no simple method given the immediately obtainable data to match to different cities with and with out that coverage. This isn’t to say that Hogan is flawed concerning the coverage impacts, simply that I don’t see any clear comparisons right here.
The artificial controls evaluation
Hogan and the others make comparisons, however the comparisons they make are to that weighted common of Detroit, New Orleans, and New York. The difficulty is . . . that’s simply 3 cities, and murder charges can fluctuate loads from metropolis to metropolis. It simply doesn’t make sense to throw away the opposite 96 cities in your information. The implied counterfactual is that if Philadelphia had continued post-2014 with its earlier sentencing coverage, that its murder charges would appear like this weighted common of Detroit, New Orleans, and New York—however there’s no purpose to anticipate that, as this averaging is chosen by lining up the murder charges from 2010-2014 (truly the counts and populations, not the charges, however that doesn’t have an effect on my common level so I’ll simply speak about charges proper now, as that’s what makes extra sense).
And right here’s the purpose: There’s no good purpose to suppose that a median of three cities that provide you with numbers similar to Philadelphia’s for the murder charges within the 5 earlier years will provide you with an inexpensive counterfactual for tendencies within the subsequent 5 years. To suppose there’s no mathematical purpose we should always anticipate the time collection to work that method, nor do I see any substantive purpose primarily based on sociology or criminology or no matter to anticipate something particular from a weighted common of cities that’s constructed to line up with Philadelphia’s numbers for these three years.
The opposite factor is that this weighted-average factor isn’t what I’d imagined after I first heard that this was an artificial controls evaluation.
My understanding of an artificial controls evaluation went like this. You need to examine Philadelphia to different cities, however there are not any different cities which are identical to Philadelphia, so that you break up town into neighborhoods and discover comparable neighborhoods in different cities . . . and if you’re completed you’ve created this composite “metropolis,” utilizing items of different cities, that capabilities as a pseudo-Philadelphia. In creating this composite, you employ a lot of neighborhood traits, not simply matching on a single final result variable. And then you definately do all of this with different cities in your remedy group (cities that adopted a de-prosecution technique).
The artificial controls evaluation right here differed from what I used to be anticipating in 3 ways:
1. It didn’t break up Philadelphia and the opposite cities into items, jigsaw-style. As an alternative, it shaped a pseudo-Philadelphia by taking a weighted common of different cities. This can be a rather more restricted method, utilizing a lot much less data, and I don’t see it as making a pseudo-Philadelphia within the full synthetic-controls sense.
2. It solely used that one variable to match the cities, resulting in issues about comparability that Wheeler discusses.
3. It was solely completed for Philadelphia; that’s the N=1 drawback.
Researcher levels of freedom, forking paths, and the way to consider them right here
Wheeler factors out many forking paths in Hogan’s evaluation, a lot of data-dependent determination guidelines within the coding and evaluation. (One factor that’s come up earlier than in different settings: At this level, you would possibly ask how do we all know that Hogan’s selections had been data-dependent, as it is a counterfactual assertion involving the analyses he would’ve had completed had the information been completely different. And my reply, as in earlier instances, is that, on condition that the evaluation was not pre-registered, we will solely assume it’s data-dependent. I say this partly as a result of each non-preregistered evaluation I’ve ever completed has been within the context of the information, additionally as a result of if all the information coding and evaluation selections had been made forward of time (which is what been required for these selections to not be data-dependent), then why not preregister? Lastly let me emphasize that researcher levels of freedom and forking paths don’t signify criticisms of flaws of a examine; they’re only a description of what was completed, and normally I don’t suppose they’re a foul factor in any respect; certainly, nearly all of the papers I’ve ever revealed embody many many data-dependent coding and determination guidelines.)
Given all of the forking paths, we should always not take Hogan’s claims of statistical significance at face worth, and certainly the critics discover that numerous various analyses can change the outcomes.
Of their criticism, Kaplan et al. say that affordable various specs can result in null and even reverse outcomes in comparison with what Hogan reported. I don’t know if I fully purchase this—on condition that Philadelphia’s murder fee elevated a lot since 2014, it appears onerous for me to see how an inexpensive estimate would discover that its coverage fee lowered the murder fee.
To me, the true concern is with evaluating Philadelphia to only three different cities. Forking paths are actual, however I’d have this concern even when the evaluation had been equivalent and it had been preregistered. Preregister it, no matter, you’re nonetheless solely evaluating to 3 cities, and I’d wish to see extra.
Not junk science, simply troublesome science
As Wheeler implicitly says in his dialogue, Hogan’s paper isn’t junk science—it’s not like these papers on magnificence and intercourse ratio, or ovulation and voting, or air rage, himmicanes, ages ending in 9, or the remainder of our gallery of wasted effort. Hogan and the others are learning actual points. The issue is that the information are observational, the information are sparse and extremely variable; that’s, the issue is difficult. And it doesn’t assist when researchers are beneath the impression that these actual difficulties will be simply resolved utilizing canned statistical identification methods. In that side, we will draw an analogy to the infamous air-pollution-in-China paper. However this one’s even tougher, within the following sense: The air-pollution-in-China paper included a graph with two screaming issues: an estimated life expectancy of 91 and an out-of-control nonlinear fitted curve. In distinction, the graphs within the Philadelphia-analysis paper all look affordable sufficient. There’s nothing clearly flawed with the evaluation, and the issue is a extra refined concern of the evaluation not totally accounting for variation within the information.