Bayesians transferring from protection to offense: “I actually suppose it’s sort of irresponsible now to not use the data from all these 1000’s of medical trials that got here earlier than. Is that very radical?”
Erik van Zwet, Sander Greenland, Guido Imbens, Simon Schwab, Steve Goodman, and I write:
We have now examined the first efficacy outcomes of 23,551 randomized scientific trials from the Cochrane Database of Systematic Opinions.
We estimate that the nice majority of trials have a lot decrease statistical energy for precise results than the 80 or 90% for the acknowledged impact sizes. Consequently, “statistically important” estimates have a tendency to significantly overestimate precise therapy results, “nonsignificant” outcomes typically correspond to vital results, and efforts to duplicate typically fail to realize “significance” and will even seem to contradict preliminary outcomes. To handle these points, we reinterpret the P worth by way of a reference inhabitants of research which might be, or might have been, within the Cochrane Database.
This results in an empirical information for the interpretation of an noticed P worth from a “typical” scientific trial by way of the diploma of overestimation of the reported impact, the likelihood of the impact’s signal being improper, and the predictive energy of the trial.
Such an interpretation offers further perception concerning the impact beneath examine and may guard medical researchers in opposition to naive interpretations of the P worth and overoptimistic impact sizes. As a result of many analysis fields endure from low energy, our outcomes are additionally related outdoors the medical area.
Also this new paper from Zwet with Lu Tian and Rob Tibshirani:
Evaluating a shrinkage estimator for the therapy impact in scientific trials
The primary goal of most scientific trials is to estimate the impact of some therapy in comparison with a management situation. We outline the signal-to-noise ratio (SNR) because the ratio of the true therapy impact to the SE of its estimate. In a earlier publication on this journal, we estimated the distribution of the SNR among the many scientific trials within the Cochrane Database of Systematic Opinions (CDSR). We discovered that the SNR is usually low, which means that the ability in opposition to the true impact can be low in lots of trials. Right here we use the truth that the CDSR is a set of meta-analyses to quantitatively assess the results. Amongst trials which have reached statistical significance we discover appreciable overoptimism of the same old unbiased estimator and under-coverage of the related confidence interval. Beforehand, we’ve proposed a novel shrinkage estimator to deal with this “winner’s curse.” We examine the efficiency of our shrinkage estimator to the same old unbiased estimator by way of the foundation imply squared error, the protection and the bias of the magnitude. We discover superior efficiency of the shrinkage estimator each conditionally and unconditionally on statistical significance.
Let me simply repeat that final sentence:
We discover superior efficiency of the shrinkage estimator each conditionally and unconditionally on statistical significance.
From a Bayesian standpoint, that is no shock. Bayes is perfect in case you common over the prior distribution and could be affordable if averaging over one thing near the prior. Particularly affordable compared to naive unregularized estimates (as here).
Erik summarizes:
We’ve decided how a lot we achieve (on common over the Cochrane Database) through the use of our shrinkage estimator. It seems to be a few issue 2 extra environment friendly (by way of the MSE) than the unbiased estimator. That’s roughly like doubling the pattern measurement! We’re utilizing comparable strategies as our forthcoming paper about meta-analysis with a single trial.
Folks generally ask me how I’ve modified as a statistician through the years. One reply I’ve given is that I’ve step by step change into extra Bayesian. I began out as a skeptic, involved about Bayesian strategies in any respect; then in grad college I began utilizing Bayesian statistics in purposes and realizing it might resolve some issues for me; when writing BDA and ARM, nonetheless having the Bayesian cringe and utilizing flat priors as a lot as potential, or not speaking about priors in any respect; then with Aleks, Sophia, and others transferring towards weakly informative priors; finally beneath the affect of Erik and others making an attempt to make use of direct prior info. At this level I’ve just about gone full Lindley.
Simply as a comparability to the place my colleagues and I are actually, take a look at my response in 2008 to a query from Sanjay Kaul about easy methods to specify a previous distribution for a scientific trial. I wrote:
I suppose the very best prior distribution can be based mostly on a multilevel mannequin (whether or not implicit or specific) based mostly on different, comparable experiments. A noninformative prior may very well be okay however I desire one thing weakly informative to keep away from your inferences being unduly affected by extraordinarily unrealistic potentialities within the tail of the distribuiton.
Nothing improper with this recommendation, precisely, however I used to be nonetheless leaning within the route of noninformativeness in a method that I’d not anymore. Sander Greenland replied at the time with a suggestion to make use of direct prior info. (And, only for enjoyable, here’s a discussion from 2014 on a subject the place Sander and I disagree.)
Erik concludes:
I actually suppose it’s sort of irresponsible now not to make use of the data from all these 1000’s of medical trials that got here earlier than. Is that very radical?
That final query jogs my memory of our paper from 2008, Bayes: Radical, Liberal, or Conservative?
P.S. Additionally this:
You’ll be able to click through to see the entire story.