Tuesday, February 28, 2023

Fool's Gold

I'll freely admit this is the first time I came across the term "Cochrane analysis." Clearly Dr. Hotez is using it here to indicate such an analysis is not a "gold standard," which makes me wonder where the NYTimes reporters got the idea that it is. Probably from one source, which was good enough for them.  Which puts me in mind of my freshman English students when I assigned a research paper and tried to force them to actually do, you know, research.  As opposed to taking the first three hits on Google and writing 2000 words from them.  Now I know that at least some of those students ended up writing for the NYT; or might as well have.

So what is a "Cochrane analysis"?  It's not a universal term for a type of analysis of data.  No, it's this:

A systematic review attempts to identify, appraise and synthesize all the empirical evidence that meets pre-specified eligibility criteria to answer a specific research question. Researchers conducting systematic reviews use explicit, systematic methods that are selected with a view aimed at minimizing bias, to produce more reliable findings to inform decision making.

Except they call it a "Cochrane review":

A Cochrane Review is a systematic review of research in health care and health policy that is published in the Cochrane Database of Systematic Reviews.

Effectively, it's a proprietary term.  It's the "gold standard" because the Cochrane Database of Systematic Reviews says it is.  Again, I'm not trained in science to the degree I can review and critique this database or one of its "reviews," but I do know as someone trained in research techniques that this is as far from the "gold standard" of academic research as you can get.  It may be good; it may be bad; but "gold standard" implies universal acceptance of the method and the results it produces.  And when someone on the internet can show the reasoning behind the study is so poor as to be either intentionally misleading in its conclusions, or the authors are pitifully out of their depth (like the NYT reporters), it really calls in to question the validity of a "Cochrane review."  Which is what Dr. Hotez was getting at.

Start here (or read the whole Twitter thread if you want).  The conclusion of the Cochrane review is that "masks don't work!" (which will come as a surprise to healthcare workers the world over).  The analysis is a bit detailed (as it should be), but it's also Twitter, so how detailed can it be?  But if you can shred the analysis itself in a series of tweets....

There is a pattern in this "review," in other words; and it becomes more and more noticeable, kind of like the reality that FoxNews is a propaganda outlet, not a journalism effort. The study referred to in that tweet supports masks; but, as other tweets point out, the Cochrane review decides it really means masks don't work. The "poor adherence to protocol" was among Muslims on pilgrimage to Mecca. They were given masks, but this was in 2015; pre-covid, IOW. Yeah, seems those facts should be taken into account.  But no, and in fact this study carries great weight in the final analysis.  And then there is the fourth study: And then there's fun with numbers! Remember this is supposed to be a "a systematic review of research in health care and health policy."  Such a review should not weight studies by simply counting them twice: There's even magic involved! Hey, presto! And what do we conclude from this farrago? And it could just be Dr. Hotez knows more about this than we non-experts do. Non-experts like the clueless reporters at the NYt.
at

No comments:

Post a Comment