Debunking a quote-mined conclusion from a CDC study – a Facebook response

I have been engaging periodically with an American-style libertarian from Colorado in comments to Governor Jared Polis‘ COVID posts. Tonight he posted a quote-mined line and a link to a CDC study that did not say what he thought it said, and I told him so. Below is a slightly edited version of my response. It’s been edited to added some emphasis that I couldn’t add on Facebook and to incorporate the links properly, but the post is otherwise identical to what I posted on Facebook.

——-

I’d love to address the WHO link, except that you’ve provided a link that goes behind a paywall at Menlo Security, and a search of the quote you reference doesn’t find it online anywhere, so you’ll need to provide a link that is reachable before I can assess it.

As for the CDC link, I’ve seen this one before and it doesn’t say what you think it does. Let’s start with your quote mining and address your misunderstanding of the language used in it.

The full context of the quote is as follows:

“There is limited evidence for [disposable medical masks’] effectiveness in preventing influenza virus transmission either when worn by the infected person for source control or when worn by uninfected persons to reduce exposure. Our systematic review found no significant effect of face masks on transmission of laboratory-confirmed influenza. (emphasis added to identify the section the libertarian quote mined)”

That first sentence rather changes the interpretation of the second, doesn’t it, given the first sentence specifically says that there is evidence that masks work, but that the evidence they analyzed isn’t particularly strong. In fact, they say so directly in the paragraph above the quote you mined, when the authors write “Most studies were underpowered because of limited sample size, and some studies also reported suboptimal adherence in the face mask group.”

What that sentence means is that several of the studies analyzed were too small to draw good conclusions from, and that, because some study participants didn’t wear their masks properly or often enough, that too affected the results.

Second, let’s look at the word “significant.” In this case, the authors are using the word in the conservative, strict sense of being within the 95% confidence interval. This is something you can see from Figure 2 of the paper (referenced in the same section you quote mined from), where they’re showing the 95% confidence interval for each of the studies they analyzed. If you’re unfamiliar with the term, a “95% confidence interval” is the range in which scientists would expect 95% of results to fall within it.

If you look closely at the dashed lines in Figure 2 (which is the average of the studies when weighted to emphasize good studies and de-emphasize poor ones), you’ll notice that the lines are all on the left, “Favors mask” side of the graph. You’ll also notice that the diamond shape, which represents the 95% confidence interval, is overwhelmingly on the left side of the center line. However, because the diamond shape does extend slightly over the center line, there is a small statistical chance that these studies in total might have found that masks don’t work. Thankfully, though, we can calculate what that interval is.

Using T-distribution statistics, we can estimate the amount of the statistical distribution that favors wearing masks vs. the amount that doesn’t. I plugged the data from the study into Excel to make the calculations easier, but what I found was the following:

– For Group A (face mask use alone), the cumulative distribution function shows an 85-87% chance that face masks work, vs. 13-15% chance that they don’t.

– For Group B (face mask and hand hygiene), the cumulative distribution function shows a 79-82% chance that masks work.

– For Group C (face masks with or without hand hygiene), the cumulative distribution function shows a 78-80% chance that masks work.

So is there a chance from this study that masks don’t work? Sure. But the chance is small, and the authors themselves admit that the studies they analyzed aren’t exactly the best studies for the purpose of determining mask effectiveness.

Put another way, contrary to your claim that this study disproves the effectiveness of masks, it nearly does the opposite, but because the data isn’t great, there’s still a small chance that you’re right.