Early Pregnancy Flu Shots: New Research Hints of Autism Link
A Nov. 19 study, of 45,231 women, published in JAMA Pediatrics, identified a heightened risk of autism spectrum disorder (ASD) diagnosis in the children of mothers who received a flu shot during their first trimester of pregnancy. The study, Association Between Influenza Infection and Vaccination During Pregnancy and Risk of Autism Spectrum Disorder, was authored by Ousseny Zerbo and his colleagues affiliated with the Division of Research at Kaiser Permanente.
While the researchers found no increased risk when the mother received flu shots in the second or third trimester, the data demonstrated a 20 percent higher risk of an autism spectrum disorder among children of mothers receiving the flu vaccine during the first trimester. That risk was statistically significant. (The P value, .01, indicates a 99 percent likelihood that the result isn't due to chance.)
However, after completing this analysis, the authors made a series of adjustments that have drawn criticism from other scientists. Most controversial was their questionable decision to apply a statistical device called the "Bonferroni Correction" to their data. Statisticians use the Bonferroni Correction in very specific circumstances—where they seek to reduce the chance for false positives in calculations involving multiple comparisons. The impact of the Bonferroni Correction is nearly always conservative; it dampens signals in data sets. In doing so it creates the risk of missing true associations. When applied to the first trimester flu vaccine dataset, the Bonferroni Correction reduced the significance of the association from 99 percent to 90 percent. Despite the fact that the adjusted result was still considered marginally statistically significant, the authors then made a second dodgy judgment, by declaring that, "this association could be due to chance."
These sweeping decisions allowed the authors to arrive at the questionable conclusion that, "There was a suggestion of increased ASD risk among children whose mothers received influenza vaccinations early in pregnancy (first trimester), although the association was insignificant after statistical correction for multiple comparisons." The researchers summed up with an acknowledgement of the uncertainty of their conclusion: "We believe that additional studies are warranted to further evaluate any potential associations between first-trimester maternal influenza vaccination and autism."
National media outlets universally missed that nuance. Journalists widely reported the study as a decisive exoneration of flu shots. NPR declared: Flu Shots Don't Increase Autism Risk In Pregnancy. Fox News celebrated: Flu—or flu vaccine—in pregnancy not tied to autism in kids. The Scientist headlined: Autism Not Linked to Flu or Flu Shot During Pregnancy, while the New York Daily News assured: No link between flu or flu vaccine in pregnancy and autism: study.
As the mainstream media celebrated, public health advocates and scholars cried "foul." Dr. James Lyons-Weiler, PhD, the CEO and director of the Institute for Pure and Applied Knowledge, and data manager of more than 100 biomedical research studies, told me that the author's "incorrect" and "unorthodox" application of the Bonferroni Correction in this circumstance risked the appearance that they were using improper methodologies to, "make an unwanted but statistically significant finding vanish in a sea of statistical wizardry."
Sander Greenland, professor of Statistics and Epidemiology at UCLA's School of Public Health and College of Letters and Science, agreed that the use of the Bonferroni Adjustment was inappropriate in this context. Greenland is among America's preeminent statisticians with more than 300 peer reviewed publications—two of which have been cited more than 500 times. He is editor of the Dictionary of Epidemiology.
Greenland said the research team's use of Bonferroni makes no sense "where there are finely correlated outcomes" and where the cost of a false negative is high—the possibly erroneous conclusion that first trimester flu shots are safe. (See at the end of the post Dr. Greenland's detailed explanation of the Bonferroni and why it was inappropriate.)
Greenland observes that "in a context like this, it's something that's usually called up, after the fact, when they get some significance like this, where they don't like it and they want to see if they can get rid of it that way. It's obvious why they used it. It makes the so-called significance go away and, of course, that's the goal. They're trying to make things go away…that's sort of a standard strategy now—by a large segment of the pharmaceutical experts that try to get rid of things. They didn't like the results and they jumped on it with the Bonferroni. It's not appropriate here." Greenland added, sympathetically, that the deception was probably not deliberate, "I don't think they think this out loud in their minds, it's just completely Freudian unconscious."