VAERS a Few Things We Need to Discuss

I've been remiss. There's a topic that I've written about in passing, but I have yet to devote an entire post to it. In all this time writing about different vaccines, studies about vaccines and anti-vaccine claims, there's a subject about which I have neglected to write more about than a paragraph here or there. I'm speaking, of course, about the Vaccine Adverse Event Reporting System, or VAERS. The handful of times I have written anything about VAERS (here, here, here and here), it's mainly been a very brief overview of what it is and how anti-vaccine activists like to abuse it (except that last link, which included a study using VAERS data).

So I thought I should rectify that situation. What prompted this was an exchange on Twitter with a doctor by the name of Jim Meehan, who tried to argue that the HPV vaccine is confirmed to have caused deaths...144 of them, to be precise. His reasoning is that there are 144 reports of death associated with HPV vaccine in the VAERS database. Therefore, he thinks HPV causes death:

He also tried to dismiss me and others by claiming we had financial conflicts of interest on the topic. Actually, he's rather fond of attacking the people he's arguing against, like suggesting that because they don't agree with him, they would probably also deny the Holocaust. (On further perusing his Twitter feed, he appears to be a full-on anti-vaccinationist himself, citing Robert Kennedy Jr.'s nonsense, "too many, too soon", "unvaccinated are healthier" and so on. And on even more perusing, I find that Dr. Meehan admits to being anti-vaccine:

If only I'd known that when I first saw this guy. Would've saved me a lot of time and explains a lot of his behavior. Perhaps at some point in the future I'll need to revisit this fellow.)

In the meantime, though, I thought it might be helpful to talk a little bit about VAERS: what it is, how it's supposed to be used and how it's abused.

What is VAERS?

The Vaccine Adverse Event Reporting System is a surveillance system for monitoring adverse events following vaccinations. The program was established in 1990 in response to the National Childhood Vaccine Injury Act of 1986 (PDF). While vaccine manufacturers and health care providers are required to report adverse event (AE) reports they receive, anyone can make a report to VAERS. Even you and I can report an AE. But VAERS is a passive surevillance system, just accepting reports, rather than proactively looking for issues, as can be done with the Vaccine Safety Datalink.

This might be a good time to go over a couple definitions that come into play. As its name implies, VAERS tracks adverse events. Adverse events are defined as:

Any untoward medical occurrence in a patient or clinical investigation subject administered a pharmaceutical product and which does not necessarily have a causal relationship with this treatment. An adverse event (AE) can therefore be any unfavourable and unintended sign (including an abnormal laboratory finding), symptom, or disease temporally associated with the use of a medicinal (investigational) product, whether or not related to the medicinal (investigational) product.

An adverse event can be just about anything bad that happens following vaccination. It may be caused by the vaccine or be completely unrelated, for example intussusception following RotaShield (VAERS ID 265605-1), an event likely caused by the vaccine, an death from blunt force trauma to the head in an auto accident (VAERS ID 209245-1), an event not at all connected with the vaccine. "Adverse events" should not be confused with "adverse reactions", which are events that are known to be (or have strong evidence suggesting they are) caused by the vaccine. Furthermore, some reports are very thorough, including details from the patient's medical record, while others are nothing but hearsay ("I heard from X" or "I read on the web that Y").

Reports to VAERS are uncontrolled and really require further investigation to determine whether or not the vaccine was causally related to the AE. Before users are allowed to access the data, they must indicate that they have read and understood a statement detailing the limitations of VAERS data:

VAERS is a passive reporting system, meaning that reports about adverse events are not automatically collected, but require a report to be filed to VAERS. VAERS reports can be submitted voluntarily by anyone, including healthcare providers, patients, or family members. Reports vary in quality and completeness. They often lack details and sometimes can have information that contains errors.

"Underreporting" is one of the main limitations of passive surveillance systems, including VAERS. The term, underreporting refers to the fact that VAERS receives reports for only a small fraction of actual adverse events. The degree of underreporting varies widely. As an example, a great many of the millions of vaccinations administered each year by injection cause soreness, but relatively few of these episodes lead to a VAERS report. Physicians and patients understand that minor side effects of vaccinations often include this kind of discomfort, as well as low fevers. On the other hand, more serious and unexpected medical events are probably more likely to be reported than minor ones, especially when they occur soon after vaccination, even if they may be coincidental and related to other causes.

A report to VAERS generally does not prove that the identified vaccine(s) caused the adverse event described. It only confirms that the reported event occurred sometime after vaccine was given. No proof that the event was caused by the vaccine is required in order for VAERS to accept the report. VAERS accepts all reports without judging whether the event was caused by the vaccine.

DISCLAIMER: Please note that VAERS staff follow-up on all serious and other selected adverse event reports to obtain additional medical, laboratory, and/or autopsy records to help understand the concern raised. However, in general coding terms in VAERS do not change based on the information received during the follow-up process. VAERS data should be used with caution as numbers and conditions do not reflect data collected during follow-up. Note that the inclusion of events in VAERS data does not infer causality.

The limitations statement is something that anti-vaccine activists tend to skip over. Some organizations even link directly to the searchable database or provide their own form to search it, bypassing this important information. They do link to the disclaimer, but fail to require users read and understand it before searching the database, but more on that later.

As the disclaimer points out, the reports are not evidence of causation. They are only confirmation that event Y is reported to have occurred after getting vaccine X. In addition, the reports can give a misleading view of the rate of AEs following vaccination. Because it is a passive surveillance system, VAERS is prone to underreporting. The less severe the event, the less likely it is to be reported. More serious events are more likely to be reported, but there is no guarantee, especially since not all people are required to report.

Another issue with the manner of reporting is that the system can be gamed or skewed, with more people reporting events than there are real reactions. This can be driven by media [ETA link] (more popular reporting of reactions leading to more people thinking that they may have had the same reaction), by litigation ("Were you or a loved one injured by vaccines?"), and there is even potential for an organized, dedicated movement to falsely inflate AE reports; anyone can report, for any reason. You can even report that a vaccine turned you or a loved one into The Hulk or Wonder Woman.

Making matters a bit worse is that even after a report is investigated, its entry in VAERS is not necessarily updated with the new findings (see that disclaimer: "VAERS data should be used with caution as numbers and conditions do not reflect data collected during follow-up."). That means that when you read a report in VAERS, further inquiry may show that the vaccine was not related at all, even though the initial report may have strongly suggested it was. It may also show that a report that leaves room for doubt actually was caused by the vaccine. This makes it extremely risky to rely on the raw reports as conclusive evidence.

Needless to say, it is very important to understand the limitations of the data. VAERS is, essentially, a collection of anecdotes. Sure, some of them are pretty detailed and well-documented, but there are no controls in place to weed out the various confounding variables, biases and other pitfalls that accompany personal stories.

VAERS! Huh! What is it good for?

So the VAERS database contains lots of stuff, some of it valuable, some of it...not so much. You might be wondering what the heck it's good for, then. How is it supposed to be used? As mentioned above, the purpose of the program is to monitor vaccines for safety once they are on the market. The goals of the program are to:

1. Detect new, unusual, or rare vaccine adverse events (VAEs);
2. Monitor increases in known adverse events;
3. Identify potential patient risk factors for particular types of adverse events;
4. Identify vaccine lots with increased numbers or types of reported adverse events; and
5. Assess the safety of newly licensed vaccines.

The database is a decent tool for researchers (manufacturers, academics, government), with the aforementioned limitations in mind. It is a springboard, a jumping off point for further, more rigorous research. Rather than being an ends in and of itself, VAERS serves as a starting point, hinting at which vaccine-adverse event associations need more study.

Because the data is uncontrolled and of wildly varying quality, it is impossible to conclude from the raw reports alone whether a vaccine caused an event or not. Not that that stops anti-vaccine activists from trying to use VAERS to claim causation.

VAERS a time and a place for everything, but honey, this ain't the time or the place

Like I mentioned at the beginning of this post, a common tactic among anti-vaccine activists is to cite VAERS raw reports as conclusive proof that vaccines cause all manner of horrible things. They go about this in a few different ways,

Some anti-vaccine folk, like the good Dr. Meehan up there, point to the reports as confirmation that the vaccines caused the adverse event. They might see a report like this (VAERS ID 320559-1):

Initial and follow up information has been received from a physician, who was told by one of his patients, that the mother of a consumer was told by a neurologist that there were 4,400 kids who have died following vaccination with GARDASIL. No product quality complaint was involved. Attempts are being made to verify the existence of patients. Additional information has been requested.

...and say that it is proof that the vaccine causes death. They ignore the poor quality that renders it essentially useless for drawing any kind of conclusion. Or they may try to rationalize away problems with the report. For example, there's 382971-1, in which the patient died from malaria about a month after the second dose of Cervarix. Rather than accept that malaria is responsible, they try to place blame on the vaccine, arguing that the vaccine damaged the patient's immune system (how?). While it's possible that the vaccine is connected, it's unlikely. Barring more rigorous, controlled study, you can't state unequivocally that there is a causal connection.

In a similar vein, people might search for a specific event reported after a specific vaccine. They might have a single event in mind (e.g., death) and try different vaccines until they get hits they like, or they might pick a vaccine and try different outcomes. Whatever method they use to anomaly hunt, they get a list, which they point to as evidence that the vaccine is bad or dangerous. Not surprisingly, there are a couple problems with this approach that are also conveniently ignored. First, the reports are accepted at face value, as above, if they're even read at all. Even if some portion of the reports are valid, we do not know what percentage have a valid, confirmed causal connection.

That brings us to a third way that anti-vaccine activists abuse VAERS. When they get their list, they might present it along the lines of "there have been 144 deaths associated with HPV vaccine!" In addition to ignoring any possible problems with the reports (problem #1) and failing to confirm which reports are valid and which are not (problem #2), they don't provide any meaningful context. Even for real causal connections between an event and a vaccine, what does that mean in terms of the population? As Scott Gavura wrote, what's the denominator? VAERS data can't be used to determine incidence or prevalence of events.

Here's what I mean. Suppose we have thrombosis as the event of interest, and we look at DTaP. We look this up in VAERS and find, hypothetically, that there are 10 reports of thrombosis following DTaP. To keep things simple, we'll assume that there were no other medications or vaccines administered and no other concomitant diseases or conditions. That's 10 cases out of what? All people who received the vaccine? How many got the vaccine but didn't suffer the event? How many got the vaccine and had that event occur, but never reported it? How many suffered thrombosis before getting the vaccine? How many got the vaccine, but didn't suffer thrombosis until months, years or decades later? Or is it out of doses administered, which, in the case of DTaP? That number will be significantly higher than the number of people who received the vaccine, since it is given in a 5-dose series.

No matter how much the limitations of VAERS are pointed out, or how much we explain the flaws in how they're using the data, anti-vaccine activists cling to their belief that VAERS offers solid proof that vaccines are bad and cause nearly every malady under the sun. Well, maybe some know that they are misusing the data, but they have an agenda to push. They have to scare people away, even if that means distorting or misrepresenting the data.

In my back-and-forth with Jim Meehan over his claim that VAERS confirmed that HPV vaccine causes death, I focused mainly on the point that VAERS does not and cannot prove that a vaccine caused the given event. His response was to claim that VAERS does not disprove that a vaccine caused the event, either (but that's sort of a "no duh!", since VAERS was not designed to disprove causation), therefore it's evidence that the vaccine does cause the event. He argues that unless his critics can go through each and every report and explain why it does not show causation, then it should be viewed as confirmation of a causal connection. Rather than providing evidence to support his own claims, he tries to place the burden on others to prove him wrong. This is not much different than the creationist approach of arguing that if science or evolution can't explain something, then it must have been God. Thus far, of the 144 reports of death following HPV vaccine that he claims he evaluated, I've only seen him admit that one can be dismissed, after someone confirmed with CDC that it was a duplicate report. Pointing out the flaws is not the same as denying that there is a connection. Rather, it's simply saying that there is insufficient evidence to conclude a connection. That takes more details than are available in the reports and further research, like this. Or this. Or this. Or this.

VAERS this leave us?

The Vaccine Adverse Event Reporting System is an important part of our vaccine safety program. It can help serve as an early warning system and guide regulators to conduct controlled research into various safety questions. VAERS data can provide hints that there might be an increased risk of a reaction following vaccination, but it takes further, more rigorous study to determine if that risk is real, or if it is just a coincidence. That question can't be answered by VAERS itself, no matter how much anti-vaccine activists would like you to believe otherwise. The next time you hear someone citing VAERS in support of their claim that vaccines cause some sort of injury, think about the limitations inherent in the data and how the data is often misused by those who either do not understand it or who purposefully misrepresent it. And the more ardently they cling to their VAERS-based claim, the more black-and-white their argument, the more likely it is that they're wrong.