The documentary Trace Amounts explains that in 1999 the CDC conducted a study (often referred to as the Verstraeten study) regarding mercury and major neurological diseases, including autism. The study showed a shocking 7.6 reading for association with autism. This number represents relative risk. It means that a child receiving a vaccine containing Mercury would be 7.6 times more likely to develop autism than a child not receiving the vaccine.
Not surprisingly, this result was unacceptable. So, the CDC “revisited” the study. When it was finished, it had knocked out about half the subjects. Still, the significant relative risk figure would not go away. Even at a reduced level of 2.48, the number was statistically very significant. So, CDC reworked the study a third time and got the figure down to 1.69. Again, that figure was way too high. What parents would want to vaccinate their child if doing so would increase the chance of becoming autistic by 69%?
In 2001, the CDC went back at the data yet another time, getting the relative risk figure down to 1.52. If you’ve been keeping score, this was the third reworking of the study. What now? You guessed it. It produced a fourth revision to the study that brought the relative risk down to 1. It published the study in 2003, four years after its initial findings. Finally, to close the door behind it, the CDC claimed it lost the raw study data. (See Methodological Issues and Evidence of Malfeasance in Research Purporting to Show Thimerosal in Vaccines Is Safe for an analysis of the flaws in this and five other studies the CDC cites to support its assertion of no association between Thimerosal and autism).
As citizens, we want to have faith in our institutions, none more so than the CDC, the entity charged with keeping us from being overwhelmed by dread disease waiting to overrun us. We tend to give the CDC the benefit of any of our doubts.
Still, the sequence of events documented above shouldn’t sit well with any rational person. So, a number of Americans resolve to educate themselves more on this study and on vaccines in general. It’s at this point that most of these intrepid citizen investigators are neutralized by the nature of the CDC’s work. They lack the expertise in science to converse at Ph. D. level. As they wade into study documents, they’re soon derailed by opaque terms such as cohort, interval and association.
These two reasons, trust in our institutions and lack of expertise to form an intelligent scrutiny, virtually shield the CDC from reasonable lay person examination. How much suspicion of this study is truly warranted? Make a mental note at this point, on a scale of 1 (lowest) to 10 (highest) of your suspicion level about the CDC’s study. Next, let’s review a hypothetical study conducted by a school district.
A school district’s dilemma
A school is trying to decide whether adding air conditioning units to the classrooms will improve student scores, something a chorus of parents has been singing about. To install air conditioning in all the classrooms would incur a major initial expense as well as continuing higher costs through the electricity they consume. In short, adoption of air conditioners will cause a major budget imbalance that will have to be addressed through significant program cuts elsewhere.
The school’s Principal meets with two teachers of the same subject and decides to conduct an experiment by having one teacher’s group of 30 students take its final exam in an air conditioned room and the other’s group take its final exam in a non air conditioned room. The final is held on a warm day. The room temperature in the air conditioned room is 72. In the non air conditioned room it is 85.
The scores are tallied for the two classes. They share with the Principal that the scores of the students in the air conditioned room are significantly higher than those in the non air conditioned room. The teachers can see by the Principal’s reaction that he is not pleased. They quickly put an end to the meeting, telling the Principal that perhaps they didn’t double-check their numbers.
The teachers sit down and look at the situation of the students taking the final in the non air conditioned room. They see that three of them came in late. And, even though they easily finished within the allotted time, their scores were lower than average. The teachers decide to throw out the scores of these three students, reasoning that they didn’t follow the test protocol. After all, they weren’t ready to begin the exam on time. The teachers re-average the remaining 27 scores and find that the disparity between the averages is slightly lower. They present their findings to the Principal, who, while pleased to see a smaller difference, is still not satisfied.
The teachers go back and take another look. By thumbing through the exams, they see that three students completed it using blue ink pens instead of the prescribed black ink. The scores of these students is lower than the rest of the class. The decision is made to eliminate their scores because they too didn’t follow the test protocol. The average of the remaining 24 students rises again, but not enough to satisfy the Principal.
The teachers huddle again to discuss their dilemma. Doing some quick math on the remaining scores, they see that the four students in the row nearest the door of the non air conditioned room had lower scores than the class average. Realizing the potential for distraction from people passing in the hallway, these students’ scores are removed. They were, after all, precluded from properly completing the exam due to factors beyond their control. Removing this group and scoring the remaining 20 exams results in a class average that is closer to that of the students taking the final in the air conditioned room, but still not presentable in the opinion of the Principal.
The teachers, tired and exasperated, go back once again to discuss the exam. Soon, they focus on three of the remaining 20 students who didn’t apply themselves all year long. These students said repeatedly that they didn’t like the subject. They missed a significant amount of classes over the course of the year. They failed to hand in a significant number of assignments. Using the rationale that these students essentially failed to meet the requirements of the course, their three scores are dropped. Averaging the scores of the remaining 17 students finally results in an average that is virtually identical to that of the 30-student class taking the final in the air conditioned room.
Weeks later at a school board meeting, the Principal confidently presents his findings that there is no connection between room air conditioners and test scores. To everyone’s relief, he confirms that there is no reason to incur the substantial expense of installing classroom air conditioners.
. . .
If you noticed, the room air conditioner study was reworked four times. Please check in again and assess your level of suspicion of the CDC study on the 1 to 10 scale we used earlier. If it’s gone up, it’s because you realized there’s simply no reason for any legitimate study to be reworked four times.