Politics Pseudoscience Science Skepticism/critical thinking

More skepticism about the “psychotics prefer Bush” study

Yesterday, I explained why a study that purports to show that psychotic patients tended to vote for President Bush in the 2004 election and is presently making the rounds to snarky gloating through the left-wing blogosphere is so utterly flawed that almost certainly does not mean what the author claims it does, given the data dredging, small sample size, and the failure even to consider alternative hypotheses to explain the observations. In my discussion, I complained that I had only found one skeptical take on the study among the credulous acceptance and use of the study to imply (or outright state) that Bush supporters are mentally ill.

Now I’ve found another skeptical take by Alon Levy. He emphasizes the small sample size and emphasizes, more than I did, the nonrandom sampling of the large population about which the study purports to make observations. He also goes on to discuss the problems with metastudies (the kind that are often presented as further evidence of a link between conservatism and various mental pathologies):

The buzzword in areas of social science that generate numerous studies is “meta-study.” It’s easy to botch these too, but when something draws enough buzz for there to be a hundred different data sets about it, a political hack will be able to use five that show the correlation is statistically significant.

In fact, usually there will be many more than five, because published studies have an existing bias in favor of data sets that show significant correlation. “There’s no link between these two things” won’t get you published unless it’s a real hot-button issue like racial IQ differences, and even that only diminishes that bias but does not eliminate it.

This is the same sort of problem we have with metanalyses in medicine and is known as publication bias. The problems with metanalyses is a topic I’ve been meaning to blog about for a while now. Maybe next week. But in the meantime, I’ll share a quote I heard from renowned burn surgeon Dr. Basil Pruitt while listening to SESAP 11:

“I have viewed metanalysis and its frequent use as sort of a means of turning a lady of the night into a Vestal Virgin.”


I’m not meaning to say that metanalyses can’t produce valid results (many do), but physicians tend to forget that, according to the time-honored computer principle of GIGO (“garbage in, garbage out”), metanalyses are no better than the studies upon which they are based. They need to remember that, and evaluate metanalyses accordingly. So does anyone else who reads and cites such studies.

By Orac

Orac is the nom de blog of a humble surgeon/scientist who has an ego just big enough to delude himself that someone, somewhere might actually give a rodent's posterior about his copious verbal meanderings, but just barely small enough to admit to himself that few probably will. That surgeon is otherwise known as David Gorski.

That this particular surgeon has chosen his nom de blog based on a rather cranky and arrogant computer shaped like a clear box of blinking lights that he originally encountered when he became a fan of a 35 year old British SF television show whose special effects were renowned for their BBC/Doctor Who-style low budget look, but whose stories nonetheless resulted in some of the best, most innovative science fiction ever televised, should tell you nearly all that you need to know about Orac. (That, and the length of the preceding sentence.)

DISCLAIMER:: The various written meanderings here are the opinions of Orac and Orac alone, written on his own time. They should never be construed as representing the opinions of any other person or entity, especially Orac's cancer center, department of surgery, medical school, or university. Also note that Orac is nonpartisan; he is more than willing to criticize the statements of anyone, regardless of of political leanings, if that anyone advocates pseudoscience or quackery. Finally, medical commentary is not to be construed in any way as medical advice.

To contact Orac: [email protected]

Comments are closed.


Subscribe now to keep reading and get access to the full archive.

Continue reading