The Fine Art of Baloney Detection

Carl Sagan, the famous author and astronomer dedicated much of his life to astrobiological explanations for the origin of the universe. In particular, he rallied strongly against religious and spiritual claims of many sorts, being a vocal advocate of rational scientific rigor, if not direct atheism. Even though he spoke out against pseudo-science and creationism, he replaced them with a sense of wonder for the natural mysteries that surround us. Before he passed away in 1996, he wrote The Demon Haunted World which summarized many of his arguments for studying the world through scientific eyes. In one section, "The Fine Art of Baloney Detection", he provides a toolkit for skeptical thinking, which is paraphrased below. I find it to be a thought provoking guide to research ethics and I keep it posted by my desk.

Baloney Detection Toolkit

1. Facts must be independently corroborated.
2. The evidence itself should be debated by knowledgeable people from many differing points of view.
3. Do not weight the veracity of arguments by who is giving them. "Authorities have made mistakes in the past. They will do so again in the future." (I am personally guilty of this one, falling victim to hero worship in science)
4. Create multiple hypotheses and come up with tests to disprove them. Also, do not get attached to a hypothesis just because it is yours.
5. Use quantitative evidence. (although to this, I would add, be careful of abuse of statistics.)
6. Every link in a chain of arguments must be sound.
7. Occam's Razor is a good way to rule out hypotheses.
8. The hypothesis must be testable. (My theory of universal creation is that the world appeared in 1987, just as it was, with a sense of the pre-1987 recorded in our brains and physical world. - disprove that!).
9. Control experiments are essential.
10. Variables must be separated. If you're testing the effect of room size and color on the mating frequency of rabbits, vary the color at a constant room size or the room size at a constant color.
11. To avoid observer bias, do double-blind experiments.

Arguement Don'ts

Sagan also provides a list of things to avoid when arguing a hypothesis. Many of these are geared more towards spiritual myth-debunking, but can be applied to scientific questions as well. Here are some of his don'ts:

1. 'ad hominem' - Dont attack the arguer. Attack the argument.
2. argue based on the adverse consequences of the decision (Sagan proffers the example of convicting a defendent in a publicized murder trial to avoid encouraging other similar episodes.)
3. claim that because something has not been proved false, it must be true. (alien abduction)
4. count the hits and forget the misses - Francis Bacon's 'enumeration of favorable circumstances'
5. misuse statistics with small sample sizes, or misunderstand their nature. I love Sagan's example: "President Dwight Eisenhower expressing astonishment and alarm on discovering that fully half of all Americans have below average intelligence." hee...
6. argument non sequiturs, and similarly post hoc arguments where B happened after A, therefore A caused B. Also, confusing correlation with causation (Do violent kids watch more violent TV? or does violent TV make kids violent?)
7. create a false dichotomy, where you argue only extremes of a continuum of possibilities. Related problems are misuse of slipperly slope arguments (without backing evidence) and the mixing of long-term and short-term arguments.

Carl Sagan's original text of "The Fine Art of Baloney Detection" can be found in multiple places online and I encourage you to read the full piece.