Where do ideas come from?
A lot of damage has been done to the human spirit through the false equation of skepticism and science. The skeptical mind is a mentality which requires more stringent proof of propositions which contradict a ruling paradigm than those which follow it. This is not a scientific approach. It's understandable why this mentality develops in order to try to distinguish between false knowledge and truth, but its effect is to perpetuate a theory to the extent that it becomes a dogma. This is because in true science there is no such thing as "proof". Proof is a mathematical concept, and experimental science, although it uses mathematics as one of its most important tools, does not allow for the "proof" of its propositions. Science relies for its effectiveness on a different form of enquiry - the hypothesis.
A scientific "theory" such as evolution by natural selection, or the Big Bang, is nothing more than a model which is intended both to explain experimental results and allow for the generation of testable hypotheses or predictions. Crucially, the success of a theory depends far less on its explanation of current experimental results than on the confirmation of its predictions - but no matter how many of these predictions are confirmed, the theory can never be said to have been "proven".
For example, I may have come up with a theory that the universe is shaped like a gigantic tetrahedron. This theory may be able to account for the existing experimental evidence, but in order for it to be a useful theory, I need to make predictions, based on my model, of future experimental results. This allows my theory to be tested. So I might say "If the universe is shaped like a giant tetrahedron, then the furthest visible galaxies should demonstrate an orderly distribution along the projected vertices." If experimental investigation shows that this does indeed seem to be the case, then this is taken as evidence in favour of my theory. If this distribution is not found, this is taken as evidence against my theory. However, in neither case is my theory said to be proven or disproven. It is simply judged according to its usefulness in making predictions, because, after all, that is what science is supposed to do. It gives us the power to predict the future, in a way, and to create technology. It is not supposed to create philosophy or express truth.
Some theories, such as quantum theory, or the theory of gravity, become so successful, and give so much power to make successful predictions, that they become widely accepted as "reality". This is inevitable, because when we observe a result (an apple falling to the ground) and create a theory to explain it, we give a name to the theory, and in this case the force (gravity). When something has a name, we tend to think of it as real, even though no one has ever seen (or experimentally detected) gravity. All we have seen are "its" effects - an apple falling towards the ground, or planets orbiting a star.
The longer a theory such as this remains in place, the harder it becomes to gain acceptance for an alternative explanation, because, more and more, evidence which might be in favour of a new theory gets "explained" in terms of the existing theory, even if a bit of a stretch has to be made to do it. This is what happened for a long time as mediaeval astronomers tried to work out the (apparently) strange paths of the other planets. Evidence was mounting all the time that contradicted the Earth-centric theory, but astronomers tried to "squeeze" this evidence into their existing theory by inventing new concepts such as retrograde motion, rather than admit that perhaps the Earth-centric model was flawed. This is the difference between science and dogma - if retrograde motion had been predicted by the Earth-centric model, and subsequently confirmed by observation, that would have been science. What happened - the creation, in hindsight, of a concept to shore up a shaky paradigm, was not science, and stemmed from the inability or unwillingness of astronomers to recognize that their theory was just a theory, not a reality1.
Even when one theory supersedes another, however, it makes no sense to say that the previous theory has been "disproved". Einstein, for example, did not "disprove" Newton, even though Einstein's theory of General Relativity turned out to be a more accurate model of gravity than Newton's. Newton's equations are still incredibly accurate, especially for terrestrial phenomena, so it cannot be said that he was "wrong", just that his theory was slightly less powerful than Einstein's. Neither theory represents the "truth" about gravitational effects, and anyone who thought so would be highly embarrassed when the next, even more accurate theory came along. Inconsistencies in Einstein's model are known, especially the difficulty of resolving it with the equally powerful Quantum Theory, and physicists are currently involved in the search for a Unified Field Theory which will link these two theories into a complete model of the Universe and its laws. In fact, the Unified Field Theory has become such a Holy Grail that everyone now presumes that it must "exist", just waiting for us to find it, and it is therefore in danger of being the only theory that became a dogma without actually coming into being.
So what has all this got to do with skepticism? Well, let's ask another question first. Why is it proving so difficult to come up with a Unified Field Theory?
This is not a simple question. It brings us to the heart of scientific method and our modern dilemma. Physicists are in the strange situation nowadays of having two extremely powerful theories describing different "aspects" of the laws of the universe - large-scale and small-scale, to put it very simply - each of which has been repeatedly strengthened by countless experimental confirmations and predictions, but which seem incapable of being resolved into one all-inclusive theory. It seems that everyone just knows that this theory must exist, but all attempts to discover or create it have, so far, failed2.
So why is it proving so difficult? We've been waiting for nearly a century - and not just waiting, either. Many of the finest minds in science have worked on this question, some for their entire lives. Einstein famously spent the last thirty years of his life searching for it. One of the problems is simply that there is so much data. Any new theory is going to have to be able to account for all experimental data amassed so far, and there is just so damn much of it by now that no human person can be aware of it all. Also, it is not at all clear where a new theory even comes from. Where do we get our ideas? If the Unified Field Theory has not yet been found, where is it going to come from, and why are we so sure that it even exists? Why is it even needed?
In the biographies of scientists, there is almost always one feature in common, and with which every human being has had some experience - the moment in which a new idea strikes the mind. The archetype of this moment is the story of Archimedes in the bath - Eureka! - but there are countless such stories. Newton and the apple. Kekule dreaming of a snake biting its own tail, and waking up with the structure of the benzene ring in his mind. Francis Crick's sudden visualization of DNA's double helix pattern.
Once the idea hits, then the theory is there, and it's time to generate hypotheses to test it. The rest is easy, because the scientist invariably "knows" almost immediately, and instinctively, that it must be correct. Mathematicians are also familiar with this feeling of the "rightness" of a theorem, which often encourages them through difficult times when a rigorous proof is elusive. So where does this sense of "rightness" originate? Where do ideas come from?
Skepticism requires new ideas to prove themselves, which is not in itself wrong. However, what I am calling the skeptical mind is not the "beginner's mind" of Zen, which is empty of preconceived ideas, but rather a mind which contains the current prevailing theory, and tests new theories against it. The burden of proof placed on the new theory, the new idea, is sometimes carried to the extent that actual evidence contradicting the prevailing theory is dismissed, or explained away. In other words, if a new theory contains this contradictory evidence, but is considered to be flawed in other respects, the evidence is often ignored along with the theory. This may be because once you have a sufficiently large body of knowledge and evidence that you are trying to contain within one theory (the prevailing one), the mind, and the theory, reach a saturation point at which they simply cannot accomodate any more information without collapsing. As this point is neared the burden of "proof" for new theories is made heavier and heavier, because thinkers and scientists - and, in a way, all human beings, will not allow the collapse of an existing theory until a new theory appears to replace it. We fear the unknown just as much now as when we were living in caves, amazed and puzzled by the stars and what we could see moving up there.
So here we all are, waiting for the Unified Field Theory that Hawking and Einstein and all the rest promised us, while the contradictions and weirdnesses and alternative theories pile up, mostly unanswered and unfairly ignored. We're all tired of hearing about the latest Russian electromagnetic healing device and/or weapon, because we have been given no paradigm by which to understand or dismiss such things. It must be fake, we think, except there are those pesky testimonials. "This machine really works!" they say, and we shake our heads, wondering who to believe. You can't really believe the Russians, we all "know" in the back of our minds. If it was announced on CNN, then maybe...wouldn't CNN report it, if it was real? Wouldn't they?
We're tired of reading about the latest evidence of government cover-ups, the latest conspiracy theories, crop circles and UFO sightings. What are we supposed to do with them? The evidence is there, overwhelmingly, but the theory is missing. It's like trying to watch a 3D movie without the glasses. It will just look fake and annoy you and give you a headache. If someone gave us the glasses, we could sit back and enjoy it - after all, we've paid our money, we're in the theater - we want the movie to make sense! In the same way, if a new theory, a new mental or historical paradigm emerged tomorrow, elegant and wild enough to accomodate all the stuff we don't know how to think about, we would surely embrace it. None of us are comfortable with the way things are. We want to understand and accept, but without a theory, we don't know how. We all say "I'll believe it when I see it," but without a theory, even when we see it, we don't know how to believe. "I must have hallucinated." "There must be a scientific explanation."
Why are we so limited? Because to admit to the existence of phenomena for which there is no current theory is to lose power over the world. That absolute power to predict, which has elevated humans above all other Earth-bound species, disappears when you say "This is happening, but we can't explain it," because the next question is, "If we can't explain this, how do we know we were right about anything else?" That's why, right after saying "The king is dead," they always say "Long live the king." There must be no power vacuum. If the king dies and there is no new king, what follows is what we fear most: chaos.
The skeptical mind sees itself as struggling against this chaos: pseudo-science, irrationality, absurdity, delusion and falsehood. However, in setting itself in opposition to chaos in this way it also sets itself agains the very source of theory and ideas: human intuition. The truth is, nobody really knows where ideas come from, but nobody can deny that they happen. So you can look at this phenomenon of ideas and say "It's intuition," or "It's the subconscious mind at work," or "It's Divine Inspiration," but in the end it doesn't matter what you call it as long as you recognize that it happens to people, and no one knows exactly why or how. Completely "scientific" ideas often strike people in the most sudden and unusual ways, during random activities - crossing the street, napping by the fire, sitting under a tree. The skeptical mind, in assuming a defensive stance against a perceived chaos at the edge of accepted theory, risks cutting itself off from the truth and order that it really wants - and, in the end, this defense is unnecessary. The scientific method is so robust that falsehood and pseudo-science will always die a natural death.
People who work on the fringes of science are often dismissed as "cranks", with the scorn usually directly proportional to the divergence between their theory and the prevailing one. The more rigorous and academic their presentation, the better their chances of receiving serious attention rather than scorn, but even so this is no guarantee of mainstream acceptance, especially by the established scientific community, most of whom will have spent their lives working within the current framework. One of the problems is that if someone is dismissed as a crank, their evidence, even if genuine, usually gets thrown out or ignored as well. This happened with Erich Von Daniken, who fabricated some of his evidence supporting his claim that the Earth had been visited by extraterrestrials in the past. In rightly condemning Von Daniken's "sin" against science, most people commit a cardinal sin themselves by ignoring his unfaked, verifiable evidence, as if it had become tarnished by association. It was left for others such as Graham Hancock to pick up where he left off, and similar things happen to Hancock, who has difficulty persuading "serious" scientists to examine the evidence he collects. Evidence is evidence, waiting for a theory to explain it, like Darwin's fossils and the movement of galaxies away from each other. No new theory can emerge as long as the "weird" stuff is ignored, because the reason we know that the prevailing theories are incomplete is because of the weird stuff. New theories begin at the periphery, not the supersaturated centre.
So the king is looking very ill indeed, and there is no heir so far. This means that the chaos is almost on top of us, and I think everyone can see this all around them at the moment, on both the small and the large scale. There is no consensus or unifying explanation, it seems, for anything any more, and some people are realizing that there may never have been; that all we have had have been local and temporal illusions of order, minor victories of control over chaos. The skeptical mind, overstrained, begins to shut down as the chaos of irrationality and "weird science" builds, and this is why there is an increasing public perception of the mainstream scientific community as intellectually bankrupt, divorced from reality, with a narrow understanding of their own field and little else. It wasn't always like this. Modern science replaced God and scientists took on the power and respect of priests because of the unprecedented predictive power and control that scientific method gave to human beings. So if God is dead and modern science is dying, what is going to take their place?
Sometimes you just have to start again.
Where do ideas come from?
1 - In this particular case, it is possible that many astronomers were held back from their natural inclinations by the restricting influence of the Roman Catholic Church, which has had a history of resisting advances in scientific theory, and whose power at this time was very great. This would be a case of one dogma leading to another, and then another. The Church were afraid that accurate scientific models of the universe would undermine the authority of the Bible.
2. Some attempts have met with more success than others, at least in explanatory (rather than predictive) terms - one of the more durable attempts has been the so-called "string theory", which has developed through a couple of incarnations into the more complex "M-theory". This theory, though incomplete, is already highly abstract, and requires complex mathematics even to understand its basic principles. It may be that in time they will, like the theories of Relativity and Quantum Mechanics, be both experimentally confirmed and more easily expressible in everyday language, but this remains to be seen. Someone once described M-theory as a piece of 21st century science that fell into the 20th century by accident, but (IMHO) it is in danger of becoming a piece of 20th century speculation that dragged into the 21st century by inertia. My intuition tells me that if there is a Unified Field Theory to be discovered, it will be instantly recognizable by its simplicity and elegance, and will bear little resemblance to any existing mainstream theory.