As the name suggests, anecdotal evidence is evidence for a claim which is based on
remembered anecdotes either first hand from the speaker or second hand from others.
Such evidence is often used to justify empirically verifiable claims. This is fine and even
appropriate in the case that the claim is of little importance or in the case that it is reasonable to
believe that few observations with little precision are needed as basis for a claim. Often, however,
anecdotal evidence is used as backing for sweeping claims about a wide class of things, like natural
phenomena or the behavior of people, which are the sort of claims usually examined by
science (natural science, life science, or social science). Below I will investigate the
reasons why anecdotal evidence is unreliable and unacceptable evidence of
scientific claims.
The basic point is that you need enough evidence to make a credible claim, so you must
justify how much data is "enough". In science this is done through statistical techniques
and error analysis. You figure out mathematically if you take so much data, what is your
probability of getting the wrong answer. Then you take enough data that that chance is
very small, normally 1-5% or less, depending on the situation. This problem of sample
size is particularly troublesome when you're making claims about people or society. It is
one of the reasons why when people make claims about "every accountant" or "every Asian" they ever met,
their conclusions are invalid; every person you have ever met is still only a miniscule
portion of all people, so your data would have to be extremely precise and unbiased in
order overcome the uncertainty from your small sample.
Reliability of Records
Anecdotal evidence is usually based on events people remember, but that makes it prey to
many different problems. The main problem is that memory is not 100% accurate nor is
it really static. You don't remember an event the way it was; you remember what you
thought was important at the time; furthermore, this memory is continually reshaped by
your later experiences and attitudes, so that the way different people remember the same
event, or the way the same person remembers the same event at two different times, can
be radically different. It is a commonly known and accepted fact in psychology that
people will reshape their memories to fit their later circumstances or convictions.
Memory is also selective. Again, you remember what you believe is, or believed at the
time was, important, and if you believe a certain thing you are often more likely to
remember instances in which that belief was validated rather than ones in which it was
not. For example, it will seem to many people that there is always a long line when they
go to the supermarket. It is the fact that you're unlikely to remember the time when you
only waited in line for a few minutes, but you'll definitely remember when you waited
for 45 minutes. And once you're already convinced the line is always long for you, then
you are likely to get even more agitated when you have to wait, making those times even
more likely to stick out in your mind. So, remembered information, without objective
recording at the time data was taken, is simply unreliable even when people are being
honest, leaving aside completely the possibility that they may intentionally bend the truth.
It is very important that one know the reliability of data and that that reliability be high
enough to justify conclusions. Measurements should be objective and the uncertainty in
them should be examined. This usually means that data should be quantitative and for
any measurement the uncertainty in that measurement should be estimated based on
statistics and examination of the theory being tested. Even if data is more qualitative, as
in social science for example, still it can be generally categorized and analyzed
statistically to estimate uncertainty. This is one of the most important parts of
experimental science. In essence, you take data such that everyone who saw the
particular event could agree on the data, and you know the answer won't be exactly right,
so you want to know about how far off it is. If you know the uncertainty you have a
measurement, otherwise all you've got is a wrong answer. Anecdotal evidence often
employs subjective claims. One particular example is when someone says he took a
medicine and "felt better". But you don't really know what that means. Essentially you
might ask, "how much better?" because what one person considers "better" another may
not. This is why you must use objective evidence, like quantitative measurements, so
that you can compare on a case to case basis and everyone can agree.
Control
This is probably the main area in which anecdotal evidence is usually lacking. In general
you are making the claim that X causes Y based on the fact that every time that X
happens Y happens. However, you must make sure there's not a third thing Z that is the
cause and just happens to be around at the same time as X. This is where control comes
in. You want to have an experiment where you only change the thing you're interested
in, and everything else is the same. This is usually done through calibration and often the
use of a control group. In anecdotal evidence, however, one is usually drawing from
experiences in which the circumstances are very different from one case to the next and
many pieces of information that may be relevant are unknown. So, you may, for example
conclude you get colds all the time because you're always congested, but a controlled
experiment may show you actually have allergies, because you only become congested
when certain allergens are present. Properly
controlling an experiment is particularly hard in medicine and the social sciences (since
you can't just do whatever you want with living people), but it can be done, often through
statistically compensating for other factors.
Review
Finally, an integral part of any honest research is the ability for others to review what has
been done. This partially rests on earlier points, because in order for others to review
your work, you must have reliable records of objective data. This is at the core of true
investigation precisely because we must be very careful about the way we collect and
interpret data, so others (who do not necessarily share our views) must be able to look
over all the facts and see if our methods and conclusions are reasonable. This is a
tradition within science, carried out through peer reviewed journals and conferences.
Also, the ability to review means the ability to compare the data of one person with that
of another person to see if results are repeatable, and it means the ability for others to use
the data later as part of a larger set of data to draw more reliable conclusions. Anecdotal
evidence does not allow for this sort of review, because there is no record other than
recollection, so you don't have a clear idea of the facts, just one interpretation, and there
are no objective measurements against which to compare. In short, if someone tells you
their conclusion but can't lay down the evidence for open inspection, then you can't be
sure that they're not in error or lying.
The basic point being made here is not particularly abstract. It's basically that if someone
is trying to convince you of something, they should show you stuff that you can verify
yourself, and they should be able to show you their entire process so that you (or
someone you trust knows what they're doing) can be sure they've done things correctly.
If someone cannot or will not do this, they may well be lying, and even if they aren't,
there's a good chance they've made mistakes.
So now the question arises, when is anecdotal evidence acceptable? Basically, you have
to take into account its limitations. If it's a situation where you don't need a big sample
size, the result is unambiguous, and basically you're not trying to draw a very far
reaching conclusion, it's ok. If you're trying to describe the laws of nature, the human
body, people, or society, then it's totally inadequate for the task and you need to use the
scientific method.
Anecdotal evidence is often used to justify pseudoscience in a number of areas. These
include
Some of the worst offenders I have heard are politicians, who use ridiculous anecdotal
evidence to "prove" particular ideas about nations or about their policies, and proponents
of alternative medicine. On the later point, I am not saying that there is nothing to
alternative medicine, only that there are those within that community who wish to
abandon the scientific method and that their conclusions are worthless. That does not
invalidate the work of others who apply scientific practices in the same field. Also, it's
not surprising that this should happen, after all, drug companies have been using such
pseudoscientific arguments in their commercials for years with things like testimonials.
Another thing that tends to creep up from this sort of sloppy reasoning is racism, which is
also often based on pseudoscientific arguments both about the nature of race and the
relationship between race and behavior.
It is very natural to use anecdotal evidence. Noticing things and inferring a pattern is a
basic human behavior and the basis of the science; however, we must understand that
while anecdotal evidence works well enough for everyday things, it is inadequate to
tackle the big questions, which is why scientific methods were invented. So, when you
hear people making sweeping claims about anything, nature, people, whatever, analyze
their evidence and see if it really holds up to the demands on it. If it's just a collection of
anecdotes, then explain to them why it doesn't prove anything.
I had some trouble organizing my ideas for this write-up into sections or headings, so if you have a good idea for a more clear organizational scheme /msg me. Also, any other suggestions are always appreciated.