Because it can reduce the fitness of the future population, which includes them!
How?
Men and women try to choose, not always consiously, a mate that will help them create the best, healthiest, babies possible. We can't see a person's genes by looking at them, but through evolution, many of our preferences have leaned toward choosing the best fit individuals as mates.
Many scientists believe that love is at least partially based on scent, and that one cannot fall in love with someone they do not find attractive, scent wise. Some of our preferences for smells are common, for example nobody likes the smell of rotting meat. These help keep us from danger, and in choosing a mate, keep us away from the seriously diseased. Other preferences are individual, based on our specific genes.
Some of the genes that help decide our scent preference are those which code for our immune system. These also affect the smell each of us has. Innately we like the smell of those who have few of these genes similar to us, and dislike the scent of those who have more of the same genes as us. This causes us to choose, or fall in love with, those with different immune genes than us.
This creates a larger genetic pool for our childrens' immune genes to be selected from, which makes for a potentially healthier child, and a more fit population. It is important to note that our noses only work to tell similar from disimilar immune system genes, not effective immune systems from less effective. Appearance is used for that, and is a whole other story.
However, when we wear cologne or perfume, our natural scent is at least partially masked. This means the best mate for producing healthy children may not be chosen, at least when it comes to their immune systems.
This was demonstrated on Human Instinct, where genetic testing was done on a man and several women, the man smelled shirts worn by all the women, and chose the ones least genetically similar to him as those smelling the best. The genes tested were six of those known to code for the immune system. Other such studies have been carried out, with the same results.
Why a larger gene pool is beneficial:
A larger gene pool (in terms of immune genes, not all genes) is beneficial for a population as a whole, rather than an individual. The more immune genes a population possesses, the more likely it is that some individuals will survive if a disease comes along. However, natural selection will tend to ensure that the genes coding for immunity to diseases common to a population are also included in most individuals' genomes.
So, while the genes that your future children get and whether those code for immunity to any particular disease that comes about is seemingly random, it is in the interest of future populations to have maximum immune gene swapping.
The Human Instinct episode with the demonstration described above aired on The Learning Channel, Dec. 16, 2002.