The reason is quite simple, and it's not what you think.
It is not because it's distasteful, disrespectful or even because it's rude.
It is because Men Are Evil.
The male body and psyche is completely governed by hormones and primitive drives and such are incapable of separating the notion of a woman as a whole from the body parts they are observing. Therefore, when a male looks at a woman's breasts, he instantly views her sum total as an extension of the breasts, not as a human person.
If a male looks a woman square in the eye at all times it is possible, albeit rare, that they can have a normal conversation and treat the female like a person. However, the moment the eyes venture down below the neck, males cease this and are not able to consider the female as anything more than a sexual object because of the primitive drives that govern their lives.
Men should not look at women's breasts because men are worthless pigs.
However, women may oogle because, unlike men, they are intelligent and thoughtful and therefore are void of all of the animalistic effects that men are bound by.
No, I'm not serious. Neither should you be.