I don't think that there is a right or wrong answer to an issue such as this. Basically, it has to do with manners (i.e. how each individual person is raised to react to these comments).
I was raised that if a man said something about my breasts then not only was he thinking of sex, but he was thinking about having sex with me. Sex is a very intimate thing and something that I want to have control over. So, in a way, a man complimenting me on my breasts could be seen as taking some of that control over my sexuality away from me.
A second point is that just because a woman acts offended, does not mean that she is. A woman may get angry with someone for commenting on her breasts (because she thinks that she is expected to-- or because she thinks that people will consider her a slut if she doesn't protest) when in fact she enjoyed the compliment.
And just a side note, commenting on a women's breast is more like commenting on a man's penis than his chest, in my opinion.