English is not a logical language, unfortunately. Although you can talk about "Semitic people" in English and use it to mean people of a Middle Eastern origin, the term anti-semitism does not actually mean anti-Semitic. Due to popular usage it has come to mean anti-Jewish and despite its etymology any attempt to say that anti-semitism is hatred or dislike of all Semitic people is as doomed to failure as someone trying to claim that the word "gay" is simply a term for a jolly, happy person and has nothing to do with sexuality.

I personally dislike the term anti-semitism because I feel it perpetuates the idea that Jews are people of a race, rather than a religion. I have met Jews who are blonde, blue-eyed and picture-perfect examples of a stereotyped Aryan. I have met black Jews, and I know there are at least some Asian Jews. None of these people could be described as Semitic in any shape or form yet prejudice against them because of their religion would get termed anti-semitism. The illogicality of English.