Asian-American girls and others uses this phrase to refer to American men and boys who have an excessive and usually shallow attraction to Asian women. It is derogatory to say that a man has yellow fever, because the women feel that a lot of these men perceive them as geisha, submissive, fashionable, or some other undefinable quality that has nothing to do with their own personalities. A lot of geek guys find themselves attracted to Asian women, for whatever reason; just don't be surprised to be the object of derision if you admit to this. Most women do not appreciate being defined by their exoticity.

Please note I'm not writing that all guys who like Asian women are somehow objectionable, I'm just defining the term. 'kay?