I don't mind saying "person" instead of man when it's a word on its own. I don't even mind the slightly odd mutations "chairperson" and "spokesperson". However, using the word "they" instead of "he" or "she" never sits right for me.
I'm sure I'm not the only person who automatically reads "they" as the plural pronoun it was originally intended to be. I was taught as a child that "they" was plural, and as with most grammatical oddities, it requires several more brain cycles to resolve.
English unfortunately lacks a gender-neutral personal pronoun. We've comandeered another personal pronoun which already has its own meaning to fill the gap, when we really should have invented a new word a long time ago. I'm not even sure a new word would have settled neatly into my brain, but it would probably stress it less than plural personal pronouns for singular objects.