Of course it isn't rude not to eat meat. There are all kinds of things I don't eat, after all. If I'm having people round for dinner, I tend to check first for allergies, dietary preferences and so on, so that I can provide a meal that all my guests will enjoy -- that's the least a polite host can do. As far as I know, it never killed anyone to miss meat for one meal, and there are so many great vegetarian recipes.
On the other hand, it is rude not to mention that you are vegetarian, when invited, if the host fails to ask -- especially if your vegetarianism is recent, and you've eaten with them before. There is nothing worse than spending all day cooking a delicious meal, to serve it, and be told "Oh, I don't eat meat (anymore), did I forget to say?" It is also incredibly rude to say "Oh don't worry about me, I'll just eat the vegetables" if, on finding you are a vegetarian, the host offers to prepare you something. By all means, say no, but don't even give a hint you are being martyred. "The vegetables here look great, thanks, I don't need anything else." would be nice.
If you have an allergy, or preference, that is way outside the host's normal lifestyle -- veganism, or a gluten allergy, for example -- you could try saying something like "I know how difficult that is to cater for, I've got some recipes I can send you if that'd help" or if you're eating out, "These restaurants have a vegan/whatever option". Any polite host will be delighted.