The protagonist is the one character most writers dwell on. It takes a fine balance between making them enjoyable and having them seem either arrogant or totally worthless.
A good protagonist usually makes or breaks a story, while the antagonist, or villain, is usually tossed in as a reason for the protagonist to exist. This is not the way to get your story to the top of the slush pile. The villain needs at least as much development as the hero.
First off, having the villain exist as a method to make the hero suffer is, with rare exceptions, wrong. A villain must have motivations, wants and needs – just like the protagonist. It just so happens the wants and needs of the hero are diametrically opposed to the villain’s motivations.
Take, for example, Sauron in J.R.R. Tolkien’s masterpiece, The Lord of the Rings. Most readers just look at Sauron as an evil presence, set there to oppose Frodo. A good writer, on the other hand, looks at why Sauron is opposing Frodo. Sauron has his own wants and needs. He wishes to return to corporeal form and rule all of Middle Earth. Remember the old saying, Absolute Power Corrupts Absolutely? Sauron is the poster child of that old adage.
He doesn’t want Frodo. He doesn’t care about anybody. All he wants is his old ring. That is his underlying goal, while the second is to conquer Middle Earth. The hard thing a writer has to do is figure out the toughest question – Why? Did he have a bad childhood? Did a dwarf insult him in front of his high school crush? Of course, these are silly things, but getting to the root of the 'why' is what makes the villain real, both to the reader and to the other characters. A writer must have the why mapped out beforehand, in order to have the villain act and react properly throughout the story.
Sauron is not a typical villain, unless you count the various watered-down rip-offs of Tolkien’s work. It is difficult to create a malevolent, god-like villain that is realistic. Yes, the protagonist must seem to be out-classed at first, but he will slowly chip away until the chink in the anti-protagonist’s armor is exposed. Making the match too overwhelming can kill a short story, because the reader will just ask why the protagonist doesn’t just give up. Again, Sauron and Frodo go against this method of thinking, but this is a rarity, and it takes exceptional skill to pull off.
This holds true for most stories, especially novella-length and smaller ones. The match must be close enough to be realistic, but not too easy for the hero to overcome. People like heroes who struggle; it helps the reader root for the protagonist, and therefore connect and bond with them.
For the most part, the villain must be well-rounded. They should have some goodness to them, even if it is for something alien. In Aliens, the mother seems evil, but she’s really not. She’s out to protect her young, just like the reader would. This helps the alien connect to the reader, and it rings true. The audience wants Ripley to win, but they can understand the motivations of something that is literally alien to them.
Now that the audience understands the motivations of the villain, allow him to be ruthless and, well, evil. He should intimidate the heroine, but he should not intimidate the author. Allow him to do things you would never do, even in your wildest dreams. Make sure he follows his own code of morality, and stick to it. Chart out how he would react, and why. Don’t make him do things just because he’s a bad-ass. He already knows how powerful he is, so let him do just what needs to be done, and not too much else. If your villain is a psycho, remember she will still follow strict rules (insane ones, true, but rules nonetheless), otherwise she would be caught after her first evil deed. Nothing kills a story faster than having everything solved in the first page of prose.
Develop a whole history for your bad guy. Know why he hates flowers, why he smokes Lucky Strikes, and why he always takes the index finger of his victims. This may be explained to the reader, or it can be withheld in order to keep the reader guessing. Either way, you will allow the villain to follow his own code, and make it realistic.
That being said, don’t over-do it. If you don’t need an actual villain for your story, don’t just put one in. Forcing a villain will just clutter up the story. Two brothers may be fighting over the same girlfriend, but neither one is necessarily evil. Conflict does not always mean a struggle between good and bad. It just means two characters have opposing goals.
If you do have a villain, make sure everything she does is consistent. Have her react according to the rules you developed when fleshing out her history. If she’s a crack dealer from Brooklyn, she will not talk or act like a Duchess from York. If there’s a reason why she would be talking and acting like a Duchess, explain it during your story. Give the reader the whys so they can understand her actions.
Finally, with rare exceptions, the villain should get their comeuppance in the end. This does not mean they have to die – think about the future by expanding that novel into a trilogy – just make sure she pays some harsh penalty. There should be some feeling of satisfaction for the reader to share. The hero must triumph, the villain must fall. If you’re thinking back to Tolkien, remember the LotR was originally one very long book. The publisher broke it into three novels. With that in mind, Frodo won, and Sauron fell.