There are several ways to remove tree stumps. Basically, you can dig it up, pull it out, burn it out, grind it out, expedite the rotting process with chemicals, or just hire a tree service to do it for you in the best professional manner. How it should be done depends on why you want the stump gone, how big it is, how fast you want it gone, how much work you are willing to do, how much you are willing to pay, safety factors and whether you've got just the one out there in the yard or an acre or so of them.
Digging
This is my preferred method. It provides physical and mental challenge, much-needed exercise, and sufficient self-satisfaction to justify a healthy feeling of superiority over the average person.
I have myself dug up several tree stumps that ranged in size from about two feet to four feet in diameter. Saplings and trees of up to a foot or so in diameter at the ground can be dug up in an hour or an afternoon with a spade and an ax. You just dig out the dirt around them and chop off any sizable roots until the stump is freed and can be pulled out. The process is the same for larger stumps, but the time and work increases by a rough factor of the cube of the stump diameter. Removing the 4 ft stump took me about a month and required digging a hole about 16 feet in diameter and 6 feet deep. Several roots were about a foot in diameter and needed the attention of a chainsaw. A winch and large levers were needed to get the freed stump up out of the hole. That particular tree required digging because of proximity to a septic tank drainage field.
Pulling
You need a stump that's not too big, a big, powerful tractor, access by the tractor to the stump, a heavy chain, and a means of securing the chain on the stump so it doesn't just slip off and give your neighbors or family a good laugh on you. This is not a practical method for the average homeowner. Better to dig or hire a pro.
Burning
Burning does not work well with fresh stumps. The best candidates have already begun to rot, and the more advanced the rot the better they burn.
Stumps can be burned out by soaking them in an inflammable fluid, which should be kerosene or other liquid that does not have a low flashpoint and burns slow and hot. (Gasoline has a very low flashpoint, evaporates quickly, and burns off quickly. Absolutely do not try gasoline.) You might have to drill some holes into the wood to get a good soak, and it can take quite a while for the kerosene to permeate the wood.
Once ignited (it may take several tries), a stump does not so much burn like a campfire as smolder. For a very long time. It's a good idea to cordon off the area with chicken wire or something and keep an eye on it for safety. The smoke will not be pleasant and may bring complaints. Rain may or may not extinguish the smolder. You have to be thorough about killing the smolder once it has progressed to your satisfaction.
Grinding
You can rent a stump grinder and go at it. Stump grinders range in size from a lawn-mower sort of thing to a back-hoe sort of thing that you sit in and pull levers to operate. You can rent them, and they are a great choice for the power tool enthusiast with money to spend. Your stump is gone quickly and you've had some good fun.
Chemicals
This is usually the first method of stump removal that comes to mind and has the dual appeal of low expense and great simplicity. You just buy the chemicals at your preferred home improvement center and apply the chemical to the unwanted stump. For fresh stumps that are still alive, you can expect several months or a year or so of waiting. It is often necessary to reapply the chemicals a number of times, after digging out the part that has decomposed.
Various chemicals are used, but they all work by expediting the natural breakdown of the wood fiber by rotting or decomposition. One of the most effective is potassium nitrate. If you have any left over, you can make bombs with it.