Caving is going out to a hole in the ground and crawling around in it for fun. Caving is all about crawling for long distances on your belly, squeezing through tight spots, and then seeing things open up into caverns with beautiful
cave formations,
stalactites,
box work. When you see a nook in the wall, you don't know whether it dies right there, extends on for ten feet until you can't get through and have to have a friend pull you out, loops back around to some place you've already been, or opens up into a room or whole
complex of rooms.
Touring a cave with a guide is a distinct experience from caving. If you don't get wet, dirty, or dangle from a rope at some point, you missed out on real caving.
You can't crawl through all caves. Many require rappelling through large drops, and then back out of the cave. This requires climbing gear: harnesses, ropes, descenders, ascenders. There's just as much fun to be had sliding down a rope into the unknown as crawling into it.
The two most important rules of caving are conservation and safety
Conservation
The beautiful cave formations take millenia to form, and can be destroyed by one careless caver. Because of this, almost all caving grottos don't publicize cave locations, and emphasize the need to leave the cave formations in the same condition they were found in.
Look, but don't touch, and certainly don't break or take anything out of a cave.
Safety
Being under a hundred feet of rock in an eight inch crevice in the dark is dangerous. Never cave alone, and use the buddy system to make sure no-one gets lost. Make sure at least one person knows the cave, unless you are all experienced cavers. Always wear a helmet, and bring at least three light sources (normally including a head lamp for the helmet.) You'll also probably want knee and elbow pads.
You can get into caving by finding a grotto of the National Speleological Society in your area, and going out on a trip with them.