A garbage matte is a technique used in motion graphics compositing. In order to understand it, it's best to picture a specific use for it. Let's say that you're making a movie, and you want a shot of one of your actors standing on the edge of the Grand Canyon. Unfortunately, you're a dirt-poor film nerd with a crappy coffee shop job. You can't afford to fly your shoot out to Arizona. What do you do?
Well, the obvious answer (as even the least media-literate dip knows, these days) is to put your talent in front of a green (or blue) screen and use your magic computer box to drop the Canyon in behind him. Sounds easy, hmm?
Not exactly. Again, you have to consider how crappy your coffee shop job is. They don't even pay you enough to put up with people ordering using the pretentious Starbucks sizes instead of "small" or "large," so odds are you can't afford a very big green screen. I know, I know. It's a green screen. It sounds simple right? Not really. It has to be perfectly flat and textureless, not to mention completely evenly lit. If you just spraypaint your bedroom wall neon green, your shot'll look like crap.
So, again, your green screen is kind of puny. So puny that it doesn't fill the full frame of the shot. If you just run your chroma key on it, there'll be a strip at the edge of the screen where you can see your dog, or a lamp, or whatever the heck happens to be behind the screen. What to do?
Use a garbage matte! In your compositing program of choice (Adobe After Effects, Discreet Combustion, et cetera), draw a geometrical shape around what you want to be included in the shot (in this case, your actor with his purely green background). Everything outside the shape is effectively no longer there. You've cropped out the "garbage" that you don't want. Run your key now, and the illusion is seamless.
In general, it's best to draw your garbage matte so it conforms roughly to your subject's size and position, and you can animate the matte over time to conform to your subject's movement. In other words, don't just crop out the gaps in the green. Crop out most of the green. The less green there is for your key to replace, the faster your render will be.
For an example of a garbage matte, check out this After Effects tutorial:
http://www.creativemac.com/2003/01_jan/tutorials/bisectinganimation.htm