A
technique used in
computer graphics (and, especially,
polygonal rendering) for a variety of
special effects, including most importantly,
reflection and
shadows, both of which are otherwise difficult to implement in real-time. It consists of a
2D array of the same size as the
frame buffer, anywhere from 1
bit to 8
bits per
pixel.
It works like a real stencil: You use it to control which parts of the framebuffer get overwritten; i.e. any value for which the stencil buffer is "true" will not be written to when rendering the scene. This might be used, for example, to implement reflections by ensuring that the reflected image is constrained only to a particular area, such as the mirror and not the rocks around it.
However, it can be used for much more complicated effects: For example in using shadow volumes, the stencil buffer is updated in a complicated way to indicate whether a point is in or out of shadow. Most systems support an 8-bit stencil buffer, and even allow comparisons other than simply true to decide whether or not to render a scene.