Stenciling, like depth-buffering, enables and disables drawing on a per-pixel basis. You draw into the stencil planes using GL drawing primitives, then render geometry and images, using the stencil planes to mask out portions of the screen. Stenciling is typically used in multipass rendering algorithms to achieve special effects, such as decals, outlining, and constructive solid geometry rendering.
The stencil test conditionally eliminates a pixel based on the outcome of a comparison between the value in the stencil buffer and a reference value. To enable and disable the test, call glEnable and glDisable with argument GL_STENCIL_TEST; to control it, call glStencilFunc.
glStencilOp takes three arguments that indicate what happens to the stored stencil value while stenciling is enabled. If the stencil test fails, no change is made to the pixel's color or depth buffers, and fail specifies what happens to the stencil buffer contents. The following six actions are possible.
GL_KEEP
Keeps the current value.
GL_ZERO
Sets the stencil buffer value to 0.
GL_REPLACE
Sets the stencil buffer value to ref, as specified by glStencilFunc.
GL_INCR
Increments the current stencil buffer value. Clamps to the maximum representable unsigned value.
GL_DECR
Decrements the current stencil buffer value. Clamps to 0.
GL_INVERT
Bitwise inverts the current stencil buffer value.
Stencil buffer values are treated as unsigned integers. When incremented and decremented, values are clamped to 0 and $2 sup n - 1$, where $n$ is the value returned by querying GL_STENCIL_BITS.
The other two arguments to
glStencilOp specify stencil buffer actions that depend on whether subsequent depth buffer tests succeed ( zpass) or fail ( zfail) (see
glDepthFunc). The actions are specified using the same six symbolic constants as fail. Note that zfail is ignored when there is no depth buffer, or when the depth buffer is not enabled. In these cases, fail and zpass specify stencil action when the stencil test fails and passes, respectively.