Stands for 'Hall of Mirrors,' the effect you get when
you clip outside the walls in Doom.
For speed's sake, a double-buffered game might not clear the framebuffer each frame, based on the premise that the entire framebuffer will be filled during the next frame anyway.
But, when the framebuffer is not completely filled (e.g. the game engine is, for some reason, not drawing in that region of the buffer) the data from the previous frame is left in -- resulting in a weird artifact.
The trippy nature of the HOM effect comes from the fact that the 'dirty' portion of the framebuffers contain portions of two different frames. The two frames, looped at high speed, provide a strobing, flashing, seizure-inducing effect.
This effect is most interesting in Quake 3 when each successive frame multiplies the artifact from the previous frame, slowly saturating the colors.