The phrase 'sanity check', meaning a double-take or a pause for thought, has its origins in the cultishly successful Call of Cthulhu Role-Playing Game. The game's premise, drawn from the works of H P Lovecraft, is that the truth of the universe is horrific and incomprehensible, so that understanding it is incompatible with sanity as humans know it. Each character has a sanity score, ranging from 1 to 99 and initially based on his or her POWer score, which indicates mental strength.
A sanity check is precipitated by something disturbing happening: a character reads the Necronomicon, sees Great Cthulhu, or tries to deal with dem bones' inbox. The character's player rolls 1d100, and compares the result to his or her current sanity score. If the result is below or equal to the character's current sanity score, no points are lost, or only a few. If on the other hand the roll exceeds the current score, a number of sanity points determined by the Keeper/GM is lost - in the case of a Great Old One, this may be 1d100 points! If five points, or 20% of current sanity, are lost, the character gains some nervous tic or outright insanity, to make life more interesting. Note also that losing sanity makes it more likely you'll fail the next check. When you run out of sanity, you're totally insane and beyond most help.