A note to explain today's outage (and cut down on the gossip and speculation)
We were down from 21:15 server time until about 02:00, and here's why:
A pair of exploring edevites came across a fairly major bug which (as was demonstrated) was capable of doing significant damage. Unfortunately they tested it on key nodes and this choice had a very visible impact on site operations. When the extent of the issue was determined, immediate shutdown was deemed to be the safest approach while it was being dealt with. Time from discovery to shutdown was about 15 minutes.
The nature of the bug, as one of the culprits explains, was rather unlikely. Compare it to scenario three of popular stone-throwing theory: you throw the stone not believing that it has a chance of hitting a really remote target and score a bullseye. You then have a devil of a time explaining yourself to someone with a bloody forehead because, well, you were trying to hit them even though you didn't think it was possible. The nasty technical details of the bug will be elaborated upon in the root log within a few days. Or, you could see this.
The offenders owe JayBonci a case of wine each. He'll get beer from the rest of us.