| A story of a novice or inattentive sysadmin wiping out vital parts of a computer system by such commands as "rm -rf /" or "rm /vmunix". It becomes more horrific if there were no readable backups.
Here's the worst thing I have yet done as a sysadmin. I had to upgrade several machines from HP-UX 9 to HP-UX 10. Having done the workstations with reasonable ease, I set aside a Saturday to do the departmental mail server. I installed HP-UX 10 from the CD-ROM HP had sent me with no problems, just a long wait spent reading Usenet on my laptop, made our standard local modifications and checked that email was up again.
Then I had a large number of patches to install from another CD-ROM from HP (both CD-ROMs were part of their Year 2000 Transition Kit). This meant another long wait, and it was quite late in the afternoon by the time they had installed. I thought that since these patches came directly from HP, and the system had been fine before I installed them, it was unlikely that it would suddenly stop working. So I let it reboot and went home without testing it.
On Monday morning, I found out that email had completely stopped working. Eventually I discovered the reason why. On our systems, /usr/sbin/sendmail is a link to /usr/sbin/exim, which is a link to the real exim, which is on the /usr/dpmms filesystem on our mail server, which is NFS-exported to our other machines. I had been careful not to overwrite anything on /usr/dpmms during the install. However, the patch program (swinstall), on being asked to replace /usr/sbin/sendmail, had followed the chain of links and replaced the REAL exim with sendmail! As you can imagine, my users were not happy about the long disruption to email. Fortunately I was able to restore exim from a working backup tape.
| 
