sysadmin

There are two kinds of sysadmins:

Those who say, "The system is there to support the users in furthering the purpose"

And those who say, "The system is beautiful and perfect and fast, until all those lousy users log on."

Be sure your sysadmin is the former type.

A system administrator, or sysadmin, is (broadly) a person employed to manage one or more computer systems. The term is a catch-all for many different kinds of jobs. Especially in a small organization, a sysadmin is likely to serve as a jack-of-all-trades for computer-related matters.

Generally, a sysadmin works with mainframe or server computers, rather than desktop and workstation systems. In many modern shops, "sysadmin" equates to a maintainer of Unix systems in particular; this may be because NT and NetWare sysadmins seem to prefer to hijack the term "engineer" for themselves.

A sysadmin is often the chief in-house technical authority on his or her systems, though s/he is not usually a ranking organizational authority (such as a manager). He or she is probably responsible for troubleshooting and upkeep on these systems, as well as resource management, user management, and some manner of policy enforcement. If the system crashes or in other ways fails, it is the sysadmin's job to bring it back to a usable state.

S/he may do the backups and other routine tasks, or may hand this off to an operator or other underling.

S/he had damn well better be thinking about security as well.

---

One of the myths of computing is that sysadmins are as a rule hostile, disgruntled, or hazardous to your health. This myth is furthered by the hilarious "BOFH" saga by Simon Travaglia et al., now continuing in The Register. Yet while many sysadmins find the BOFH's exploits -- and his gleeful incineration of lusers -- wildly entertaining, few will actually kill you for asking dumb questions.

Understand, though, that sysadmins (like everyone else) get frustrated when folks keep them from getting work done. In general it is the sysadmin's job to make the servers function properly. S/he is not helpdesk or desktop support, and likely is not being paid to remove cheese from your floppy drive. Neither is it the sysadmin's job to throw security, good practice, and the good of the organization aside in order to grant you more network access, system privileges, or quota for your MP3s.

It is those sorts of things -- along with dodgy hardware, buggy software, tight budgets, spam, and miscellaneous cluelessness -- which increase your sysadmin's blood pressure and cause him or her to cackle all the louder at the BOFH.

Let me give you an idea.

The network is too slow. In order to speed it up, install a new 10/100 ethernet switch.

Then watch as the network slowly goes to hell. First it's slower, then it stops working entirely. Everyone in your office stops working and stares at you.

Swap the old hub back in. Everything is fine again.

Wait on hold for 45 minutes. Get an RMA number. Mail back the switch. Wait seven days. A replacement arrives in the mail.

Plug it in.

The network starts out fine, and then gets slow again. Although this time, it never stops working altogether.

As before, the server (Linux) shows (cat /proc/net/dev) hundreds of thousands of collisions AND carrier drops. (Switches are never supposed to have collisions - one reason why they're theoretically faster.)

Notice that when I unplug and plug the server back in again, it goes to half-duplex - from full-duplex. At half-duplex, things are suddenly passable - back up from LocalTalk speeds to 10BT speeds. Collisions? About 5,000 per megabyte, but at least things work now. Keep in mind, this is a switch. Also keep in mind, everything was fine before.

Replace the server's patch cable.

Collisions/carrier drops decrease - speed remains identically impaired.

There's no easy way to get ethernet diagnostics from the other windows boxes or macs on the network... Other Linux boxes, however, are not reporting any collisions or carrier drops.

I want to test out fast file transfers on another Linux machine with a different ethernet card, so I can monitor collisions/drops and try to build evidence the server has a bad ethernet card. Well, bad for this particular switch, anyway. I decide to set up Linux on my own machine, which fits the bill perfectly.

I happen to have a Win98/Win2k dual booter. I install PartitionMagic (so that I can resize my windows partitions and make room for Linux).

During the install, I'm asked to create some "rescue disks" in case anything goes wrong. Scrupulous person that I am, I do.

My computer is one of the fancy new ones with an IDE SuperDisk floppy drive. It's a little funky. I'm idly curious as to what will happen during the disk creation process. Sure enough, when I attempt to format my floppy disks, I'm given the option to format them "1.38MB" - and no alternatives. I do, and then run the disk creation utility. It says "not enough room on floppy."

I look around, and spot the closest real floppy drive - in a Linux machine. mformat a:... and now the floppy works fine. I make my rescue disks - two for PartitionMagic, and one for BootMagic.

Just for the hell of it, I defragment my Windows 2000 partition. Once this is done, I notice a demure "Things have changed - make a new Emergency Boot Disk" warning.

That's great, but how do I do that exactly? I search around for 5 minutes, and then give up.

I repartition the disk, shrinking the unused win98 partition and using the leftover space for Linux.

I Reboot with the Redhat 7 CDs. The install runs smoothly, except that Redhat 7 doesn't recognize my Radeon (doh). The whole process (custom install) takes about three and a half hours.

Reboot, attempt to run Win2k.

Error on boot. "Unable to find..." something. NTLDR, or ntsokernel, or something.

Reboot with win2k CD. Try "emergency repair console." Hunt around through the command list until I see something called “fixboot”. Run it. Reboot.

Error on boot. "Unable to find..."

Reboot with win2k CD. Try "emergency repair console" again. Run “fixboot” again, followed by “fixmbr”.

Error on boot.

Reboot with win2k CD. Run “Automated repair.” The repair system “Can't find windows.” It asks for an “Emergency Repair Disk.”

That’s right. The one I didn’t make. Then it gives me an “Internal error 2210.” Crash.

Reboot. Error. "Unable to find..."

Run “Automated repair” in Manual mode. “Internal error 2210.” Crash.

Reboot. Error.

Try BootMagic rescue disk. It gives an error and crashes out of its autoexec.bat. From the DOS prompt, I see an EXE, and run it. It lets me select a partition to boot from. Select Win2k’s parition.

Reboot. Error. "Unable to find..."

Boot in win98. Install PartitionMagic. Turn win2k’s logical partition into a primary one.

Reboot. Error.

Repeat the above 5 steps. Repeat the above 5 errors.

Try to reboot into win98 again.

Error.

Reboot with win98 CD. "Setup not found" error. Drop to DOS prompt.

scandisk: not found.

chkdisk: not found.

Reboot with "win98 emergency repair disk" I made in 1998. Run scandisk. The win98 partition is swiss cheese.

Boot with PartitionMagic rescue floppies. These, at least, work. They just take 5 minutes to start. Nuke the win98 partition.

Reboot with win98 CD. Yes, it really does do nothing but say "Setup not found" and bounce me to a shell prompt. Yes, this really is a Microsoft imprinted, holographically stamped Windows 98 Second Edition CD – and it really is completely useless.

Reboot with PartitionMagic rescue floppies. Nuke the Linux install I just spent all this time over. Attempt to reposition the win2k partition to the front of the disk so the win2k repair system can "find it." PartitionMagic fails - internal error 4001.

Reboot. Repeat. Crash - internal error 4001.

Boot with win2k CD. Emergency repair console. CHKDSK /p on the win2k partition. The screen fills up with errors that are being corrected. Win2k partition is now also swiss cheese. However, now it's at least consistent, right?

Reboot with PartitionMagic rescue floppies. Attempt to reposition the win2k partition to the front of the drive again. PartitionMagic fails - internal error 4001.

Reboot with win2k CD. Run automated repair. Can't find a windows installation to repair. Internal error 2210. Crash.

Wonder whether my files are still there.

Reboot with win2k CD. Install a new version of Windows 2000 into the unpartitioned space on the hard drive. This takes 2 hours.

Reboot. Discover that drive C: is your old Windows 2000 drive ("System"), drive F: is your new boot drive ("Boot"), neither drive letter can be changed. I am NOT going to try to press on like this.

Reboot with PartitionMagic rescue disks. Discover that win2k setup made an extended partition at the beginning of the drive and put its new install of win2k into it. This shouldn't even boot at all. Let's not even wonder how it decided it was drive F:. Nuke it. Make a new (primary) fat32 partition in its place myself, format it, and "hide" the old win2k partition.

Oops, bad block checking (completely wasted effort with modern hard drives) is on. Wait 4 hours for this operation to complete.

Reinstall win2k a second time. 2 hours later, it boots.

Begin to install drivers.

Install video driver, Ultra ATA driver...

Discover that Win2k is now hard-locking the computer - including the power button - within 3-5 minutes of booting up... every time it boots up. This takes about 4 crashes.

Go to the website of the manufacturer whose Ultra ATA driver you just installed to get the new version. It's a passworded driver area. This is completely asinine. Fortunately, you have a password. Unfortunately, it was written in Outlook. Before your computer crashed.

Fortunately, you have it mirrored in your palm pilot. Look it up. Type it in. Start the download. 68%... 71%... 74%... crash.

Reboot in "Safe Mode with networking." Go back to the same website.

Enter your username. Hit tab.

MSIE crashes.

End Task. Restart MSIE. Go back to the website. Enter your username again. Hit tab again. MSIE crashes again. Yep, it’s definitely the tab key.

End Task again. Restart MSIE again. Go back to the website. Enter your username again. Cleverly click on the password field instead of pressing tab to get to it. MSIE crashes again.

Scream. Stamp on the floor. Search desperately about for something to fragile to smash against the wall. Fantasize about skipping your next vacation so you can afford to beat the living daylights out of the computer, throw it out the window onto the street, and then light it on fire, and then buy a new one.

Consider new career in street corner hot dog vending. After all, you get all the hotdogs you want for free.

sys-frog = S = sysape

sysadmin /sis'ad-min/ n.

Common contraction of `system admin'; see admin.

--The Jargon File version 4.3.1, ed. ESR, autonoded by rescdsk.

When you're in High School, learning the ropes of Linux or OS/400 or whatever else, IT seems like it would be a pretty relaxing career. Sitting in a comfy office with terminals connected to powerful and largely self-sufficient machines in a nice, cool, datacenter, responding at your leisure to respectful requests for features from users... what's not to like? Sadly, the truth isn't like that. It isn't even close.

There are three big IT Scenarios that will result in hell for you, the chipper young IT girl: the Upgrade, the New Installation, and the Awful Legacy System. The Upgrade and the New Installation are closely related, while the more distinct Awful Legacy System can occasionally make the Upgrade necessary or even a good thing. Sooner or later, every installation of any size - even ones with a few dozen users - will go through one, two, or even all three of these situations.

If you're really unlucky, or maybe just desperate for work, you'll find yourself in the middle of the New Installation. If you get off easy, your employer wants something simple that can be handled by LAMP, or perhaps Windows Server and Active Directory, with minimal customization and commodity hardware. Ultimately, of course, this will probably evolve into an Awful Legacy System, but in the mean time, your job is easy and the system can hopefully be set up and tested in a few days. If your employer has higher requirements for reliability or performance, it's your lucky day: you get the rare and lovely opportunity to talk to one of the three Big UNIX Vendors, Oracle, HP, and IBM.

What a lot of armchair sysadmins on forums miss is that technologically, for most applications, these guys are roughly interchangeable. Yes, Power7 is faster than Itanium, and Itanium is faster than SPARC, but the extent that it matters in the real world is relatively minimal. Most of the tradeoffs here are financial. For instance, IBM licenses software per core, rather than per socket like HP; that means that in the money saved from licensing by going HP, you could potentially get a 64-socket Superdome (HP's high-end) instead of a 32-socket 795 (IBM's high-end). You can also ignore the list prices these companies have posted, since they exist almost purely so that the sales rep can give you 80% off of the list price and talk about what a great discount he's getting you. Ignore anything you find on the Internet, and instead ask each sales rep these questions.

• How much will the system cost, from start to finish, to acquire? Include software, hardware, and services. Do not let the rep bullshit you.
• What are the industry-standard benchmark results for the type of application you wish to run on the system? If you're running an enterprise Java application, demand SPARCjbb. If it's a database server, tell the vendor you want a TPC-C or TPC-H result for the type of system they're trying to sell you. If it's technical computing, check SPECcpu2006. Do not allow the vendor to try to present some kind of proprietary benchmark - both IBM and Oracle are notorious for this.
• Ask to see a roadmap for at least the next five years. Look over it very carefully, since on UNIX systems vendor lock-in is a huge factor.

After you've done this, figure out a quick price/performance estimate for each, and calculate how much having each system on a service contract for the next five years will cost. Go with the cheapest.

The Upgrade is the least fun, especially since it can bring along its friend, the Migration To Another Platform, if you weren't careful enough during the New Installation or if the vendor did something unpredictable. The Upgrade involves a lot of the same things as the New Installation, except you have an application that needs to be ported, and user data, and users that will get very, very upset if there's any downtime. You'll soon find that C is not really a portable language, and that even Java has its own bizarre porting difficulties. Auxiliary stuff - libraries, database components, native routines - will need to be ported, and frequently this is not easy. If you're sticking to the same platform, but a newer box, you can sometimes do an automated migration (HP-UX is exceptionally good at this), but most of the time you won't be so lucky. Even once you're congratulating yourself on getting your PA-RISC/HP-UX application running on SPARC/Solaris, it's not really time to celebrate; in five minutes a user will call and start complaining about how it's running painfully slow, much more so than before, which shouldn't even be possible. Then you get to dig through logs and source code, figure out why, and fix it.

Sometimes, you'll be job-hunting. You'll see a promising post - maybe it will say "Active Directory admin for small network needed," or perhaps "Looking for a laid-back HP-UX guru." Horrors lurk within such things. The interview will go well, you'll sit in your new office, and a higher-up will casually mention that there are no Group Policy Objects and all users are managed with scripts in their Startup folder. The scripts are esoteric and the guy who actually knows how they work left a year earlier. The sinking feeling in your stomach confirms it - you have entered the land of the Awful Legacy System! Meanwhile, users will be sending emails demanding that you pick up their computers (1.8GHz Pentium 4, 256MB RAM) and fix them because Vista is running slow. As you frantically try to get the Active Directory configuration into something resembling usable, calls will start coming in about how a legacy application you've never touched and never been told about isn't working right. "What's the error message?" you ask. "I don't remember," they answer. Things go nowhere good from here.

When I entered the workforce, I thought IT seemed like the coolest thing ever. I survived a year of it with a degree of sanity and without my love of computers burned out, and now I've found my true calling: software development. It taught me skills that I wouldn't trade for anything, but I am never, ever going back to anything resembling a sysadmin job.