Author: JW Smythe
I’ve been a sysadmin for just less than 10 years, and a Linux user for a bit longer. I’ve worked with almost everything out there. I am now the senior sysadmin at my company. Our environment is exclusively Linux servers, the majority being our own roll of Slackware called LMLinux.
My home environment is also Linux friendly. My primary workstation and laptop are both Linux machines (AMD64 3200+ with Slamd64). For the sake of Windows users who have a Windows-specific problem, I keep a Windows XP machine around. I used to claim I kept the XP machine for gaming, but realized I play games less than once a year.
But enough about me — on to the toolbox.
Slackware Linux
Slackware Linux may be a bit too big to be considered a tool, but it tops the list nonetheless. I have the distro CD and our own home-rolled version of Slackware, known as LMLinux, with me at all times. It’s been amazingly useful, even for fixing other distros. When other consultants call, saying, “I have a hosed box, can you help?”, I’ll throw the Slack CD (or LMLinux CD) in, boot right up, and fix their problem.
LMLinux is a good server install, pruned of fluff, with a few additional libraries and tools, and tarred and gzipped back up to make an easily installable package. We even have install and update scripts, but here’s the general idea, should typing “install.os” be too easy:
fdisk /dev/hda
mke2fs -j /dev/hda1
mkdir /new/ ; mount /dev/hda1 /new/
mkdir /cdrom/ ; mount /dev/hdc1 /cdrom/ #or whatever your CD is.
cd /new/ ; tar xvpzf /cdrom/os.vX.X.X.tar.gz ; lilo #where vX.X.X is the version number
reboot
It’s a perfectly good install which is accomplished in five minutes from cold boot to running server.
XFree86 / X.Org
Every admin needs a windowing environment, and for that you’ll need to have XFree86 or X.Org installed. Otherwise, you’ll go nuts switching consoles. I’ve spent my fair share of time switching consoles, and now I’m beyond that. I can’t lay claim to any favorites in the window manager war. I like GNOME and KDE, but for years I used Enlightenment. Now that Slackware comes with KDE as the default, I’ve been using that.
Firefox and Evolution
Every admin needs to be able to browse the Web to find answers to problems, and you’ll need to get email from your co-workers from time to time.
GNU Coreutils and other utilities
GNU’s Coreutils is a collection of utilities that all system administrators need from time to time, such as head, tail, wc, cat, and uniq. I also find a lot of use for grep, sed, and Gawk.
Filesystem tools
Oddly enough, file systems do get broken, usually from some overzealous newbie admin rebooting machines too often because they’re overloaded. OK, we’re all guilty of that, but sometimes it feels right to just yank the plug and start over fresh. When you do that, you’ll be cleaning up the filesystem using the filesystem tools such as fsck and mkfs.
Vim
Vim, the old, trustworthy, and found (almost) everywhere editor. Even though I have other choices for text editing, I prefer Vim.
SSH, secure copy (scp), and rsync
These three are invaluable. I know, rsync isn’t by the same folks, but the way we use them, they’re clumped together. Obviously, we use ssh to shell into machines, but we also run commands over ssh. The scp utility is good for sending one or more files, but rsync is so much better for synchronizing files if we don’t feel like resending the whole tree every time it may or may not have been overwritten.
ClusterSSH
ClusterSSH seems to be a lesser-known utility, but it’s very useful. I suppose not everyone has hundreds of machines to maintain, but some of us do. It’s very useful to batch things together. Think about installing a new monitoring program on, say, 200 machines. You could do it 200 times, or you could write a script to log in automatically and do it for you (assuming your SSH keys are in place), but then you have to pray that there are no odd errors.
I did exactly this, recently. We installed OCS Inventory NG (OCSNG) on all of our machines. I performed the installs 25 machines at a time using ClusterSSH. There were a few odd problems, but nothing that ruined the attempt. Of 25 machines, I’d find one or two older ones that were missing bits that kept it from installing. I’d run our own installer script to attempt all 25, and then go back and clean up the one or two failures. The whole process was very easy — using ClusterSSH, I installed OCSNG on 200 machines in less than an hour. If I’d had to do the installs individually, I would have had to exceed one machine every 18 seconds to do it as quickly, and the repetition would have killed me.
I’m using ClusterSSH today to work on a group of 10 new machines. There’s a feeling of power, when you tell 10 machines to rebuild their kernels all at once.
Perl
Perl is the sysadmin’s best friend. I’ve found that it can do virtually anything. The most intense script that I’ve written, just to prove I could do it, must be our monitoring program, BoT. (Sorry, he’s not good enough for distribution. He’s my baby, but I can’t let him out into the world until I work out more bugs.) BoT has been running constantly for approximately seven years. He notified us of some server faults that the Windows application we used for monitoring couldn’t see. We call BoT a “him” because he uses Festival to talk on the local machine, and has a male voice. He also sends out text messages and nagging reminders. He’s the most obnoxious person on staff, and never gets paid.
My task for this week is writing scripts to pull together our MRTG data, MAC addresses from the switches, and data from the OCS Inventory. Everything in the back end is in Perl.
Other things I’ve used Perl for include webcam mirror farm maintenance for a live cam site, general maintenance, and plenty of one-off purposes. Of course, I’ve used Perl extensively on Web sites as well. What fun is a Web site if you can’t do something dynamic with it? For the last few years, I’ve been switching my new Web development over to PHP.
Apache, PHP, MySQL
If you’re working on the Internet, Intranet, or even your local host, you probably should be using these. Again, we have our own pre-rolled version that makes distribution easy. Throw the tarball on, extract it, and type build.apache. It’ll build, it’ll upgrade, it’ll chop lettuce.
I’ll include the tiny/turbo/throttling HTTP server (thttpd) here also. This is a very noteworthy Web server, great for static content.
Captive
Captive, which implements NTFS read/write access for Linux, is my newest friend. I had to remove a particularly troublesome virus on a client’s Windows XP box. It was pretty much impossible to convince it to go away through any of the traditional means while in Windows. I could identify where they were, but they were impossible to get rid of. So, I went in with Captive from a Linux drive. I booted, wiped out all the pesky files, and rebooted back into Windows. The final parts of the cleanup were easy, and the client was satisfied.
I found Captive through the Trinity Rescue Kit (TRK). The TRK is very interesting, but development is very slow. I’m tempted to put my own kit together sometimes, assuming I ever have the time. I believe this is where the TRK folks are too — lots of work, and not enough time to accomplish it in.
MRTG
I probably shouldn’t rank Multi Router Traffic Grapher (MRTG) quite so low. MRTG graphs pretty much anything, and it does it very well. We’ve managed to find a few quirks in it, but it usually takes gobs of bandwidth, and overrunning counters to do it.
Let us know about your most valuable utilities (there need not be 10 of them), and if we publish your work, we’ll pay you $100.
