Author: JT Smith
netcat
By far, the most popular suggestion from readers was to replace Telnet with netcat. Netcat was so popular, in fact, that I decided to write up an introductory CLI Magic column on netcat for readers who might not be familiar with the joys of netcat.
Netcat can be used to test services in the same way that Telnet can be used to test services, but is much more flexible, with additional features you won’t find with Telnet.
tcpdump
Readers also suggested tcpdump as an application that one should have on hand when doing systems administration. As the name suggests, tcpdump will print out (dump) packets passing through a network interface, allowing an admin to eyeball network traffic more closely. You can use the application to dump packets to standard out and watch them fly by, or save them to a file to review later, or both.
The tcpdump utility is handy for a lot of troubleshooting and testing. For instance, it lets you see what kind of traffic an application is sending from your system — and, sometimes more importantly, where it’s sending it to.
But tcpdump wouldn’t be terribly useful if you had to look through all traffic transmitted on a network interface, when you’re only concerned about a small subset of traffic. Using tcpdump, it’s possible to get very specific about what traffic is captured and presented. For example, it’s possible to tell tcpdump to display only traffic to one host, or of one protocol, or packets that are transmitted over a specific port — or to exclude a specific host, port, or protocol and display everything else. Combinations of those elements are also possible, of course.
The bottom line is that tcpdump is invaluable in many situations.
Ethereal
While tcpdump is extremely useful at capturing packets for examination, its presentation can leave something to be desired. Admins who need to deal with this kind of data on a regular basis should look into the Ethereal network protocol analyzer.
Ethereal can read tcpdump’s capture files, as well as a slew of other formats used by other popular tools, such as Sun’s snoop and atmsnoop, Microsoft’s Network Monitor, Novell LANalyzer, HP-UX nettl, and many others.
Unlike most of the tools that I’ve touched on so far, Ethereal is a GUI application. It presents protocol information in an easy-to-read fashion that makes analyzing captured traffic easy. According to the Ethereal site, it “dissects” 724 protocols, including HTTP, AIM, IMAP, NetBIOS, MySQL, and many, many more.
Ethereal also offers the ability to follow a specific TCP stream, can create I/O graphs, provide traffic summaries and statistics by protocol, and a lot more. If you’re doing any serious work that requires analyzing network traffic, you want Ethereal handy.
nmap
Need to scan your network, find out whether a specific IP address is in use, or want to find out what services are running on a host? Nmap is a must-have. A quick nmap hostname can show an admin what services are running on a host.
The nmap utility can also help determine whether a firewall is stateful, or merely blocking incoming packets. It can usually detect a remote host’s operating system, and even the remote host’s uptime. In short, nmap can reveal a wealth of information about remote systems, which can be extremely useful when you’re examining your servers to see what potential attackers might be able to learn about your systems.
MultiTail
Most admins are already familiar with using tail -f logfile to watch system, application, and error logs when they’re troubleshooting. However, the tail utility only follows one file at a time. If you need to watch two or more logfiles at the same time, which is fairly common, the MultiTail utility by Folkert van Heusden is an excellent tool to have handy.
Basically, MultiTail handles two or more files simultaneously, and presents them in a split-screen view that makes it convenient to watch two, three, or more logfiles at the same time. This has come in particularly handy for me when troubleshooting issues with Web sites and mail issues. Technically, one can replicate the MultiTail experience by using GNU Screen to create a split screen environment and using tail -f logfile multiple times — but why? MultiTail makes it simple.
find
Several readers suggested the find utility as an essential tool, and I have to agree. As a rule, I don’t need find to actually find files, but I use it pretty often to perform operations on multiple files at the same time.
As an example, if I wanted to find all files that have not been modified for more than 60 days, I’d run find -mtime +60 to get the list. If I wanted to do something with those files — probably delete them — all that I would need to do is run find -mtime +60 -exec rm {} ;. (I recommend running the find command without the -exec command the first time, however, if you’re going to delete or modify files, so that there are no surprises.)
I’ve also used find to change ownership on multiple files. Let’s say you have a directory tree with a lot of files that are owned by user bert, and you want to change the ownership to ernie. Brute force chown -R ernie * isn’t the answer, since some of the files may not be owned by bert, and you don’t want to touch them. But find -user bert -exec chown ernie {} ; does the trick nicely, without modifying any files that aren’t owned by bert.
Find is, of course, also exceedingly handy for finding files. If you’ve misplaced a file or two, you can track them down by file name, date, file type (symlink, regular file, directory, etc.), user, group, permissions, and so forth.
If you’re new to the find utility, be sure to check out Joe Barr’s excellent CLI Magic column on GNU find.
xargs
The xargs utility is used to take output from other programs to build a command line. For example, you might have a text file with a list of files that you want to delete using rm. The rm command doesn’t read from a file, and won’t accept output from cat directly — so you can’t just cat a list of file names at rm and expect it to do the right thing. However, xargs allows you to work around this limitation by using cat to feed the list of file names to xargs, which can then send the file names to the rm command, like this: cat filename | xargs rm.
That, of course, is a very simplistic demo of xargs’ capabilities. Another use for xargs is to limit the number of processes that are executed at one time. This can be useful, for example, when deleting a large number of files when the rm command complains that there are too many arguments.
I don’t use xargs more than once or twice a week, on average, but it is extremely useful to have around when I need it.
AWK and Sed
A couple of readers deemed Sed and AWK as “must have” tools for systems administration. Sed is a “stream editor,” and mightily handy for editing text on the fly. There are plenty of times when admins need to process a stream of text, either when it comes directly from another application, or when you’re making changes to a large number of files at the same time.
The AWK programming language is also well-liked by systems admins. The implementation you’ll find on Linux (and the one I use) is the GNU Project’s implementation, Gawk. Like Sed, AWK is the cat’s meow for manipulating batches of text files or extracting data from text files.
bash
Another popular suggestion is the bash shell itself. Again, I heartily agree that bash is extraordinarily useful. I didn’t count bash originally, because I wasn’t really thinking of it as a separate tool — it’s part of the environment, like the basic utilities that one expects to have on almost any *nix operating system: cp, mv, ls, ps, top, and so forth.
But, the more I thought about it, I think bash does deserve some recognition as a tool in its own right. If I’m using an OS that doesn’t have bash installed by default, say FreeBSD, installing bash is one of the first things I do.
CVS
Most people think of Concurrent Versions System (CVS) as a tool for developers rather than system administrators. However, there are a lot of reasons why system administrators find CVS useful as well — and I’m not just talking about pulling down code from CVS repositories.
For example, many systems administrators check configuration files into CVS to be able to provision new machines quickly, or to roll back to known-good configurations if a modification munges things up. Some admins are even known to use revision control to store their home directory.
Of course, this isn’t specific to CVS. Any of the popular revision control systems, like Subversion or monotone, can serve the same purpose.
Share your toolbox
Thanks again to all the readers who wrote in or posted comments with great suggestions of tools and utilities for system administrators. Even with the expanded list, I’m sure there are plenty of other tools and utilities that we’ve yet to cover, and we’d also like to hear about how they’re being used. Tell us about your favorite tools and utilities, and how you use them, and if we publish your sysadmin toolbox article, we’ll pay you $100.
