Author: Nikos Mouat
OpenSSH (ssh/scp)
You cannot manage a machine unless you can access it safely and securely. Using Telnet to remotely access a machine is equivalent to not only leaving your car keys in the ignition while parking at the airport for an extended vacation, but also leaving the engine idling while doing so.
With ssh you can securely access all your machines, and with scp you can transfer files around to your heart’s content.
ps (procps)
You cannot manage a machine unless you know what is running on it. If you’re having performance problems, you need to be able to see what’s chewing up CPU, and if there are problem processes running. The ps utility, part of the Procps package, lets you see what processes are running on your system in great detail. I almost included top, but you can really see all you need to see with ps.
netstat
The ps utility lets you know what processes are running, but what really matters if you’re trying to run a secure server is what network services that you are offering. One of the first things I look at after a fresh install is the output of netstat -n to see what TCP and UDP ports I’ve got open. I then decide which of those I really want to have open, and close the others. As a bonus, you can see your route table with netstat -nr or list your network interfaces netstat -ni.
lsof
The lsof utility not only tells you what file every running process has open, it also lets you know what programs have what network ports open. If netstat shows me an open port that I am unfamiliar with, running lsof -n | grep -i tcp is my next step to figure out what process has that port open, and what user it’s running as.
vmstat
The vmstat utility reports information about your system’s CPU activity, disk input/output (I/O), processes, memory, and much more. If you’re trying to get an idea of what type of load your server is under, using vmstat 5 will help a lot in seeing this. If things are slow, it will give you an idea of where to look — from too much swapping, to being blocked too much waiting for I/O, or just being CPU-bound, this utility will let you know what’s up.
iostat
The iostat utility is kind of the next step from vmstat. It reports on the system’s CPU usage and I/O statistics for block devices and partitions. If you’re seeing a ton of processes in the “wa” (time spent waiting for I/O) field in vmstat reports, or with a process state of “D” (uninterruptible sleep) in ps output, this means you’re probably block waiting on I/O. The iostat utility will give you the details on I/O performance, and where your bottlenecks might be.
Pine
I can’t live without Pine, one of the more popular text-mode mail clients. My choice of Pine over Mutt, or other full screen text-mode clients, was somewhat arbitrary — based on it being the only available mail client on the machine I was using at the time.
ping
When you’re managing a server, you need to fix it when it goes down. Most of the time when any of my servers becomes unavailable, it’s not because of something going on with the server itself, but rather with the network connecting the server to everyone else. Ping is the first thing I use to diagnose a problem when it seems a server is unavailable.
traceroute
If ping is the first tool I use when my server is not responding, traceroute is the second. If you cannot ping something, then packets are getting lost somewhere, and traceroute tells me where.
tcpdump
Tcpdump helps me figure out what’s going on with my network. There are many other deserving packet monitoring tools — for instance, APS does a nice job of disassembling the headers and printing text in a readable format — but no packet sniffer is as ubiquitous as tcpdump.
There are, of course, many more tools that I need. As was mentioned in the original My Sysadmin Toolbox article, vim is a must have. Also dmesg, uptime, netcat, nmap, and even the who and last commands are all deserving of a spot in the toolbox.
Nikos Mouat currently serves as vice president of engineering at Ygnition Networks, an Internet and cable TV provider specializing in apartment communities. He also owns a small consulting practice specializing in Linux implementations, Internet routing, and network security.
Do you have a list of tools you’d like to share? Let us know about your most valuable utilities (there need not be 10 of them), and if we publish your work, we’ll pay you $100.
