Out of the box, Linux runs just fine for many uses. But if you find yourself needing to ferret out performance problems or tune the kernel for better performance, Linux has more than enough tools to measure and tweak system performance. In this guide, we’ll take a look at five of the best utilities to measure system performance and tweak the Linux kernel.
You’ll find many, many, many utilities for measuring and improving Linux system performance. The utilities discussed here are chosen because they’re widely available, and provide a good scope functionality. Also, I chose to avoid tools that have too steep a learning curve. As an example, SystemTap is a bit more complex than should be covered in an overview article. I left out some of the more basic tools such as ps and top because most Linux users are exposed to them fairly early on. And I didn’t try to tackle any application-specific tools.
This won’t be the last word on system performance and administration on Linux.com, so don’t worry. In future tutorials, I’ll dive in a bit deeper with specific tools and tuning: but this should be a good starter kit for newer Linux admins.
KDE System Monitor
The KDE System Monitor and GNOME System Monitor are both good for getting a quick visual representation of the state of your system. For me, the KDE System Monitor has a bit of an edge over its GNOME counterpart because it allows you to monitor remote systems in addition to the local host, so we’ll look at that one in a bit more depth. Of course, you can run the KDE System Monitor on a GNOME desktop, and the accompanying daemon that provides data can be run on a system without a desktop environment at all.
If you don’t have the KDE System Monitor installed, look for a package called ksysguard. You’ll also want to install ksysguardd, which is the daemon that provides data to KDE System Monitor frontend.
Once installed, just run the KDE System Monitor from the menu or ksysguard from the command line. When it starts up, you’ll see three tabs by default that include a process table, system load, and read/write totals for the system disks. Nice, but hardly all of what this tool can do. KDE System Monitor has a set of “sensors” that can be used to display memory usage, CPU load, disk throughput, network activity, and more.
To fire up your very own custom monitors, go to File -> New Tab. You’ll be asked to specify the number of rows and columns for the tab. I usually opt for one to four monitors per tab. Once you’ve added a tab, you can drag one or more of the sensors to one of the cells on the tab. It’s not entirely obvious, but you can have more than one sensor per cell. So, for instance, if you would like to have the free and used memory in swap displayed in a single cell, you can do that.
If you want to monitor a remote system, go to File -> Monitor Remote Machine. Assuming you have ksysguardd running on the remote host, you should be able to just accept the defaults. You’ll probably want to have an SSH key set up so you don’t need to provide a password to log into the remote system, as the default connection is made over SSH.
When you’ve set up a tab to your liking, you can save it for later use. KDE System Monitor will save the file as a .sgrd file. KDE System Monitor is extremely easy to use, and shouldn’t take much time to get set up to be useful. The only real problem with this tool is that it’s mostly targeted at real-time monitoring, and doesn’t have a logging option.
Monitor Your System With Dstat
Like the KDE System Monitor, dstat is a sort of general-purpose system profiling tool, but it’s CLI-based. I like dstat for a number of reasons. First, because it covers all the bases: it’s usable to log data or view it in real time; it can show CPU load, disk I/O, network receive/send, etc. It also produces colorized output at the console, which makes it somewhat easier to scan results quickly.
Dstat is written by Dag Wieers. Unlike some of the performance tools, it includes a decent amount of documentation. And it’s extensible; plugins can be added to Dstat with Python. If you want to analyze your data later, you can use the –output option to write the information to a CSV file suitable for import into OpenOffice.org, Gnumeric, or whatever your favorite spreadsheet might be.
If you want to see a “full” output that includes CPU usage by CPU (for SMP systems), disk read and write, network traffic, and paging, use dstat -f.
The most recent version of dstat ships with quite a few plugins that are suitable for servers and personal systems. This includes plugins to examine MySQL performance, Postfix, Sendmail, Qmail, and much more. You can combine the plugins with built-in functions of dstat as well. So, for instance, if you want to watch MySQL I/O along with the -a (all) option, you can use dstat –mysql-io -a. Some experimentation may be required and not all options can be combined with other options.
One word of caution: dstat can have some impact on system performance itself. You might want to profile the plugins you’re using and get a baseline of system performance running with dstat to make sure that you’re accounting for its impact. See the performance.txt file under the docs directory in the dstat tarball. Packages are available for quite a few distros as well, though the pointers on the dstat site itself may lead to some older releases. (For example, Ubuntu Breezy, which is a bit long in the tooth at this point.)
The Sysstat Auite
Sysstat is actually a collection of several utilities, including sar, iostat, mpstat, and pidstat. You’ll actually find a few versions of sar floating around. There’s the atsar package, and sar included with the sysstat package. Here we’re going to look at the sysstat tools, but you might run into slight variations if you’re using the atsar version.
Why am I mentioning this one if I’ve already covered dstat? The individual tools in the sysstat suite are good to know, and you might want to familiarize yourself with the “standard” tool that is pre-packaged and more widely available.
The sar utility is good for collecting data over the long term. When you install the sysstat package on most distros, it will automatically add cron jobs that will collect data to be displayed with sar. Note that you can get a good graphical display of sar data using the isag package or kSar.
Just running sar will show CPU usage. sar -b will give you information about I/O being written and read from discs, and sar -B will give you paging information. The sar -A command will display pretty much all of the day’s data, including reads and writes by partition, memory usage, paging information, and much more.
The pidstat utility can be used to display a great deal of information about processes running on a system. You can examine one process or all active processes. For example, running pidstat -r 5 will display running processes that have page faults and show how many are minor and major faults. A lot of major page faults and you will start seeing serious performance problems. This can help nail down which processes are the problem.
The iostat command will let you get a quick read (get it?) on I/O stats for the CPU, local disks, and NFS mounts. You can opt to view all devices or narrow down by device. The mpstat command will let you see the CPU usage and idle information for the system and by individual CPUs. It takes some practice to become adept at using the sysstat suite, but it’s worth learning.
Network Monitoring with Ifstat
The ifstat utility is to network interface activity what iostat is to device I/O and CPU activity. You can use it to display activity on one or more network interfaces. When you run ifstat without any arguments or options, it will display the traffic for your standard network interfaces. You can specify one or more interfaces, and add a few options to make the results easier to read and work with.
If you’re running ifstat on a system that has unused or idle interfaces, use ifstat -z to hide interfaces that are idle (for example, a system with VMware that may not be using all the vmnet interfaces at the time).
To see all the bandwidth being pushed by a system, the -T option tells ifstat to display a total of all interfaces in addition to the individual tallies. To add a timestamp at the beginning of each line, use -t. By default the information is updated every second. If you want to crank this down a bit, you can specify a delay by adding a number at the end of the command. So, ifstat -Tt 3 will give you a display with the count updated every three seconds, and a total tally at the end of the display.
Finally, if you only want ifstat to run a few times, you can specify a second number to tell it to repeat that many times. To update 10 times, for example, you might use ifstat -tT 5 10 to get 10 updates five seconds apart. Generally speaking, ifstat is easy to use and get started with.
Tune the Linux Kernel with Sysctl
When it comes to actual performance tuning, you’ll want to know sysctl. The sysctl utility is actually used to configure kernel parameters stored in /proc/sys/. The parameters will change with each kernel version, and how it’s configured. To see all the parameters that are available, run sysctl -a, or sysctl -a | sort if you want them alphabetically.
It should go without saying that if you’re going to tweak these settings you should either be working on a test system, or be quite sure what you’re doing. Or both. Preferably both.
To change one of the parameters, run sysctl -w key.value=”newvalue”. Note that this will only hold the change until the next time the system is rebooted. To make a setting permanent, you can add it to /etc/sysctl.conf.
That’s all for now. If you’re interested in doing some further, more in-depth, reading you’ll want to start with IBM’s Redbooks Guide Linux Performance and Tuning Guidelines, which is relatively current (updated April 2008) and specifically looks at Red Hat and SUSE Linux. It’s a good, detailed look at performance tuning.
