In previous articles in this series, we’ve whet our whistles with a quick look at the Process Table and pseudo filesystems, and we talked about /dev and /proc. Now let’s explore a few useful but unrelated command lines, which may save the day at some point.
Counting Processes
You might be concerned that a single member of a cluster of identical machines is beginning to give up the ghost. You could check how many processes the system has created since its last reboot by using this command:
# grep processes /proc/stat
With your shovel at the read, now take a look inside the “/proc/$PID/cmdline” file (replacing “$PID” for your Process ID), and there you will find some buried goodies. This file retains the entire and complete command line for a process (note that zombie processes are an exception, so don’t fret if your mileage varies).
Alongside that file, in the PID’s directory, is the “cwd” symbolic link (or symlink) to the current working directory of that process.You may need to be logged in as root to see this. To discover the current working directory of that process, you can run this command:
# cd /proc/$PID/cwd; /bin/pwd
You do probably need to be root sometimes, because the symlink is hidden to normal users. I won’t claim to understand all of the intricacies of these pseudo files, but if you use the commands “cat” or “less” to view some of these files, then usually a little more light is shed on their raison d’ etre.
One such pseudo file (which is less mysterious, thanks to its name) is the disk input/output statistics file, named “io”. You can run the following command and see its output in Listing 1:
# cat io rchar: 0 wchar: 0 syscr: 0 syscw: 0 read_bytes: 342 write_bytes: 11 cancelled_write_bytes: 0
Listing 1: Here we can see what this process has been up to in relation to disk activities.
Among many others, one particularly useful addition is the “maps” file. In Listing 2, we can see the memory regions and their access permissions for a process by using:
# cat maps 7eff839c7000-7eff839de000 r-xp 00000000 fd:01 3221 /lib64/libpthread-2.12.so 7eff839de000-7eff83bde000 ---p 00017000 fd:01 3221 /lib64/libpthread-2.12.so 7eff83bde000-7eff83bdf000 r--p 00017000 fd:01 3221 /lib64/libpthread-2.12.so 7eff83bdf000-7eff83be0000 rw-p 00018000 fd:01 3221 /lib64/libpthread-2.12.so 7eff843ac000-7eff843b3000 r-xp 00000000 fd:01 3201 /lib64/libcrypt-2.12.so 7eff843b3000-7eff845b3000 ---p 00007000 fd:01 3201 /lib64/libcrypt-2.12.so 7eff845b3000-7eff845b4000 r--p 00007000 fd:01 3201 /lib64/libcrypt-2.12.so 7eff845b4000-7eff845b5000 rw-p 00008000 fd:01 3201 /lib64/libcrypt-2.12.so 7eff82fb4000-7eff83025000 r-xp 00000000 fd:01 3478 /lib64/libfreebl3.so 7eff83025000-7eff83224000 ---p 00071000 fd:01 3478 /lib64/libfreebl3.so 7eff83224000-7eff83226000 r--p 00070000 fd:01 3478 /lib64/libfreebl3.so 7eff83226000-7eff83227000 rw-p 00072000 fd:01 3478 /lib64/libfreebl3.so
Listing 2: An extremely abbreviated sample of the “maps” file for a process.
Apparently, the legend for the permissions is as follows:
r = read w = write x = execute s = shared p = private (copy on write)
The “maps” file can be useful to see how a process is interacting with the system’s files. Or, maybe you’re curious as to which libraries a process needs and you have forgotten the correct options to add to a tool like the super-duper “lsof” that I discussed in the previous article.
Any eagle-eyed console champions will spot that the file sizes, having used “ls” to list the
files in the directory, all appear as zero bytes. Clearly, these pseudo files are different animals than we’re used to.
Kernel Support
Let’s now move onto other benefits of /proc and not just on a per-process basis. For example, you might look at the filesystems which were compiled into the kernel by checking out the “/proc/filesystems” file.
In Listing 3, you can see (in abbreviated form) what filesystems our kernel supports and even then it’s a sizeable list.
nodev sysfs nodev rootfs nodev bdev nodev proc nodev cgroup nodev cpuset nodev tmpfs nodev devtmpfs nodev binfmt_misc nodev debugfs nodev securityfs nodev sockfs nodev usbfs nodev pipefs nodev anon_inodefs nodev inotifyfs nodev devpts nodev ramfs nodev hugetlbfs nodev pstore nodev mqueue
Listing 3: Here we can see an abbreviated list of the types of filesystems supported by the kernel without having to make any tricky changes.
You may have heard of an excellent lightweight utility called “vmstat”, which reports back dutifully with screeds of useful memory statistics.
You may not, at this juncture, fall off your seat in surprise when I say that in order to retrieve this useful information, the following files are used. Note the asterisk for all PIDs etc.
/proc/meminfo /proc/stat /proc/*/stat
Another aspect that the kernel deals with on a system is the hardware ports — you know, the kind that accept a keyboard or a mouse. Have a peek at Listing 4, as seen in the “/proc/ioports” file, again abbreviated for ease.
0000-001f : dma1 0020-0021 : pic1 0040-0043 : timer0 0050-0053 : timer1 0060-0060 : keyboard 0064-0064 : keyboard 0070-0071 : rtc0 0080-008f : dma page reg 00a0-00a1 : pic2 00c0-00df : dma2 00f0-00ff : fpu
Listing 4: Displaying an abbreviated output from the “/proc/ioports” file.
As expected, we can also look at network information. I’ll leave you to try some of these yourself. You might want to begin by trying this:
# ls /proc/net
Or, for live ARP information (the Address Resolution Protocol), you can use the “cat” command in front of this:
# cat proc/net/arp
If that output looks familiar, then think of how many utilities you’ve used in the past that refer to this file.
But, why stop with ARP? SNMP statistics are also readily available:
# cat /proc/net/snmp
There’s a whole heap (no memory puns intended) of other options that we could explore, but instead let’s look at changing some of the “sysctl” settings I mentioned earlier.
Sadly, the majority of the powerful changes you can make to a running system’s kernel are for another day (there’s simply too many to cover), but let’s take the popular “/proc/sys” directory for a quick spin before we finish.
What if the system-wide limit of the number of open files for every process on the system is holding you up when you’re trying to launch your latest ground-breaking application? This setting can cause all sorts of system headaches if it’s abused (and it appears that not only the “root” user can change it on occasion). Warnings aside, it’s very useful to arm yourself with this quick fix if it’s ever necessary.
Try this command yourself on a test box:
# echo 999999 > /proc/sys/fs/file-max
Once you’ve conquered that particular outage-inducing issue, taking all the glory as you go, how about checking that the new setting has stuck? To do this, you can treat it like any other file:
# cat /proc/sys/fs/file-max
I should have, of course, mentioned that you should check the original value before making any changes (so you can revert the value if need be). My computer used “6815744”.
What about another setting? This usually uses a start/stop daemon to make the change, on some systems, but let’s do it on the fly:
# echo "ChrisLinux" > /proc/sys/kernel/hostname
In case that you’re wondering, you will probably struggle to edit these files with an editor (you may get “file has changed” errors) hence the use of “echo” as above.
On other distributions, you can affect the same change with:
# hostname "ChrisLinux"
I’m sure you get the idea; it’s hardly computer science after all.
End of File
I have briefly looked at the very clever Unix-like process table, procfs, and /dev. I have also covered how to read and sometimes manipulate tunable settings and how they integrate with components running on Unix-like systems.
I hope these articles have given you some insight to know what to look if you encounter issues in the future. Above everything else, however, it’s important to find the time to fire up a test machine, dig into otherwise buried file paths, and tweak the available settings to increase your knowledge.
Incidentally, should you ever find yourself at a loss on a rainy afternoon, try running this command:
# /sbin/sysctl -a
That should definitely provide some food for thought. It helpfully lists all of the tunable kernel parameters on a system, each of which can be altered with “echo”; simply read the “dots” as “slashes” to find them inside /proc. Just don’t make ill-advised, arbitrary kernel changes on a production machine, or you will likely rue the day!
If you weren’t aware of some of the settings that I’ve covered here, then I hope you will enjoy your newly gained sys admin knowledge and use it creatively in the future.
Read the previous articles in this series:
Chris Binnie is a Technical Consultant with 20 years of Linux experience and a writer for Linux Magazine and Admin Magazine. His new book Linux Server Security: Hack and Defend teaches you how to launch sophisticated attacks, make your servers invisible and crack complex passwords.
