I watch a lot of the Linux boards and various distribution sites these days and notice the trend of wanting everything to be bigger, better and flashier. With great projects like Beryl (I love distracting Windows administrators with the cube…it’s like watching a bug fly into one of those electric blue lights), and developers in the open source community constantly improving and designing ever more functional GUI interfaces (think tools such as Yast2 or Synaptic) it is easy to start letting your CLI skills get rusty and forget about your little but mighty friends on the console. Unfortunately with the sheer amount of sensitivity in the world these days I need to be certain I preface that I use GUIs and am by no means belittling, demeaning, pointing fingers and laughing nor looking to taunt you a second time-ah, however, I do believe that maintaining CLI skills and commands are imperative to good Linux administration. Nothing like being woken at two in the morning by an urgent alarm that your server sporting Tux has gone down and will not boot back to a display manger (yes yes, I know this doesn’t happen…but let’s think about this for a moment strictly as pilots in a simulator…training). Below are a few command line tools that I have found to be very helpful and informative when it comes to administrating Linux. They may not always be readily visible or well known, but they pack a very powerful punch. Now to the nitty and gritty…
WATCH — watch is a neat and versatile tool. Ever sat there frustrated running a particular command over and over just to see what is changing? watch automates the command you wish to run and repeats it every x seconds. The format is watch -n . The default time is 2 seconds if you leave out the -n flag. Lets say you want to watch connections on port 80. By issuing “watch lsof -i TCP:80” watch will run the command “lsof -i TCP:80” every 2 seconds and display the results in a terminal. You will see that I started out with just consulting the oracle (google), but then I figured I’d go see how my boys were doing up at Ft. Sill and cruise the news pages over on yahoo. Watch happily provided the results.
#watch lsof -i TCP:80
Every 2.0s: lsof -i TCP:80 Fri Jul 20 07:58:31 2007
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
iceweasel 10030 user 36u IPv4 29054 TCP 172.31.31.131:59331->hs-in-f99.google.com:www (ESTABLISHED)
iceweasel 10030 user 37u IPv4 28547 TCP 172.31.31.131:58539->yo-in-f103.google.com:www (ESTABLISHED)Every 2.0s: lsof -i TCP:80 Fri Jul 20 07:58:38 2007 <---- notice the time
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
iceweasel 10030 user 34u IPv4 31794 TCP 172.31.31.131:46377->unknown.xeex.net:www (ESTABLISHED)
iceweasel 10030 user 42u IPv4 32130 TCP 172.31.31.131:44143->jc-in-f99.google.com:www (ESTABLISHED)
iceweasel 10030 user 43u IPv4 33178 TCP 172.31.31.131:51561->l1.ycs.vip.mud.yahoo.com:www (ESTABLISHED)
iceweasel 10030 user 44u IPv4 32178 TCP 172.31.31.131:44130->yo-in-f147.google.com:www (ESTABLISHED)
iceweasel 10030 user 45u IPv4 32587 TCP 172.31.31.131:55275->204.0.3.91:www (ESTABLISHED)
iceweasel 10030 user 46u IPv4 32190 TCP 172.31.31.131:51485->sill-www.army.mil:www (ESTABLISHED)
iceweasel 10030 user 47u IPv4 33177 TCP 172.31.31.131:51560->l1.ycs.vip.mud.yahoo.com:www (ESTABLISHED)
iceweasel 10030 user 48u IPv4 32907 TCP 172.31.31.131:45806->204.0.3.75:www (ESTABLISHED)
iceweasel 10030 user 52u IPv4 33181 TCP 172.31.31.131:51562->l1.ycs.vip.mud.yahoo.com:www (CLOSE_WAIT)
Maybe you want to watch a particular user and what processes they are running: watch -n 1 ‘ps aux | grep ‘ . If you toss in the -d flag, each time the console refreshes it will highlight the differences from the last execution. You can get creative with watch, just remember it is actually executing the command.
JUMPGATE – jumpgate became my friend awhile back when I administered some wireless access points that did not have a connection to the backhaul due to specialized protocols and security. In effect I needed a TCP forwarder to send data back and forth between boxes. By issuing jumpgate you can take a machine and have it handle the forwarding and receiving for your client. Format: jumpgate -b <localhost or IP address> -l -a -r
For example, in my situation I needed my local client to authenticate to a certificate server to be issued a certificate to allow it to join a secure network. The problem was my client could not reach the internal network where this certificate server resided. However, it could reach the gateway linux server and that server could reach the certificate box. I ran jumpgate on the gateway to forward my request.
#jumpgate -b 192.168.1.1 -l 80 -a 192.168.15.2 -r 8080
Then when I opened my browser and was prompted for the address of the certificate server by my local linux box I gave it the gateway’s address instead (192.168.1.1). Once the packet hits the gateway, jumpgate is listening on that particular port and forwards it to the remote destination (192.168.15.2) where the certificate server was actually listening for requests on port 8080. Jumpgate includes options for logging or being interactive such as: jumpgate -i -l 32000 -f jumpgaterequest.log. This tells jumpgate to bind and listen for connections on port 32000, then interactively ask the user where they want the file transferred to when a connection is made and log the session to file jumpgaterequest.log.
LSOF – lsof is a personal favorite of mine. In fact I submitted a tip on lsof before. It’s versatility is great, especially when you are troubleshooting an issue and need more information about process or connection details. This command elegantly stands for list open files. Linux treats most everything as a file. Sockets, devices, directories, etc, can all be viewed as files. When a process or application interacts with these files it has to “open” them if you will. Using this command you can delve into and see what your system is up to.
For instance to show all the open TCP files - Will return what service is running, who is running it, the process ID and the connections on all TCP ports:
# lsof -i TCP
Show open TCP files on port 80 -
# lsof -i TCP:80
returns --> httpd2-wo 7010 wwwrun 3u IPv6 14787 TCP *:http (LISTEN)
Show open LDAP connections on TCP -
# lsof -i TCP:636
Want to know what files are open by a particular command (substitute your command after the c, and yes you can abbreviate it matches the closest command)
# lsof -c mysq
mysqld 991 admin cwd DIR 8,3 240 148743 /home/admin/novell/idm/mysql/data
mysqld 991 admin rtd DIR 8,3 536 2 /
mysqld 991 admin txt REG 8,3 5464060 148691 /home/admin/novell/idm/mysql/bin/mysqld
mysqld 991 admin 0r CHR 1,3 41715 /dev/null
mysqld 991 admin 1w REG 8,3 1250 149954 /home/admin/novell/idm/mysql/mysql.log
mysqld 991 admin 2w REG 8,3 1250 149954 /home/admin/novell/idm/mysql/mysql.log
mysqld 991 admin 3u IPv4 86990 TCP *:63306 (LISTEN)...
Want to know what files are open by a particular device?
#lsof /dev/cdrom
bash 30904 admin cwd DIR 3,0 2048 63692 /media/cdrecorder/linux/user_application_provisioning
You can change TCP to UDP and narrow down your requests to very specific items you want to target (i.e. is there an established connection from xyz.somesite.com?).
# lsof -i This e-mail address is being protected from spambots. You need JavaScript enabled to view it .0.2:636 (lists LDAP connections to my server)
returns --> java 890 root 18u IPv6 8365030 TCP myserver.somecompany.com:42936->myserver.somecompany.com:ldaps (ESTABLISHED)
ndsd 6520 root 262u IPv4 8390927 TCP myserver.somecompany.com:ldaps->myserver.somecompany.com:43123 (ESTABLISHED)
ATOP – As an administrator you are used to typing ‘top’ to see real-time tasks and system information. atop takes it a step further and injects some steroids (in a safe and humane fashion) to allow you to flex some muscle and watch your processes and system information in much greater detail. I like the fact that I can watch read/writes from the disk as well as that it offers a quick snapshot of my network device and memory usage (VGROW and RGROW). Another neat feature is that it takes samples and logs these in /var/log/ for later review. This can be very useful for system analysis (i.e. you keep hearing complaints that the server is sluggish during certain times of the day…you can review these samples watching for high utilization or memory leaks). When you fire up atop you can press h and it will bring up the help screen. There are many options for you to tweak this command.
#atop
ATOP ? MYLAPTOP 2007/07/25 08:25:29 10 Seconds elapsed
PRC | sys 0.04s | user 0.26s | #thr 150 | #zombie 0 | #exit 0 |
CPU | sys 1% | user 3% | irq 2% | idle 195% | wait 0% |
cpu | sys 0% | user 2% | irq 2% | idle 96% | cpu000 w 0% |
cpu | sys 0% | user 1% | irq 0% | idle 99% | cpu001 w 0% |
MEM | tot 2.0G | free 982.6M | cache 491.5M | buff 123.1M | slab 60.6M |
SWP | tot 3.0G | free 3.0G | | vmcom 869.0M | vmlim 4.0G |
DSK | sda | busy 0% | read 0 | write 4 | avio 2 ms |
NET | transport | tcpi 1 | tcpo 1 | udpi 0 | udpo 0 |
NET | network | ipi 5 | ipo 1 | ipfrw 0 | deliv 1 |
NET | dev eth0 | pcki 6 | pcko 1 | in 0 Kbps | out 0 Kbps |
PID SYSCPU USRCPU VGROW RGROW USERNAME THR ST EXC S CPU CMD 1/2 5463 0.02s 0.13s 0K 0K root 1 -- - R 2% Xorg
6152 0.00s 0.07s 0K 0K user 1 -- - S 1% metacity
10953 0.01s 0.03s 0K 0K user 8 -- - S 0% firefox-bin
11100 0.01s 0.01s 0K 0K root 1 -- - R 0% atop
6143 0.00s 0.01s 0K 0K user 1 -- - S 0% gnome-panel
6701 0.00s 0.01s 0K 0K cupsys 1 -- - S 0% cupsd
6148 0.00s 0.00s 0K 0K user 1 -- - S 0% nautilus
6270 0.00s 0.00s 0K 0K user 2 -- - S 0% gnome-terminal
7269 0.00s 0.00s 0K 0K user 1 -- - S 0% notification-d
6167 0.00s 0.00s 0K 0K user 4 -- - S 0% gnome-cups-ico
6244 0.00s 0.00s 0K 0K user 1 -- - S 0% gnome-screensa
IFTOP – Curious about what network traffic is flowing around you? Pop this command into your console and get instant feedback. Of interest (although it is not shown in my example below) is that bars will appear as bandwidth is used up. Since I’m on a test network while I write this, there is not enough traffic generation for iftop to scale up. I like it because you can toggle DNS resolution, show source or destination or both, sort by column or source or destination and a sweet feature is to freeze the order. If you have something you want to specifically watch, you can type o (the letter not the number) and it locks onto just those source/destination connections. Again, typing h brings up the help and options screen. To be honest I usually leave iftop up and running during the day to keep an eye on what is going on with my network.
#iftop
1.91Mb 3.81Mb 5.72Mb 7.63Mb 9.54Mb
+-----------------------------------------------
255.255.255.255 => 192.168.10.12 0b 0b 0b
<= 256b 179b 166b
192.168.1.255 => user1.machine.net 0b 0b 0b
<= 0b 169b 133b
172.18.62.255 => 172.18.62.107 0b 0b 0b
<= 0b 0b 55b
172.18.62.255 => user1.machine.net 0b 0b 0b
<= 0b 0b 55b
172.18.62.255 => user2.machine.net 0b 0b 0b
<= 0b 0b 55b
192.168.1.1 => ALL-SYSTEMS.MCAST.NET 0b 0b 6b
<= 0b 0b 0b
------------------------------------------------
TX: cumm: 654KB peak: 0b rates: 0b 0b 0b
RX: 5.59MB 2.02Kb 256b 348b 470b
TOTAL: 6.23MB 2.02Kb 256b 348b 470b
PSTREE – Run ‘ps aux’ to have it spew out its results and you are stuck combing through the list checking which process is related to the another process. It’s ingrained to type ‘ps aux’ as an administrator but pstree simplifies the display by taking the process status and building it out as tree. The results are clean and it allows you to rapidly check parent and child relationships (good parenting skills are always a bonus in life…remember to teach your children Linux early on). The key word there is rapidly. If you need or desire to see the PID inside the tree insert the -p flag.
#pstree
init---NetworkManager---3*[{NetworkManager}]
+-NetworkManagerD
+-acpid
+-amarokapp---ruby
? +-6*[{amarokapp}]
+-atd
+-atop
+-avahi-daemon---avahi-daemon
+-bonobo-activati---{bonobo-activati}
+-cron
+-gksu---firestarter---{firestarter}
+-gnome-keyring-d
+-gnome-power-man
+-gnome-screensav
+-gnome-settings----{gnome-settings-}
+-gnome-terminal---bash---iftop---3*[{iftop}] <--- check it out, it shows iftop and atop running
? +-bash---atop
? +-bash---su---bash
? +-bash---su---bash---pstree
? +-gnome-pty-helpe
? +-{gnome-terminal}
NETCAT (NC) –netcat or nc is aptly nicknamed the swiss army utility knife of networking. This is another personal favorite. Although this tool is still popular often times folks forget about it (except those who wear b/w/g hats). Official sounding terminology from the man page, is: it is “a simple unix utility which reads and writes data across network connections, using TCP or UDP protocol. It is designed to be a reliable “back-end” tool that can be used directly or easily driven by other programs and scripts”.
Using an earlier scenario of machines that were separated, and now adding the fact I needed to flash images, I could combine the use of jumpgate and dd with netcat and have my remote system flash itself (no cops or court dates appear with this type of flashing).
Quick imaging: On the box I wish to flash I issue:
# nc -l -p 23000 | dd of=/dev/hda <--- tells netcat to listen on port 23000 and pipe whatever it receives to dd
On the client-side (box I am sending the image from) I issue:
# dd if=/dev/hda | nc 192.168.1.1 23000 <--- takes the output from dd and pipes it over to a listening server
Don't have nmap loaded? No worries, netcat will scan ports for you:
# netcat -vv -z 192.168.1.20 8000-9200 <--- using -vv tells netcat to be very verbose and -z tells it to scan the range (8000-9200) you can
substitute your own port range values
(UNKNOWN) [172.18.67.81] 9104 (?) : Connection refused
(UNKNOWN) [172.18.67.81] 9103 (bacula-sd) : Connection refused <--- interesting it has a network backup daemon
(UNKNOWN) [172.18.67.81] 9102 (bacula-fd) : Connection refused
(UNKNOWN) [172.18.67.81] 9101 (bacula-dir) : Connection refused
(UNKNOWN) [172.18.67.81] 9100 (?) open <--- Hey, I discovered I have a device that is running print service using HP JetDirect
(UNKNOWN) [172.18.67.81] 9099 (?) : Connection refused
(UNKNOWN) [172.18.67.81] 9098 (xinetd) : Connection refused
If you do not care about or need a secure transmission you could skip using scp and modify nc to transfer files, on the server side:
# nc -l -vv -p 9000 > myfilename.txt <--- sets up netcat to listen on port 9000, output whatever it receives to said filename, and be verbose about it so
I can watch (if I have a console open on the server)
And on the client side I issue:
# nc -vv 192.168.1.1 9000 < myfilename.txt <--- feed my file using nc to the server at 192.168.1.1 port 9000 and tell me about it while doing it
STRACE — strace is a wicked little debugger. This command is fancy to ToTo in the Wizard of Oz, it will peel back the curtain and show you what levers and wheels the great Oz is working when you execute a command. Ever wonder why or specifically where your compile and makefile was vomiting? Pondering why your application hangs and just appears to be caught in a time/space vortex? strace allows you to watch step by step what the kernel is performing when you request action. Now the warning for you is that it reveals A LOT of information that will require a little patience on your part when combing through the data. However, this little bit of patience will pay off when you are able to find where you need to modify your code or add libraries or whatever the problem is when your command errors out. Below is an example output (truncated/edited for space) from syncing the hardware clock to system clock:
#strace hwclock --hctosys
execve("/sbin/hwclock", ["hwclock", "--hctosys"], [/* 32 vars */]) = 0
brk(0) = 0x608000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b7148539000
uname({sys="Linux", node="L01395", ...}) = 0
access("/etc/ld.so.nohwcap", F_OK) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b714853a000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=78580, ...}) = 0
mmap(NULL, 78580, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2b714853c000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = 0
open("/lib/libc.so.6", O_RDONLY) = 3
read(3, "177ELF211ÔøΩÔøΩÔøΩÔøΩÔøΩÔøΩÔøΩÔøΩÔøΩ3ÔøΩ>ÔøΩ1ÔøΩÔøΩÔøΩ340331"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1367432, ...}) = 0
mmap(NULL, 3473592, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2b714873a000
mprotect(0x2b7148881000, 2097152, PROT_NONE) = 0
mmap(0x2b7148a81000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP...
brk(0) = 0x608000
brk(0x629000) = 0x629000
open("/usr/lib/locale/locale-archive", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/locale.alias", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2586, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b714853c000
read(3, "# Locale name alias data base.
#"..., 4096) = 2586
...
stat("/etc/adjtime", {st_mode=S_IFREG|0644, st_size=46, ...}) = 0
open("/etc/adjtime", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=46, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b714865f000
read(3, "0.293746 1185223896 0.000000
118"..., 4096) = 46
close(3) = 0
munmap(0x2b714865f000, 4096) = 0
open("/dev/rtc", O_RDONLY) = 3
ioctl(3, RTC_RD_TIME, {tm_sec=26, tm_min=51, tm_hour=7, tm_mday=24, tm_mon=6, tm_year=107, ...}) = 0
ioctl(3, RTC_RD_TIME, {tm_sec=26, tm_min=51, tm_hour=7, tm_mday=24, tm_mon=6, tm_year=107, ...}) = 0
...
close(3) = 0
munmap(0x2b714865f000, 4096) = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=861, ...}) = 0
settimeofday({1185281487, 0}, {300, 103079215111}) = 0
exit_group(0) = ?
Process 9729 detached
These commands and suggestions are strictly my two cents (and when adjusted for inflation usually worth less than that). However, central to having an understanding of Linux is having an understanding of how to interact and interpret data that the kernel will serve up. These CLI tools will run on any Linux distribution. Not all of these tools are included by default though, so included below are links to those that probably are not already installed. I like GUIs…so no I’m not writing this tip sheet in vi…but I do believe learning CLI and using the Linux console is akin to learning to drive a standard. Once you understand the clutch and gas routine you can drive anything. CLI is the standard that never lets you down. Happy computing!
