3 Best Network Programming Debugging Tools
==========================================
It is always time consuming if we don’t use the right network debugging tools when do we socket programming or trying to run a client server program for the first time.
When we do network programming sometimes you want to know why send() from your client or
serverfailing, why I’m not re-start my server program, whether any other process is already using the port that
you are planning to use for your server etc.
There are many Tools available today in Linux. But we will see the most important 3 Tools in this article.
I.netstat
=========
Netstat command displays various network related information such as network connections, routing tables, interface statistics, masquerade connections, multicast memberships etc
1) Show the list of network interfaces
OpenSuse12.3#netstat -i
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0 1500 0 0 0 0 0 0 0 0 0 BMU
lo 65536 0 45 0 0 0 45 0 0 0 LRU
wlan0 1500 0 25092 0 0 0 22958 0 0 0 BMRU
2) To list all Ports(both listening and non-listening, TCP, UDP, Unix)
OpenSuse12.3#netstat -at
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 localhost:smtp *:* LISTEN
tcp 0 0 192.168.1.5:6688 *:* LISTEN
tcp 0 0 192.168.1.5:49875 safebrowsing:www-http ESTABLISHED
tcp 0 1 192.168.1.5:60804 fls.doubleclic:www-http FIN_WAIT1
tcp 0 0 192.168.1.5:43589 safebrowsing.c:www-http ESTABLISHED
tcp 0 0 *:33532 *:* LISTEN
unix 2 [ ACC ] STREAM LISTENING 8645 /var/run/sdp
unix 2 [ ] DGRAM 12241
3) List only TCP Port
OpenSuse12.3#netstat -at
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 localhost:smtp *:* LISTEN
tcp 0 0 192.168.1.5:6688 *:* LISTEN
tcp 0 0 localhost:ipp *:* LISTEN
tcp 0 0 *:33532 *:* LISTEN
Similarly for UDP, “netstat -au”
3) List the Sockets which are in Listening state
OpenSuse12.3#netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 localhost:smtp *:* LISTEN
tcp 0 0 192.168.1.5:6688 *:* LISTEN
tcp 0 0 *:52980 *:* LISTEN
tcp 0 0 localhost:ipp *:* LISTEN
tcp 0 0 *:33532 *:* LISTEN
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 10227 private/scache
unix 2 [ ACC ] STREAM LISTENING 11714 @/tmp/dbus-NmyF9Qx2gH
List only listening TCP Ports using netstat -lt
List only listening UDP Ports using netstat -lu
List only the listening UNIX Ports using netstat -lx
4) Display PID and program names in netstat output using netstat -p
5) Print netstat information continuously
netstat -c
6) Find out on which port a program is running
OpenSuse12.3#netstat -ap | grep servermine
tcp 0 0 192.168.1.5:6688 *:* LISTEN 2135/servermine
II. tcpdump
===========
tcpdump allows us to capture all packets that are received and sent. This helps us to see what all tcp segments are
sent and received (like SYN, FIN, RST etc) and can understand the root cause of our issue.
1) Capture packets from a particular ethernet interface using tcpdump -i
tcpdump capture for a simple tcp client & server example starting from SYN to FIN/ACK with one data packet in between.
OpenSuse12.3#tcpdump -i lo
11:05:27.026304 IP 192.168.1.5.34289 > 192.168.1.5.6688: Flags [S], seq 1990318384, win 43690, options [mss 65495,sackOK,TS val 6116309 ecr 0,nop,wscale 7], length 0
11:05:27.026331 IP 192.168.1.5.6688 > 192.168.1.5.34289: Flags [S.], seq 3856734826, ack 1990318385, win 43690, options [mss 65495,sackOK,TS val 6116309 ecr 6116309,nop,wscale 7], length 0
11:05:27.026357 IP 192.168.1.5.34289 > 192.168.1.5.6688: Flags [.], ack 1, win 342, options [nop,nop,TS val 6116309 ecr 6116309], length 0
11:05:27.026689 IP 192.168.1.5.6688 > 192.168.1.5.34289: Flags [P.], seq 1:27, ack 1, win 342, options [nop,nop,TS val 6116310 ecr 6116309], length 26
11:05:27.026703 IP 192.168.1.5.34289 > 192.168.1.5.6688: Flags [.], ack 27, win 342, options [nop,nop,TS val 6116310 ecr 6116310], length 0
11:05:27.026839 IP 192.168.1.5.34289 > 192.168.1.5.6688: Flags [F.], seq 1, ack 27, win 342, options [nop,nop,TS val 6116310 ecr 6116310], length 0
11:05:27.027445 IP 192.168.1.5.6688 > 192.168.1.5.34289: Flags [.], ack 2, win 342, options [nop,nop,TS val 6116311 ecr 6116310], length 0
11:05:32.026898 IP 192.168.1.5.6688 > 192.168.1.5.34289: Flags [F.], seq 27, ack 2, win 342, options [nop,nop,TS val 6121310 ecr 6116310], length 0
11:05:32.026920 IP 192.168.1.5.34289 > 192.168.1.5.6688: Flags [.], ack 28, win 342, options [nop,nop,TS val 6121310 ecr 6121310], len
2) Capture only N number of packets using tcpdump -c
OpenSuse12.3#tcpdump -c 100 -i lo
capture only 100 packets
3) Capture the packets and write into a file using tcpdump -w
OpenSuse12.3# tcpdump -w myprogamdump.pcap -i lo
tcpdump: listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
9 packets captured
18 packets received by filter
0 packets dropped by kernel
4) Reading/viewing the packets from a saved file using tcpdump -r
OpenSuse12.3#tcpdump -tttt -r myprogamdump.pcap
reading from file myprogamdump.pcap, link-type EN10MB (Ethernet)
2013-11-30 11:12:55.019872 IP 192.168.1.5.34290 > 192.168.1.5.6688: Flags [S], seq 2718665633, win 43690, options [mss 65495,sackOK,TS val 6564303 ecr 0,nop,wscale 7], length 0
2013-11-30 11:12:55.019899 IP 192.168.1.5.6688 > 192.168.1.5.34290: Flags [S.], seq 2448605009, ack 2718665634, win 43690, options [mss 65495,sackOK,TS val 6564303 ecr 6564303,nop,wscale 7], length 0
2013-11-30 11:12:55.019929 IP 192.168.1.5.34290 > 192.168.1.5.6688: Flags [.], ack 1, win 342, options [nop,nop,TS val 6564303 ecr 6564303], length 0
2013-11-30 11:12:55.020228 IP 192.168.1.5.6688 > 192.168.1.5.34290: Flags [P.], seq 1:27, ack 1, win 342, options [nop,nop,TS val 6564303 ecr 6564303], length 26
2013-11-30 11:12:55.020243 IP 192.168.1.5.34290 > 192.168.1.5.6688: Flags [.], ack 27, win 342, options [nop,nop,TS val 6564303 ecr 6564303], length 0
2013-11-30 11:12:55.020346 IP 192.168.1.5.34290 > 192.168.1.5.6688: Flags [F.], seq 1, ack 27, win 342, options [nop,nop,TS val 6564303 ecr 6564303], length 0
2013-11-30 11:12:55.020442 IP 192.168.1.5.6688 > 192.168.1.5.34290: Flags [.], ack 2, win 342, options [nop,nop,TS val 6564304 ecr 6564303], length 0
2013-11-30 11:13:00.020477 IP 192.168.1.5.6688 > 192.168.1.5.34290: Flags [F.], seq 27, ack 2, win 342, options [nop,nop,TS val 6569304 ecr 6564303], length 0
2013-11-30 11:13:00.020506 IP 192.168.1.5.34290 > 192.168.1.5.6688: Flags [.], ack 28, win 342, options [nop,nop,TS val 6569304 ecr 6569304], length 0
5) Receive only the packets of a specific protocol type like arp, tcp, udp, ip etc
OpenSuse12.3#tcpdump -i wlan0 ip
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wlan0, link-type EN10MB (Ethernet), capture size 65535 bytes
11:18:04.193704 IP 132.213.238.6.http > 192.168.1.5.32991: Flags [.], seq 2723848246:2723849686, ack 3820601748, win 6432, options [nop,nop,TS val 786299612 ecr 6873162], length 1440
11:18:04.194241 IP 192.168.1.5.50414 > 192.168.1.1.domain: 36798+ PTR? 5.1.168.192.in-addr.arpa. (42)
11:18:04.196315 IP 132.213.238.6.http > 192.168.1.5.32991: Flags [P.], seq 1440:2880, ack 1, win 6432, options [nop,nop,TS val 786299612 ecr 6873162], length 1440
5) Receive packets flows on a particular port using tcpdump port
tcpdump -i eth0 port 4040
6) Capture packets for particular destination IP and Port
tcpdump -w mypackets.pcap -i eth0 dst 192.168.1.6 and port 22
III. lsof
=========
lsof meaning ‘LiSt Open Files’ is used to find out which files are open by which process. As we all know Linux/Unix considers everything as a files (pipes, sockets, directories, devices etc).
So by using lsof, you can get the information about any opened files. But here we primarily see options related to
network files.
1) List all network connections
OpenSuse12.3#lsof -i
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 1 root 32u IPv6 6955 0t0 TCP *:ipp (LISTEN)
avahi-dae 475 avahi 11u IPv4 9245 0t0 UDP *:mdns
avahi-dae 475 avahi 14u IPv6 9248 0t0 UDP *:46627
master 766 root 12u IPv4 10100 0t0 TCP localhost:smtp (LISTEN)
2) List processes which are listening on a particular port
OpenSuse12.3#lsof -i :6688
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
servermine 3127 prince 3u IPv4 1256979 0t0 TCP 192.168.1.5:6688 (LISTEN)
3) List all TCP or UDP connections
OpenSuse12.3#lsof -i tcp; lsof -i udp
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 1 root 32u IPv6 6955 0t0 TCP *:ipp (LISTEN)
master 766 root 12u IPv4 10100 0t0 TCP localhost:smtp (LISTEN)
master 766 root 13u IPv6 10102 0t0 TCP localhost:smtp (LISTEN)
gnome-ses 800 prince 13u IPv6 11789 0t0 TCP *:33532 (LISTEN)
gnome-ses 800 prince 14u IPv4 11790 0t0 TCP *:52980 (LISTEN)
cupsd 1029 root 4u IPv6 6955 0t0 TCP *:ipp (LISTEN)
cupsd 1029 root 10u IPv4 12739 0t0 TCP localhost:ipp (LISTEN)
4) List all IPv4 and IPv6 network files
OpenSuse12.3#lsof -i 4
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 1 root 33u IPv4 6956 0t0 UDP *:ipp
avahi-dae 475 avahi 11u IPv4 9245 0t0 UDP *:mdns
avahi-dae 475 avahi 13u IPv4 9247 0t0 UDP *:37715
master 766 root 12u IPv4 10100 0t0 TCP localhost:smtp (LISTEN)
gnome-ses 800 prince 14u IPv4 11790 0t0 TCP *:52980 (LISTEN)
dhclient 926 root 6u IPv4 12038 0t0 UDP *:bootpc
OpenSuse12.3#lsof -i 6
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 1 root 32u IPv6 6955 0t0 TCP *:ipp (LISTEN)
avahi-dae 475 avahi 12u IPv6 9246 0t0 UDP *:mdns
avahi-dae 475 avahi 14u IPv6 9248 0t0 UDP *:46627
master 766 root 13u IPv6 10102 0t0 TCP localhost:smtp (LISTEN)
gnome-ses 800 prince 13u IPv6 11789 0t0 TCP *:33532 (LISTEN)
dhclient 926 root 21u IPv6 12022 0t0 UDP *:55332
cupsd 1029 root 4u IPv6 6955 0t0 TCP *:ipp (LISTEN)
5) To list all the running process of open files of TCP Port ranges from 1-1024
OpenSuse12.3#lsof -i TCP:1-1024
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 1 root 32u IPv6 6955 0t0 TCP *:ipp (LISTEN)
master 766 root 12u IPv4 10100 0t0 TCP localhost:smtp (LISTEN)
master 766 root 13u IPv6 10102 0t0 TCP localhost:smtp (LISTEN)
cupsd 1029 root 4u IPv6 6955 0t0 TCP *:ipp (LISTEN)
cupsd 1029 root 10u IPv4 12739 0t0 TCP localhost:ipp (LISTEN)
6) List all network files in use by a specific process
OpenSuse12.3#lsof -i -a -p 234
7) list of all open files belonging to all active processes
OpenSuse12.3#lsof
COMMAND PID TID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 1 root cwd DIR 8,6 4096 2 /
systemd 1 root rtd DIR 8,6 4096 2 /
systemd 1 root mem REG 8,6 126480 131141 /lib64/libselinux.so.1
systemd 1 root mem REG 8,6 163493 131128 /lib64/ld-2.17.so
systemd 1 root 0u CHR 1,3 0t0 2595 /dev/null
systemd 1 root 6r DIR 0,18 0 3637 /sys/fs/cgroup/systemd/system
systemd 1 root 16u unix 0xffff88007c0ec100 0t0 3857 socket
