Author: Preston St. Pierre
The nightmare started when I returned to my office to discover 17 missed calls on my cell phone. A moment later, the phone rang again. A frantic customer complained of “eyes” on his Web site. I visit the site, and, lo and behold, discovered it proclaiming being “owned” by a hacking group. Upon further investigation, I discovered all the other Web sites on the Debian-based server had been defaced.
The attacker managed to scare the pants off of me, and worse yet, generated concern throughout my customers. On the other hand, this was a blessing. The attacker blatantly announced himself when he could have easily installed a small-footprint root kit and hidden in the trenches of my system sucking confidential customer and network information. To reduce the possibility of this kind of attack, I highly advise making use of system integrity software such as chkrootkit and tripwire daily. These programs help detect stealthy intrusions. Other signs of a system in jeopardy may be log files that appear to have been spontaneously erased and unfamiliar occurrences in root’s ~/.bash_history file.
Initial response
Once a system has been compromised, it is best to physically take it off the network. This severs communication with the hacker and allows you to investigate the status of the machine without the worry of further vandalism. In my case, I had to log in via SSH to see what was happening. Throughout all of my forensic procedures, I had to keep in mind that every system utility I was using (sshd, ls, vi, find, and ps) might have been altered to produce false results. I instantly became aware of compromised system executables when I tried to run ls -lh and the command failed to recognize -h as a valid option. Regardless, I was still able to successfully block the hacker’s access to my machine.
After logging on and becoming root with su, I checked for the presence of the hacker using the w command, which outputs the users logged on to the system and the command they’re running. There was an occurrence of the root user (other than mine) executing a command, and a user “w” that I’ve never heard of logged on. The situation was grim, so I changed my (root’s) password with the passwd command, edited /etc/passwd to delete the unknown “w” account (and all other unfamiliar accounts), and kicked the attacker off the system by killing his shell process (bash). At this point, it was time to secure the server.
Securing the server
To secure the server, you need to shut down all unnecessary services and strengthen the firewall. While my httpd, DNS, and email servers are necessary services, I still had to shut them down, because it’s likely that one of them was used by the attacker to gain access to the system. I shut down all my services except for the SSH daemon using /etc/init.d scripts. I then strengthened my firewall to allow network connections from my IP address only and restarted it.
With the server relatively secure, begin forensics by listing processes using the command ps -auxfw. The results of this command should be taken with a grain of salt, since if you’ve been hacked it is likely to have been replaced by an altered version designed to hide malicious processes installed by the hacker. On my system the command showed some very deviant processes; one called ./shell, another ./httpd, and yet another ./x0a. I saved the output of the ps command for later reference and killed the deviant processes. I noticed that user www-data was the owner of these processes, leading me to believe that the attacker gained elevated privileges by exploiting a vulnerability in a Web script. I then looked in temp directories /tmp, /var/tmp, and /www/php/tmp and found files installed by the hacker. I backed up the directories for future reference and deleted their contents.
I checked my site for obvious backdoors by typing netstat -lntpe. Aloha! There was a strange daemon named ttyload listening on port 4444. This process did not show up when I used the hacked ps command. I killed the process and archived and removed the file.
Now it was time to examine logs. Unsurprisingly, my /var/log directory was completely empty. After an attack is made, an attacker will try to cover his tracks by altering log files. After some digging I found the hacker had neglected to clear root’s /root/.bash_history file, and the unknown user’s history file as well! These files list the commands issued by a user, and thus provide solid evidence as to what the hacker executed. Looking them over, I noticed several programs were compiled, an exploit in the kernel’s BRK command was issued to gain root, and a rootkit (a program used by hackers to hide a back door into the system) was installed.
Because logs and history files are often altered, I recommend employing a redundant method to retain them. Set up syslog to log to a remote, well-firewalled server as well as locally. Another excellent method of protecting files and further securing your system is to implement LIDS, which can protect log files, as documented in the project’s FAQ.
Cleaning the system
In an effort to clean the system I installed chkrootkit from chkrootkit.org. This program is often updated with signatures of known rootkits and runs checks for them as well as altered commands and process trees. The program should be run as a daily cron job and its output reviewed or scanned to alert the system administrator of malicious actions. Executing chkrootkit resulted in the program finding an infected ifconfig command, process tree, and the t0rn V8 and showtee rootkits.
Clean-up time! I began by reinstalling system software. I replaced sshd, ifconfig, find, ls, md5sum, and the process tree using apt-get install -reinstall. The files were all part of base Debian packages, and I located the necessary .deb files by searching the debian.org package page. RPM-based distributions can replace files using rpm --install -force.
With reliable system utilities in place, I searched the system for files which had been modified since the time of the attack using find --newer /path/to/file_modified_before_attack > ~/hacked_files. I examined the output file (~/hacked_files) and deleted suspect files and directories. Interestingly, I noticed that user “500” and “501” was the owner of some of these files — a user obviously not existent on the system because no accounts exist in /etc/passwd with a uid of 500 or 501. Utilizing this knowledge, I found even more files installed by the hacker by typing find -user 501 in the / (root) directory. After removing all these files, I ran chkrootkit again. It found no infections! I then examined my system startup files (init scripts) for the “ttyload” hidden backdoor discovered during initial forensics. To do this, I executed for $i in *; do cat $i | grep ttyload done in the /etc directory. I found an entry calling ttyload in /etc/inittab and wiped it out.
Going live again
I had yet to nail down the attacker’s point of entry. I knew it was sourced in the Web server, so I checked the permissions of all Web sites and scripts. Nothing was seemingly awry.
After an attack like this, make sure your hosted Web sites are running the latest version of installed Web scripts. Employing a vulnerability scanner such as Nessus will aid you in detecting dangerous unpatched software. Update and or reinstall all your core services, like Apache, Exim/Qmail/Courier, PHP, and MySQL.
Fearful of another attack, I rebooted the system and stood on watch. As I stood vigilant over my logs, I was hit! The attacker was on the system again. With adrenaline pumping, I kicked him off, as I had before, and started the same process of cleaning the system again. This time, however, I had my logs, and they were being replicated remotely.
A revealing entry was toward the bottom — a HTTP request to awstats.pl with some malformed get data. The attacker was using awstats, my Web statistics program, to gain access to my server. Namely, he was using wget to retrieve a backdoor program to be executed by the Web server by using two simple malformed awstats.pl parameters.
Needless to say, I patched the program and restricted it access. Searching through SecurityFocus, a site that tracks such exploits, I found the vulnerability published, and shamed myself for not being on their mailing list and for allowing awstats to go unpatched. If you aren’t on the SecurityFocus mailing list, or that of another security advisory publication, such as linuxsecurity.com, sign up!
Wrapup
As all hacks are different, you shouldn’t consider my experiences and the lessons I learned from them to be a comprehensive rulebook for dealing with jeopardized systems. In a perfect world, an administrator is on-site and ready to pull a server from the network the moment it gets attacked — not to mention that the system would be impenetrable and the attacks just play. In my world, my Web server was hacked while I was miles away with only the option to rely on a presumably unreliable shell (hacked SSH daemon) to fix the system.
Still, in general, the main idea is to determine the method of intrusion — the steps the hacker took to compromise the system. Once the method is identified, you should reinstall your distribution from scratch while taking measures to plug the vulnerabilities. I hope my tips will help. And trust me, I know how you feel.
