One of the keys to protecting a Linux system is to know what’s going on inside it — what files change, who accesses what and when, and which applications get run. Incrond was used up until some years ago for the former, but, despite rumors to the contrary, development seems to have stopped since about four years ago. Nonetheless, you can still download and use it and try out the examples I talked about in a tutorial published elsewhere.
The newer systemd contains some features that allow for monitoring, but it is a bit clumsy, and the feedback you get is far from detailed enough for a forensic analysis or to run an event-specific application.
These days, your best bet to monitor all your stuff is probably auditd. Auditd is also a good option because, apart from running comprehensive checks, the auditing itself happens at the kernel level, below userspace, which makes it much harder to subvert. This is an advantage over shell-based auditing systems, which will not give accurate information if the system is already compromised before they run.
Audit is actively developed by Red Hat and is available for most, if not all, major distributions. If it is not already installed on your system, you can find it by searching in your system’s repositories. In Debian-based systems, the package is simply called audit, while in RPM-based systems, it shows up as auditd. In most Red Hat-related systems, such as Fedora and CentOS, auditd is usually installed by default.
Auditd is made up of several components, but for our purposes today, what you need are: auditd itself, which is the actual daemon that monitors the system, and aureport, a tool that generates reports culled from the auditd’s logs.
Installation
First things first, though. Install the audit or auditd package using your distribution’s software manager and check that it is running. Most modern Linux distributions run auditd as a systemd service, so you can use
> systemctl status auditd.service
to see if it’s active once installed. If it is there, but not running, you can jumpstart it with
> systemctl start auditd.service
or configure it to run at boot with
> systemctl enable auditd.service
Before checking reports and so on, let it run for a while, so it can fill up its logs with events.
Reporting System
Right out of the box, auditd already logs some stuff it deems critical, no extra configuration needed. You can check what it is looking up by running aureport without any arguments — note that you must be root or have root privileges (i.e., use sudo) to access audit’s toolbox:
> aureport Summary Report ====================== Range of time in logs: 18/05/16 09:47:34.453 - 22/05/16 11:28:03.168 Selected time for report: 18/05/16 09:47:34 - 22/05/16 11:28:03.168 Number of changes in configuration: 195 Number of changes to accounts, groups, or roles: 30 Number of logins: 5 Number of failed logins: 0 Number of authentications: 136 Number of failed authentications: 9 Number of users: 4 Number of terminals: 12 Number of host names: 2 Number of executables: 13 . . .
So, this is already interesting! Take a look at the line that says Number of failed authentications: 9. If you see a large number here, somebody may be trying to access your machine by brute-forcing a user’s password.
Let’s dig in a little deeper:
> aureport -au Authentication Report ============================================ # date time acct host term exe success event ============================================ 1. 18/05/16 09:47:56 sddm ? ? /usr/lib/sddm/sddm-helper yes 187 2. 18/05/16 09:48:09 paul ? ? /usr/lib/sddm/sddm-helper yes 199 3. 18/05/16 09:41:29 root ? pts/1 /usr/bin/su yes 227 4. 18/05/16 09:57:16 root ? pts/2 /usr/bin/su yes 231 5. 18/05/16 10:01:57 root ? ? /usr/sbin/groupadd yes 235 . . .
The -au option allows you to see details pertaining to authentication attempts: aureport gives you dates and times, the account being accessed, and whether the authentication was successful.
If you narrow down the output down by filtering it through grep:
> aureport -au | grep no 37. 18/05/16 12:18:24 root ? pts/0 /usr/bin/su no 217 38. 18/05/16 12:18:38 root ? pts/0 /usr/bin/su no 218 47. 18/05/16 12:41:15 root ? pts/1 /usr/bin/su no 262 66. 18/05/16 14:09:55 root ? pts/4 /usr/bin/su no 220 67. 18/05/16 14:10:05 root ? pts/4 /usr/bin/su no 221 102. 20/05/16 12:37:07 root ? pts/5 /usr/bin/su no 191 117. 21/05/16 12:10:39 root ? pts/0 /usr/bin/su no 229 129. 21/05/16 17:59:08 root ? pts/1 /usr/bin/su no 208 134. 21/05/16 18:32:05 root ? pts/0 /usr/bin/su no 248
you get all the failed authentication attempts. For our little experiment, remember the last line of the report, line 134, and the time and date, 21/05/16 18:32:05, of the last failed access attempt to access the root account.
Let’s now look at the users report:
> aureport -u -i User ID Report ==================================== # date time auid term host exe event ==================================== 1. 18/05/16 09:47:35 unset ? ? /usr/lib/systemd/systemd 136 2. 18/05/16 09:47:35 unset ? ? /usr/lib/systemd/systemd-update-utmp 137 3. 18/05/16 09:47:35 unset ? ? /usr/lib/systemd/systemd 138 4. 18/05/16 09:47:45 unset ? ? /usr/lib/systemd/systemd 139 5. 18/05/16 09:47:45 unset ? ? /usr/lib/systemd/systemd 140 . . .
The -u option tells aureport to show user activity, and -i tells it to show the user names instead of their ID numbers.
Again, aureport gives us a lot to sift through. What we want to know is who last tried to access root, but failed. If we copy the date and time of the last authentication attempt, that is, 21/05/16 18:32:05 and use that with grep to filter out some of the data, we get:
> aureport -u -i|grep "21/05/16 18:32:05" 2324. 21/05/16 18:32:05 paul pts/3 ? /usr/bin/su 201
Whoops! It was me. Let me clarify that I was not up to anything nefarious. It’s just that I am a klutz and often mistype the root password in my own computer.
With a bit of command-line fu, you could do all of the above in one fell swoop:
> accessdate=`aureport -au | grep no | tail -1 | cut -d ' ' -f 2,3`; aureport -u -i | grep "$accessdate"; unset accessdate 2324. 21/05/16 18:32:05 paul pts/3 ? /usr/bin/su 201
Or you could use a temporary intermediate file to show all failed authentication attempts:
> aureport -au | grep no | cut -d ' ' -f 2,3 > accessdates.log; aureport -u -i | grep -f accessdates.log; rm accessdates.log 986. 18/05/16 14:09:55 paul pts/4 ? /usr/bin/su 220 987. 18/05/16 14:10:05 paul pts/4 ? /usr/bin/su 221 1577. 20/05/16 12:37:07 paul pts/5 ? /usr/bin/su 191 1785. 21/05/16 12:10:39 paul pts/0 ? /usr/bin/su 229 2050. 21/05/16 17:59:08 paul pts/1 ? /usr/bin/su 208 2090. 21/05/16 18:32:05 paul pts/0 ? /usr/bin/su 248
As I said, I am a bit of a klutz.
More to Come
There is, of course, a lot more to auditd. I haven’t even touched upon what it is most often used for, that is, customized monitoring of files and directories. I’ll be looking at how to do that and much more in future installments.
