In the first article in this series, I introduced the rsyslog tool for logging, and in the second article I provided a detailed look at the main config file. Here, I’ll cover some logfile rules and sample configurations.
I’m a Lumberjack
Now for the juicy stuff as we get our hands a bit dirtier with some logfile rules. Listing 1 shows us the rules included by default with rsyslog on my Debbie-and-Ian machine:
auth,authpriv.* /var/log/auth.log *.*;auth,authpriv.none -/var/log/syslog #cron.* /var/log/cron.log daemon.* -/var/log/daemon.log kern.* -/var/log/kern.log lpr.* -/var/log/lpr.log mail.* -/var/log/mail.log user.* -/var/log/user.log
Listing 1: The standard rules included on the Debian Jessie operating system.
Since I covered the syntax previously, I hope there are no nasty surprises in Listing 1. If you wanted to add lots of content to one log file in particular (the following example is from a Red Hat box) then you would separate entries as so:
*.info;mail.none;authpriv.none;cron.none /var/log/messages
As you can see we’re throwing a fair amount at the “messages” log file in the example above. Each entry, let’s use “mail.none” as our example, follows a “facility.priority” format.
So, in the Red Hat example above for the “mail” facility, the config “mail.none” speaks volumes whereas to capture “all” mail logs, the config would be “mail.*” as seen in Listing 1. The “none” may be merrily be replaced with any of the 0-7 error codes shown in the very first listing shown in the first article, such as INFO.
The docs talk about both the “facility” and the “priority” being case-insensitive and how they can also receive decimal numbers for arguments. Take note from the manual, however, that’s generally a bad idea: “but don’t do that, you have been warned.”
And, news just in (not really): the documentation is explicit about the “priority” settings “error,” “warn,” and “panic” no longer being used as they are deprecated. Note that this is not visible in other docs that I have read so it likely applies to newer versions.
A final point would be on the way that rsyslog deals with its error levels (a reminder of what we saw in previously and also to take heed that some of those are now deprecated in newer versions). The manual is typically very helpful in its order of “priority” and discusses them as is displayed in Listing 2.
emerg (panic may also be used) alert crit error (err may also be used) warn (warning may also be used) notice info debug
Listing 2: Preferred rsyslog error levels with those now deprecated struck-through (version v8-stable as of writing).
Onwards we cheerily go. From a “facility” perspective, you can use the options as displayed in Listing 3.
auth authpriv cron daemon kern lpr mail mark news security (equivalent to “auth”) syslog user uucp local0 ... local7
Listing 3: Available options for the “facility” setting, abbreviated and missing “local1” to “local6”.
With your newfound knowledge, I’m sure that it goes without saying that if you see any asterisks mentioned then it simply means that “all” of the available “facility” options or all of the “priority” options are included.
Note the configurable “local” settings from zero to seven missing from the abbreviated content in Listing 3. This brings us nicely onto our next section, namely how to configure a remote rsyslog server.
Ladies And Gentlemen
I hope you’ll agree that the above configs are all relatively easy to follow. What about setting your logs live so that they are recorded onto a remote rsyslog server? If you’re sitting comfortably, here’s how to do just that.
First, let’s think about a few things. Consider how busy your logs are. If you’re simply pushing a few errors (because of an attack or a full drive) over to your remote syslog server, then your network isn’t going to be under much pressure. Imagine a very busy web server, however, and you’re going to want to analyze the hits that it receives, using something like the Kibana logging analysis tool via Elasticsearch, for example. That busy server might be pushing any number of uncompressed gigabytes of data across your LAN, and it’s important to bear in mind that these hits will occur 24/7 on a popular website.
In such a scenario, it is clearly key that your logs are all received without fail to ensure the integrity of your log analysis. The challenge is that the logs grow continually, unremittingly, and are generated every second of every day as visitors move around your website.
There’s also a pretty serious gotcha in relation to the rotation of logs (which there may well be a way of circumventing that am I yet to discover on the version (v5.8.10) of Syslog I was using). When you’re creating compendious logs, the sizes can grow so large that you feel like you might begin to encroach on your nearest landfill site. As a result, at some point your disks start to creak at the seams (no matter how big they are) and you have to slice up your logs and preferably compress them, too.
One of the most popular tools for rotating logs is the truly excellent logrotate, of which I’m a big fan. The clever logrotate is well-built, feature-filled, and most importantly highly reliable (logs are valuable commodities after all; especially for forensic analysis following an attack or after an expensive web infrastructure investment to ensure that the bang-for-buck ratio is satisfactory).
The gotcha, which I referred to a moment ago, surfaces in a fairly simple guise. When a log is rotated, the usually reliable rsyslog stops logging at the remote server end — even though the local logs continue to grow. It looks like some people have had problems on other distributions.
When faced with such a pickle, from what I could see at least, there simply weren’t config options to provide a workaround (even having tried different “Polling” configs and $InputFilePersistStateInterval workarounds; these might make more sense shortly). However, and I hold my hands up, it’s quite possible that I may have missed something. In my defense, I was stuck with an older version that couldn’t be upgraded (it’s a long story) and possibly that made a difference. Before we see the solution I chose, let’s look at how to create the remote logging config.
Remember the directory which we looked at in addition to the config file? I’m referring to the /etc/rsyslog.d directory. Well, that’s where we insert our remote rsyslog server config. We dutifully create a file called something like www-syslog.chrisbinnie.tld.conf to refer to our logging server’s hostname, appending a .conf on the end and a www- for the service being in question, which is logged. I’m using the hostname as an example in case your sanity is truly questionable and you want to push different application logs off to various servers. This naming convention should serve you well, if so.
Next time, we’ll look at the entirety of the /etc/rsyslog.d/www-rsyslog.chrisbinnie.tld.conf file and discuss some important networking considerations.
Read the other articles in this series:
Remote Logging With Syslog, Part 1: The Basics
Remote Logging With Syslog, Part 2: Main Config File
Chris Binnie is a Technical Consultant with 20 years of Linux experience and a writer for Linux Magazine and Admin Magazine. His new book Linux Server Security: Hack and Defend teaches you how to launch sophisticated attacks, make your servers invisible and crack complex passwords.
Advance your career in system administration! Check out the Essentials of System Administration course from The Linux Foundation.
