A problematic scenario, sometimes caused as a result of an application’s logging misconfiguration, is when the /var/log/messages file fills up, meaning that the /var partition becomes full. Thanks to the very nature of how systems work, there are always times when your system’s logging cause unexpected issues. Thus, it’s key that you understand and have the ability to control how your logging works and where your logs are saved.
Over the years when using Unix-like systems, I’ve been exposed to three or four different versions of the operating system’s default logging tool, known as Syslog. Here I will look at the logging daemon called rsyslog, a superfast Syslog product.
Before looking at the package itself, we’ll start by exploring the configuration of your system logging locally. Then we will use that knowledge to go one step further and configure a remote Syslog server. We will also explore certain aspects which might cause you problems. With some background information in addition we will also improve your troubleshooting.
As mentioned, in this series, we will be looking at rsyslog. We will focus on a long-established variety of Syslog, dating back to 2004, which has become a favorite among Linux distributions. As a result, rsyslog is now bundled as the default Syslog daemon on a number of the popular Unix-like flavors. It’s a substantial piece of software which is super fast, extensible and reportedly available on around 10 Unix-like distributions.
To get started, I’ll provide an introduction into how your Linux system thinks of its logging at a basic level.
Logging Detail
For years, sysadmins have debated at what level of detail to log their system data. There are a number of settings to affect the amount of logging detail which your server generates. We will look at how to configure the varying levels shortly. It’s a tradeoff between using up disk space too quickly versus not having enough information in your logs.
Let’s look at some of the detail settings which we can choose between. In Listing 1, we can see how the kernel ranks its errors, from zero to seven.
#define KERN_EMERG "<0>" /* system is unusable */ #define KERN_ALERT "<1>" /* action must be taken immediately */ #define KERN_CRIT "<2>" /* critical conditions */ #define KERN_ERR "<3>" /* error conditions */ #define KERN_WARNING "<4>" /* warning conditions */ #define KERN_NOTICE "<5>" /* normal but significant condition */ #define KERN_INFO "<6>" /* informational */ #define KERN_DEBUG "<7>" /* debug-level messages */
Listing 1: How the system file “kernel.h” defines logging levels in some cases.
The rsyslog basics
When first looking at a new package it is usually a good indication of how tricky that package might be to pick up from the syntax of its configuration. The following syntax is the type that “sysklogd” used in the past and the modern rsyslog uses too:
mail.info /var/log/mail.log mail.err @ server.chrisbinnie.tld
Thankfully, it is very simple once you understand the basics. The function of this logging example is also known as forwarding. The info logging mentioned (or in other words “informational errors”) is simply dropped locally into the file /var/log/mail.log. Whereas the err, or error condition logs, are sent off somewhere else to another server. In this case that server is called server.chrisbinnie.tld. Incidentally, the “INFO” and “info” settings are not case-sensitive and its case is frequently interchanged for clarity within Syslog config.
The writers of the powerful Syslog software, rsyslog, (Rainer Gerhards is apparently the main author) refers to it as “The Rocket-Fast System For Log Processing.”
This high-performance package can seemingly manage a million messages per second (when asked to drop local logging events to disk if “limited processing” is applied)! That’s impressive by any measure.
The home page of the rsyslog includes a few of its features:
-
Multi-threading
-
TCP, SSL, TLS, RELP
-
MySQL, PostgreSQL, Oracle, and more
-
Filter any part of syslog message
-
Fully configurable output format
-
Suitable for enterprise-class relay chains
That is exactly the functionality that you might look for in a best-of-breed Syslog daemon. In the next article, I’ll take a detailed look at the main config file. Then, I’ll cover logfile rules, log rotation, and some important networking considerations.
Chris Binnie is a Technical Consultant with 20 years of Linux experience and a writer for Linux Magazine and Admin Magazine. His new book Linux Server Security: Hack and Defend teaches you how to launch sophisticated attacks, make your servers invisible and crack complex passwords.
Advance your career in system administration! Check out the Essentials of System Administration course from The Linux Foundation.
