Troubleshooting with Apache logging

The Apache Web server (Apache) comes with a powerful logging framework. In the default configuration, Apache logs all errors to an error log and all access requests to an access log. The default level of logging is sufficient for analyzing traffic patterns and for getting basic information about errors, but it may be inadequate for troubleshooting purposes. Familiarity with all the logging features can help you troubleshoot the Web server or applications hosted on Apache.

In the default installation of Apache on Fedora, you can find the access log at /etc/httpd/logs/access_log and the error log at /etc/httpd/logs/error_log. The access log captures one line of information for each request. The error log captures the date and time of a request, the severity level of an event, the client’s IP address, and the description of the error. Error logging is a part of the core functionality of Apache, while other bits in the logging functionality come from modules such as mod_log_config, mod_dumpio, and mod_log_forensic.

You can customize the format of access log by using the configuration directive LogFormat in the configuration file httpd.conf (in /etc/httpd/conf directory on Fedora). For instance, you can use %b in the format specifier line to include the size of the response excluding the HTTP headers in the access log, and %t to include the time the request was received. You can use the directive LogFormat "%v %h %u %t "%r" %>s %b" to log the canonical server name of the server, remote host, authenticated remote user, time the request was received, first line of the request, status of the original request, and the size of the response in bytes. You should customize the access log to include all the data that you would need for analysis of traffic or troubleshooting. You can change the location of log files by editing the CustomLog and ErrorLog directives in httpd.conf.

The LogLevel directive determines what kind of events trigger logging to the error log. In decreasing order of severity, the log levels are emerg, alert, crit, error, warn, notice, info, and debug. When you choose a level of severity for error logging, Apache logs all events associated with that level and higher. The format of error log file is fixed, but every entry in the error log corresponding to a request has an associated entry in the access log, and you can customize the access log to collect additional information you need to troubleshoot errors.

It is good practice to have Apache log errors to an error log in $INSTALLDIR/logs/, but you can also write error events to syslog by using a config directive like ErrorLog syslog:local7 in httpd.conf. This directive configures Apache to log errors to syslog using the facility called local7. You can look up the man pages for syslogd and syslog.conf for more information about the syslog facilities.

Forensic logging

mod_log_forensic is an Apache module that helps in forensic analysis. With it, you can log the entire contents of a HTTP request to a file, which can be useful in troubleshooting access-related issues. You can load the module by using the directive LoadModule log_forensic_module modules/mod_log_forensic.so and specify the forensic log file by using the configuration directive ForensicLog in httpd.conf.

With mod_log_forensic loaded, Apache generates a unique ID called forensic-id for each request and includes it in the forensic logs. mod_log_forensic does not offer any flexibility in the log format. It logs two entries for each request. The first entry contains all the details related to the request, including the entire set of request headers, and is written before the processing of the request. The second line is the post prcessing line written to the log file after the processing is over; it contains just the forensic-id to indicate that the request was successful and a page was served to the client. A script bundled with Apache, check_forensic, can help you find out the requests without a post processing line. Based on this, you can identify any patterns in the failed requests and fix them.

Here is a sample entry from a forensic log file that shows the log for a successful request:

+cfe:475e285e:0|GET /test.php HTTP/1.1|Host:192.168.1.33|User-Agent:Mozilla/5.0 (X11; U; Linux i686; en-US; rv%3a1.8.1.3) Gecko/20070417 Fedora/2.0.0.3-4.fc7 Firefox/2.0.0.3|Accept:text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5|Accept-Encoding:gzip,deflate|Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.7|Keep-Alive:300|Connection:keep-alive
-cfe:475e285e:0

forensic-id generated by mod_log_forensic is also available for other moodules to use. You can add this unique ID into your access logs by adding %{forensic-id}n to LogFormat directive in httpd.conf. This gives you the ability to correlate entries in the access and forensic logs.

Forensic logs contain only the request data. When this is not sufficient to troubleshoot the problem, you can use the module mod_dumpio to dump the entire request and/or the entire response to the error log file. You need to load the module by adding LoadModule dumpio_module modules/mod_dumpio.so to the configuration file. You will need to enable the actual dumping of data by the configuration directives DumpIOInput On and DumpIOOutput On. You can capture the entire transaction between the Web server and the client by enabling IOInput and IOOutput; it can be useful in debugging problems related to the server.

Logging for other Apache modules

Some modules of Apache have their own logging directives. For instance, mod_rewrite has the directives RewriteLog and RewriteLogLevel. Rewrite log helps in troubleshooting when your rewrite directives do not work as expected.

You can log errors related to your CGI scripts using ScriptLog, ScriptLogBuffer, and ScriptLogLength. Remember to make the script log file writable for whichever user Apache runs as. ScriptLog captures the request headers and the text that the script writes to the standard output and standard error.

Most of the logging directives mentioned here should only be used for troubleshooting, as they can have a negative impact on the performance of the server and also use up a lot of disk space. You should use them with caution, and using them on a production server for long periods is not a good idea at all. But when used carefully, Apache’s logging functionality can help you greatly in zeroing in on problems.