3 Cool Linux Service Monitors

The Linux world abounds in monitoring apps of all kinds. We’re going to look at my three favorite service monitors: Apachetop, Monit, and Supervisor. They’re all small and fairly simple to use. apachetop is a simple real-time Apache monitor. Monit monitors and manages any service, and Supervisor is a nice tool for managing persistent scripts and commands without having to write init scripts for them.

Monit

Monit is my favorite, because provides the perfect blend of simplicity and functionality. To quote man monit:

monit is a utility for managing and monitoring processes, files, directories and filesystems on a Unix system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations. E.g. Monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses too much resources. You may use Monit to monitor files, directories and filesystems for changes, such as timestamps changes, checksum changes or size changes.

Monit is a good choice when you’re managing just a few machines, and don’t want to hassle with the complexity of something like Nagios or Chef. It works best as a single-host monitor, but it can also monitor remote services, which is useful when local services depend on them, such as database or file servers. The coolest feature is you can monitor any service, and you will see why in the configuration examples.

Let’s start with its simplest usage. Uncomment these lines in /etc/monit/monitrc:

 set daemon 120
 set httpd port 2812 and
     use address localhost  
     allow localhost        
     allow admin:monit      

Start Monit, and then use its command-line status checker:

$ sudo monit
$ sudo monit status
The Monit daemon 5.16 uptime: 9m 

System 'studio.alrac.net'
  status                  Running
  monitoring status       Monitored
  load average            [0.17] [0.23] [0.14]
  cpu                     0.8%us 0.2%sy 0.5%wa
  memory usage            835.7 MB [5.3%]
  swap usage              0 B [0.0%]
  data collected          Mon, 04 Sep 2017 13:04:59

If you see the message “/etc/monit/monitrc:289: Include failed — Success ‘/etc/monit/conf.d/*'” that is a bug, and you can safely ignore it.

Monit has a built-in HTTP server. Open a Web browser to http://localhost:2812. The default login is admin, monit, which is configured in /etc/monit/monitrc. You should see something like Figure 1 (below).

Click on the system name to see more statistics, including memory, CPU, and uptime.

That is fun and easy, and so is adding more services to monitor, like this example for the Apache HTTP server on Ubuntu.

check process apache with pidfile /var/run/apache2/apache2.pid
    start program = "service apache2 start" with timeout 60 seconds
    stop program  = "service apache2 stop"
    if cpu > 80% for 5 cycles then restart
    if totalmem > 200.0 MB for 5 cycles then restart
    if children > 250 then restart
    if loadavg(5min) greater than 10 for 8 cycles then stop
    depends on apache2.conf, apache2
    group server    

Use the appropriate commands for your Linux distribution. Find your PID file with this command:

echo $(. /etc/apache2/envvars && echo $APACHE_PID_FILE)

The various distros package Apache differently. For example, on Centos 7 use systemctl start/stop httpd.

After saving your changes, run the syntax checker, and then reload:

$ sudo monit -t
Control file syntax OK
$ sudo monit reload
Reinitializing monit daemon

This example shows how to monitor key files and alert you to changes. The Apache binary should not change, except when you upgrade.

    check file apache2
    with path /usr/sbin/apache2
    if failed checksum then exec "/watch/dog"
       else if recovered then alert

This example configures email alerting by adding my mailserver:

set mailserver smtp.alrac.net

monitrc includes a default email template, which you can tweak however you like.

man monit is well-written and thorough, and tells you everything you need to know, including command-line operation, reserved keywords, and complete syntax description.

apachetop

apachetop is a simple live monitor for Apache servers. It reads your Apache logs and displays updates in realtime. I use it as a fast easy debugging tool. You can test different URLs and see the results immediately: files requested, hits, and response times.

$ apachetop
last hit: 20:56:39         atop runtime:  0 days, 00:01:00             20:56:56
All:           12 reqs (   0.5/sec)         22.4K (  883.2B/sec)    1913.7B/req
2xx:       6 (50.0%) 3xx:       4 (33.3%) 4xx:     2 (16.7%) 5xx:     0 ( 0.0%)
R ( 30s):      12 reqs (   0.4/sec)         22.4K (  765.5B/sec)    1913.7B/req
2xx:       6 (50.0%) 3xx:       4 (33.3%) 4xx:     2 (16.7%) 5xx:     0 ( 0.0%)

 REQS REQ/S    KB KB/S URL
    5  0.19  17.2  0.7*/
    5  0.19   4.2  0.2 /icons/ubuntu-logo.png
    2  0.08   1.0  0.0 /favicon.ico

You can specify a particular logfile with the -f option, or multiple logfiles like this: apachetop -f logfile1 -f logfile2. Another useful option is -l, which makes all URLs lowercase. If the same URL appears as both uppercase and lowercase it will be counted as two different URLs.

Supervisor

Supervisor is a slick tool for managing scripts and commands that don’t have init scripts. It saves you from having to write your own, and it’s much easier to use than systemd.

On Debian/Ubuntu, Supervisor starts automatically after installation. Verify with ps:

$ ps ax|grep supervisord
 7306 ?        Ss     0:00 /usr/bin/python 
   /usr/bin/supervisord -n -c /etc/supervisor/supervisord.conf

Let’s take our Python hello world script from last week to practice with. Set it up in /etc/supervisor/conf.d/helloworld.conf:

[program:helloworld.py]
command=/bin/helloworld.py
autostart=true
autorestart=true
stderr_logfile=/var/log/hello/err.log
stdout_logfile=/var/log/hello/hello.log

Now Supervisor needs to re-read the conf.d/ directory, and then apply the changes:

$ sudo supervisorctl reread
$ sudo supervisorctl update

Check your new logfiles to verify that it’s running:

$ sudo supervisorctl reread
helloworld.py: available
carla@studio:~$ sudo supervisorctl update
helloworld.py: added process group
carla@studio:~$ tail /var/log/hello/hello.log
Hello World!
Hello World!
Hello World!
Hello World!

See? Easy.

Visit Supervisor for complete and excellent documentation.