Mail server benchmarking with Postal

The Postal project includes three programs aimed at benchmarking mail server performance. The main program, postal, sends email messages to a specified list of destination addresses at a specified rate. Postal can let you see how fast your system can process incoming email and thus can help you measure improvements to your mail server when you are making software and hardware changes. For example, you can use postal to tell you whether switching to a different IMAP server will allow you to deliver more messages per second on the same hardware.

The other two programs included in the Postal suite are rabid, which is a POP client that can poll and consume new messages from a mail server at a specified rate, and bmh, the Black Hole Mailer, which appears like a server for sending email but simply discards any messages sent to it.

Packages of postal are available for Ubuntu Hardy, but not for Fedora or openSUSE. For this article I’ll built the program from source on a 64-bit Fedora 8 machine using Postal version 0.70, following the normal ./configure; make; sudo make install procedure. I had compilation issues and was forced to configure with --disable-gnutls in order to get a clean compilation.

The main parameters of the postal command are the name of a file containing the email addresses that should have mail sent to them, the name of a file containing the addresses that should be used as the sender of the emails (-f), the SMTP server address to send the mails through, and the percentage of message that should be sent using SSL (-s 0). There are also some secondary command-line parameters to control how many threads are started (-t 1), the maximum and minimum email message size to send in kilobytes (-m 10 and -M 0 respectively), and the target number of emails to send per minute (-r 24000).

Shown below is a postal invocation that sends 30 emails per minute using two threads through the mail server on localhost to two addresses on the same machine. Keeping the volume of messages low to start allows you to inspect the mailboxes and see that things are working as expected before you let postal supply a heavier load to the SMTP server by increasing the -r messages per minute parameter. The output on the console is updated every minute, letting you see how much data is sent and, more importantly, how many messages are making it through or generating errors.

# cat user-list-filename
ben@localhost
toast@localhost
# postal -t 2 -r 30  localhost user-list-filename
time,messages,data(K),errors,connections,SSL connections
22:07,2,16,0,4,0
22:08,30,164,0,30,0
22:09,30,149,0,30,0

The above command will generate email messages that are random followed by the string MD5: and the md5 checksum of the message body. This allows you to verify that the mail server is delivering your mail load correctly. With the above invocation messages will have been sent from both ben and toast to both ben and toast on localhost. If you are simply supplying an email load to see how your system holds up, using the same list of accounts as the sender and destination like this should not be an issue. If you wish to change the sender list, provide postal another file name containing the list of sender addresses using the -f command-line option.

rabid

While the postal command is all about sending email messages, the rabid command is all about retrieving them. Some command-line parameters are similar between postal and rabid; -r is used to limit the number of connections per minute so you can fix the rate at which messages are consumed by rabid, -p is the number of rabid processes to create to allow multiple accounts to be checked concurrently, and -s nominates the number of email retrievals that should use SSL. The last two parameters are the address of the POP server and the name of a file containing a list of accounts to check. The last argument is similar to that used by postal, except that each account in the file should be followed by a space and the password to use to retrieve that account’s email. By the way, you can use the file that contains the POP passwords that rabid needs with the postal command, and it will ignore the password field and simply use the email addresses listed at the start of each line.

Shown below is a command that checks the two listed accounts using the POP protocol over only SSL connections. The -u option is meant to tell rabid to use only the user name part of the mail address for each account, but I had issues with authentication when using email addresses like ben@localhost with the -u option. When I explicitly removed @localhost from the lines in the file, authentication proceeded as expected. The -r 5 option limits checking to five connections per minute. The -p 2 argument spawns two processes to allow concurrent email downloads from two accounts. The -s 100 option tells rabid to use SSL for 100% of its connections.

The manner in which ports are specified for postal tools does not follow the normal hostname:port convention. To specify the port you must enclose the host name in brackets and immediately follow this with the port number. The first part of each line in user-list-filename.rabid is identical to user-list-filename; on each line there is a space followed by the password to use for that account to collect email through POP.

# cat user-list-filename.rabid
ben password1
toast password2

# rabid -r 5 -p 2 localhost user-list-filename.rabid
time,messages,data(K),errors,connections,SSL connections,IMAP connections
...
# rabid -r 5 -p 2  -s 100 [localhost]995 user-list-filename.rabid 

Although the use of postal to generate messages and rabid to collect them should give assurance that the message read was identical to the one written, I was getting a lot of messages from rabid informing me of corruption. I am still debugging what caused these mismatches. The setup was a fairly standard Fedora 8 install using sendmail version 8.14.1-4.2.fc8 and uw-imap version 2007b-1.fc8.

...
MD5 mis-match, calculated:bb2e26c3312af5df15615f71ec5fa0bd, expected e42c889641ffdf348ae20d8c8d055dd2!
Account name ben
...

The Black Hole Mailer program is useful if you want to test how well your mail server handles email messages sent to a list of recipients, as you might if you were benchmarking mail list server performance.

Wrap up

If you have a mail server that sees a fair bit of action, you can employ postal to experiment with how your software is configured or to see if adding new hardware gives you the improvement in performance you anticipate. Being able to limit the rate at which email messages are produced and consumed lets you experiment with how other aspects of the system affect the mail server’s performance — for example, you can see whether enabling server-side spam filtering is causing incoming messages to be rejected.