Hoist your applications with petardfs

The petard filesystem is designed to produce only errors — but you can stipulate what conditions generate the errors and what those errors should be. That makes petardfs useful for system and unit testing — for example, making sure that an application gives a sane error message if it fails to open a file, or if there is a read error at byte 5000 of a file.

Petardfs uses Filesystem in Userspace (FUSE) to allow easy setup without requiring a kernel recompile or new kernel modules. In normal configuration you specify a “base filesystem” and give a mountpoint — for example, saying that /home/ben/foo is the base filesystem and mounting the filesystem at /home/ben/petard-foo. Without any other configuration, any files in foo will be available in petard-foo unchanged. Petardfs uses an XML configuration file to tell which files to report errors for and what error code to use. For example, foo.txt can have an EIO error at bytes 34 to 37.

Building and installation of petardfs follows the conventional configure, make, make install procedure. Petardfs builds on the fuselagefs filesystem, which is a C++ wrapper for FUSE and provides handling of base filesystem to mount point mapping. With it, developers can quickly create other FUSE filesystems which work by exposing a base filesystem with a little bit of additional functionality.

You can download both petardfs and fuselagefs from SourceForge.net.

Running over petard

The two most interesting options to petard are –error-definitions (-e), with which you specify the XML file containing error definitions, and –url (-u), to specify the base filesystem. The base filesystem should always be specified as an absolute path. You must also specify the path that the petard filesystem will be mounted at as the final argument. From the above configuration, the base filesystem will be specified as -u /home/ben/foo and the final argument on the command line will be where we want petardfs to be available: /home/ben/petard-foo.

Petardfs command-line options that affect the filesystem’s interaction with FUSE are prefixed with --fuse-. Normally petardfs will run in the background until the FUSE filesystem is unmounted with fusermount -d. The use of the FUSE option –fuse-forground will keep petardfs in the foreground in the terminal until either you issue a fusermount command or press Ctrl-c in the terminal. The –fuse-forground option is mainly interesting if you suspect a bug in petardfs itself. Running it in the foreground allows you to use stderr messages, and petardfs can be directly invoked from the GNU debugger.

The configuration file

You can specify a list of errors for each major filesystem-related function in petardfs. The outer-most element in the configuration file is petardfs-config. This element contains an errors element. Each function can appear as an element under errors. For each function you can specify a list of error conditions.

All file paths are specified relative to the base filesystem. The root of the filesystem paths is also the base filesystem. For example, if the base filesystem is /home/ben/foo and is mounted at /home/ben/petard-foo then the following error configuration will generate an EIO error when /home/ben/petard-foo/file2.txt is read:

<petardfs-config>
    <errors>

      <read>
	<error path="/file2.txt">
	  <n start-offset="4096" end-offset="4196" error-code="&EIO;"/> 
	</error>
      <read>

    <errors>
<petardfs-config>

Two error conditions which are more subtle are the EINTR and EAGAIN errors. These errors both mean that an application should retry the same I/O function call again. For example, reading the file3.txt file in the following configuration will be allowed to succeed but will require the application to retry reading byte 20000 three times before it will be successfully returned.

<error path="/file1.txt">
  <n start-offset="10000" end-offset="10000" error-code="&EINTR;" times="10"/>
</error>
<error path="/file3.txt">
  <n start-offset="20000" end-offset="20000" error-code="&EAGAIN;" times="3"/>
</error>

All error codes in the above examples are XML entities. These are defined in the header of the XML file itself to avoid spreading arbitrary integer error codes throughout the XML file. Petardfs will simply return whatever the numeric error code is that is specified for a particular error condition.

<!DOCTYPE petardfs-config [

<!ENTITY	EPERM		 "1">
<!ENTITY	ENOENT		 "2">
<!ENTITY	ESRCH		 "3">
...

In the current release of petardfs the list of valid functions for which error conditions can be specified are: read, write, fsync, mkdir, symlink, unlink, rmdir, rename, link, chmod, chown, ftruncate, utime, and open.

Running petardfs

You can find a complete XML configuration file in the petardfs distribution tarball at testsuite/sampledata/config-simple-filesystem-simple-error-in-read-file1.xml. The simple-filesystem.tar contains a small filesystem with a collection of files with names file1.txt, file2.txt, and so on. The configuration file defines various errors to occur for those sample files.

Shown below is the mounting of a petardfs and a simple interaction with the error-generating filesystem. In place of the … paths shown toward the top you can substitute the path leading to the file in the petardfs tarball distribution. Notice that directly running diff on the file1.txt in the petard filesystem fails with “Interrupted system call.” However, the cat command will gracefully retry when it gets this error, so using cat to create a copy of the file and then using diff on the copy works fine. The error for file2.txt is a simple EIO error at the 4096th byte. cat gracefully reports the error to stderr and exits.

$ mkdir /tmp/petard
$ cd /tmp/petard
$ cp .../simple-filesystem.tar .
$ cp .../config-simple-filesystem-simple-error-in-read-file1.xml
$ mkdir -p foo petard-foo
$ cd foo
$ tar xvf ../simple-filesystem.tar
$ cd .. 
$ petardfs -e config-simple-filesystem-simple-error-in-read-file1.xml 
   -u `pwd`/foo  petard-foo
$ ls -l petard-foo/
total 512
-rw-rw---- 1 ben ben 50000 2007-03-14 16:15 barB.txt
-rw-rw---- 1 ben ben 50000 2007-03-14 16:15 barC.txt
-rw-rw---- 1 ben ben 50000 2007-03-14 16:15 file1.txt
-rw-rw---- 1 ben ben 50000 2007-03-14 16:15 file2.txt
-rw-rw---- 1 ben ben 50000 2007-03-14 16:15 file3.txt
-rw-rw---- 1 ben ben 50000 2007-03-14 16:15 file4.txt
-rw-rw---- 1 ben ben 50000 2007-03-14 16:15 file5.txt
-rw-rw---- 1 ben ben 50000 2007-03-14 16:15 fooA.txt
-rw-rw---- 1 ben ben 50000 2007-03-14 16:15 simplefile.txt
drwxrwx--- 2 ben ben  4096 2007-03-14 16:16 sub1
drwxrwx--- 2 ben ben  4096 2007-03-14 16:16 sub2

$ diff -Nuar petard-foo/file1.txt foo/file1.txt
diff: petard-foo/file1.txt: Interrupted system call
$ cat petard-foo/file1.txt >|/tmp/file1.txt
$ diff -Nuar /tmp/file1.txt foo/file1.txt

$ cat petard-foo/file2.txt >|/tmp/file2.txt
cat: petard-foo/file2.txt: Input/output error
$ diff -Nuar petard-foo/file2.txt foo/file2.txt
diff: petard-foo/file2.txt: Input/output error

Dejagnu

The petardfs distribution also serves as an example of how to integrate petardfs into an application’s test suite. Petardfs includes a limited test suite that uses DejaGnu to test that petardfs functions as expected. The top Makefile.am includes DejaGnu in AUTOMAKE_OPTIONS. The subdirectory testsuite/petardfs.test contains the beginnings of a TCL/expect DejaGnu test suite for petardfs in petardfs.exp.

The testing performed by petardfs.exp is to ensure that petardfs itself works as expected. However, the commands to test petardfs or to test an application running against petardfs are very similar. For example, both will require running petardfs with various configuration options, and both will need to use a function similar to proc unmount_petardfs {} to unmount a petardfs filesystem again.

The core of the test suite is the run_all function at the end of the petardfs.exp file:

proc run_all {} {

    verbose "Running all existing tests... this will take a long time..."
    run_noerror_petardfs_test_suite
    run_simple_petardfs_test_suite
}

run_all

The run_simple_petardfs_test_suite function mounts a petardfs filesystem with a specific error configuration and runs a collection of specific test cases against it, checking that the results are what was expected. In the petardfs.exp case, the test suite expects to generate specific error messages by running various commands against the petard filesystem. Finally run_simple_petardfs_test_suite unmounts the petard filesystem so that other test functions in the test suite can mount petardfs with a different configuration at the same mount point, and no instances of petardfs are left running after the test suite is complete.

Some utility functions are included in and used by petardfs.exp. The rt_expected{ testname expected cmd } function runs a command (cmd) using a testname for reporting diagnostics or success. The expected string needs to appear in the output of the command for the test to succeed. The verify_output_identical { cmdA cmdB diffflags } function runs two commands and compares the output using diff -Nsuar and some extra flags diffflags passed to diff. Such utility functions make creation of DejaGnu test suite cases for petardfs interaction fairly easy. The utility functions handle application timeout errors and spawn/expect pairs.

proc run_simple_petardfs_test_suite {} {

    global PTT
    global UTBASE
    global SDATA
    global verbose
    global PETARDFS

    set ERRORDESC "$SDATA/config-simple-filesystem-simple-error-in-read-file1.xml"
    set BASEDIR $PTT
    setup_basedir_and_input;

    cd $BASEDIR
    system $PETARDFS -e "$ERRORDESC" -u "$BASEDIR/input" "$BASEDIR/fusefs"
    if { $verbose > 1 } {
	send_user "Mounted petardfs with specified errors:$ERRORDESC...n"
    }

    set testname [rt_version "io-error-at-4096" ]
    rt_expected $testname  "Input/output error" "cat $BASEDIR/fusefs/file2.txt"

    # file1.txt has some EINTR returns
    verify_output_identical 
	"cat $BASEDIR/input/file1.txt" 
	"cat $BASEDIR/fusefs/file1.txt" "-Nuar"
 
    ....

    unmount_petardfs;
}

Conclusion

Petardfs allows you to be mean to your applications and see how gracefully they respond under error conditions. Errors in your project’s error-handling code can be detected and fixed by running your application through a test suite against petardfs. In many error conditions, such as failure to open a file at startup, an application might just have to exit gracefully, but things such as failure to truncate a log file might only call for the application to inform the user of the error and continue. Petardfs can help you test both common errors such as read failures and rarer conditions such as truncate(2) failures.