Author: Dan Farmer and Wietse Venema
One of the simplest things to understand and use in an investigation is MACtimes. MACtimes are not the McDonald’s version of a time zone. They are simply a shorthand way to refer to the three time attributes — mtime, atime, and ctime — that are attached to any file or directory in Unix, Windows, and other file systems.
(Microsoft’s file systems have four similar times: ChangeTime, CreationTime, LastAccessTime, and LastWriteTime. Linux also has the dtime attribute, which is set when a file or directory has been deleted. In particular, this doesn’t affect files in the visible file system, only deleted files.)
The atime attribute refers to the last time the file or directory was accessed. The mtime attribute, in contrast, changes when a file’s contents are modified. The ctime attribute keeps track of when the contents or the meta-information about the file has changed: the owner, the group, the file permissions, and so on. The ctime attribute may also be used as an approximation of when a file was deleted.
For all of these attributes, however, it is crucial to note the word last. MACtimes only keep track of the last time a file was disturbed; once the file has been changed, historical MACtime data is impossible to uncover — at least, most of the time. Journaling file systems can reveal recent history that would otherwise be lost in the system, as we’ll see in a moment.
On Unix systems, these times may be viewed with the humble ls command (see the ls man page for more details). For NT file systems, various third-party tools are available. In real situations, however, it’s often easier to use the Coroner’s Toolkit’s mactime tool or to simply rely on the lstat() system call (which mactime itself uses), as evidenced by this simple Perl code fragment:
($dev, $inode, $mode, $nlink, $uid, $gid, $rdev,
$size, $atime, $mtime, $ctime, $blksize, $blocks) = lstat($filename);
print "$filename (MAC): $mtime,$atime,$ctimen";
MACtimes returned by the Perl lstat() function call are displayed as the number of seconds since January 1, 1970, 00:00:00 UTC. NTFS keeps file times in 100-nanosecond chunks since January 1, 1601; thankfully, Perl converts this for you.
If you’ve never looked at MACtimes before, it can be surprising how useful they can be. The listing below shows a bit of what the security team found in one investigation.
Jul 19 2001
time
size MAC permissions owner file name
----
---- --- ---------- ----- ---------
16:47:47 655360 m..
-rw-r--r-- root /usr/man/.s/sshdlinux.tar
16:48:13 655360 ..c
-rw-r--r-- root /usr/man/.s/sshdlinux.tar
16:48:16 395 ..c
-rwxrw-r-- 2002 /usr/man/.s/ssh.sh
880 ..c -rw-r--r-- 2002
/usr/man/.s/ssh_config
537 ..c -rw------- 2002
/usr/man/.s/ssh_host_key
341 ..c -rw-r--r-- 2002
/usr/man/.s/ssh_host_key.pub
16:48:20 1024 m.c
drwxr-xr-x root /usr/man/.s
16:51:31 1024 m.c
drwxr-xr-x root /home
1422 m.c -rw-r--r-- sue
/home/sue/.Xdefaults
24 m.c -rw-r--r-- sue
/home/sue/.bash_logout
230 m.c -rw-r--r-- sue
/home/sue/.bash_profile
124 m.c -rw-r--r-- sue
/home/sue/.bashrc
16:57:57 1024 m.c
drwx------ sue /home/sue
9 m.c -rw------- sue
/home/sue/.bash_history
It’s no coincidence that this output looks very similar to Unix’s ls -l output. The big difference here is the inclusion of the “MAC” column. This shows which of the three file time attributes (mtime, atime, and ctime) correspond to the dates and times in the first column.
This output shows that on July 19, just before 5:00 p.m., a user with root privileges created and unpacked a tar file (a popular Unix file archive format) with a file name suspiciously looking like it contained a replacement for ssh server. The file itself was in an even more suspicious location (he or she might as well have named the directory “Kilroy was here”). Finally, soon after the file was created, user “sue” logged off.
You might have noticed that there were no atimes listed in the MACtime output. This is because the “helpful” administrator who copied all the files for safekeeping also destroyed a wide variety of evidence at the same moment. Backing up files before gathering other evidence is a very poor idea, because it runs against the order of volatility. The OOV demands that more ephemeral data be harvested before more stable data. In this case, reading (or copying) the files changed their atime attributes to the time the file was read. Directories also have atimes; accessing a directory’s contents updates its atime as well. Note, too, that running a program changes the atime of its executable file, because the contents of the executable file must be read before execution.
Many systems allow root to disable atime updates, which is something to remember when examining time stamp information. When investigating a system, turning off atimes can also be useful to avoid destroying atime information when it is not possible to mount a disk as read-only.
Journaling file systems and MACtimes
Journaling file systems have been a
standard feature of enterprise-class systems for a long time
and have more recently become available for popular systems
such as Linux and Microsoft Windows. Examples are Ext3fs, JFS,
NTFS, Reiserfs, XFS, Solaris UFS, and others. With a journaling
file system, some or all disk updates are first written to a
journal file before they are committed to the file system
itself. Although at first this seems like extra
work, it can significantly improve the recovery from a system
crash. Depending on what optimizations the file system is
allowed to make, journaling does not need to cause loss of
performance.
Why does the world need journaling file
systems? Every nontrivial file system operation, such as
creating or appending a file, results in a sequence of disk
updates that affect both file data (that is, the contents) and
file metadata (such as the location of file contents, and what
files belong to a directory). When such a sequence of updates
is interrupted due to a system crash, non-journaling file
systems — such as FFS, Ext2fs, and Windows FAT — can leave their file metadata in
an inconsistent state. The recovery process involves programs
such as fsck and scandisk and can take several hours with large file
systems. By comparison, recovery with a journaling file system
is almost instantaneous: it can be as simple as replaying the
“good” portion of the journal to the file system
and discarding the rest.
Journaling file systems differ widely in
the way they manage their information, but conceptually they
are very easy to understand. There are two major flavors: those
that journal metadata only, and those that journal both data
and metadata. In this section, we look only at
MACtimes — that is, metadata — although we are aware
that journaled file contents have great forensic potential,
too.
From a forensics point of view, the
journal is a time series of MACtime and other file information.
It is literally a time machine by itself. Whereas normal
MACtimes allow us to see only the last read/write operation or
status change of a file, journaled MACtimes allow us to see
repeated access to the same file. This listing shows an example
of repeated access that was recovered more than 24 hours after
the fact from an Ext3fs file system. For the sake of clarity,
dynamically linked libraries were omitted. Files with the same
time stamp are sorted alphabetically.
time
size MAC permissions owner file name 19:30:00 541096 .a. -rwxr-xr-x
root /bin/bash 19:30:00 26152 .a. -rwxr-xr-x
root /bin/date 19:30:00 4 .a.
lrwxrwxrwx root /bin/sh -> bash 19:30:00 550 .a. -rw-r--r--
root /etc/group 19:30:00 1267 .a. -rw-r--r--
root /etc/localtime 19:30:00 117 .a. -rw-r--r--
root /etc/mtab 19:30:00 274 .a. -rwxr-xr-x
root /usr/lib/sa/sa1 19:30:00 19880 .a. -rwxr-xr-x
root /usr/lib/sa/sadc 19:30:00 29238 m.c -rw-------
root /var/log/cron 19:30:00 114453 mac -rw-r--r--
root /var/log/sa/sa19 19:40:00 541096 .a. -rwxr-xr-x
root /bin/bash 19:40:00 26152 .a. -rwxr-xr-x
root /bin/date 19:40:00 4 .a.
lrwxrwxrwx root /bin/sh -> bash 19:40:00 550 .a. -rw-r--r--
root /etc/group 19:40:00 1267 .a. -rw-r--r--
root /etc/localtime 19:40:00 117 .a. -rw-r--r--
root /etc/mtab 19:40:00 274 .a. -rwxr-xr-x
root /usr/lib/sa/sa1 19:40:00 19880 .a. -rwxr-xr-x
root /usr/lib/sa/sadc 19:40:00 29310 m.c -rw-------
root /var/log/cron 19:40:00 115421 mac -rw-r--r--
root /var/log/sa/sa19
Regular system activity can act as a
heartbeat, showing up in logs and in other locations such as
the file system journal. Here it turns out that cron, the
scheduler for unattended command execution, is running a
maintenance program every ten minutes. Besides the information
that we have learned to expect from normal
MACtimes, the MACtimes from the journal also reveal how log
files grow over time, as shown by the file sizes of /var/log/cron
and /var/log/sa/sa19.
Rather than trying to cover all the major
journaling file system players, we’ll take a brief look
at the Ext3fs implementation. Ext3fs is particularly easy to
use because of its compatibility with its predecessor, Ext2fs,
and it has become the default file system in many Linux
distributions. Although Ext3fs stores the journal in a regular
file, that file is usually not referenced by any directory, and
therefore it cannot be accessed by name. The Linux tune2fs command
reveals where the journal is kept:
linux# tune2fs -l /dev/hda1 | grep -i journalFilesystem features: has_journal
filetype needs_recovery sparse_super
Journal UUID:
<none>
Journal inode:
8
Journal device:
0x0000
This output shows, among other things, that the journal is stored as a regular file with inode number 8. What it does not
show is that the journal has a fixed size of 32 Mbytes. As part
of an investigation, it is therefore worthwhile to save the
contents of the journal with the Coroner’s
Toolkit’s icat command early, before the data is
overwritten with newer information. However, be sure to save
this new file to a different file system; otherwise the journal
may end up destroying itself with its own contents.
linux# icat /dev/hda1 8 >journalfile
The Linux debugfs file system debugger may be used to examine the
file system journal in some detail. The following command dumps
recent access times for the /etc/passwd file:
linux# debugfs -R 'logdump -c -i /etc/passwd'
/dev/hda1 | grep atime atime: 0x4124b4b8 -- Thu Aug[. . .]
19 07:10:00 2004
atime: 0x4124b5e4 -- Thu Aug
19 07:15:00 2004
atime: 0x4124b710 -- Thu Aug
19 07:20:00 2004
To examine a saved journal file, we would
specify “-f journalfile” on the logdump command line.
The standard debugfs command
is sufficient if we want to look at only one file at a time,
and if we already know what file we are looking for. However,
to produce multi-file reports such as the one above,
we had to use a modified version that allows us to see all the MACtime information in the journal. This
software is available via the book’s Web site.
As with all tools that are used for
unintended purposes, debugfs can produce
unexpected results at times. The version that we used (1.35)
did not always recognize where the journal terminates, and thus
it would misinterpret the remainder of the journal file. So you
need to use some judgment when interpreting the results.
The amount of MACtime history that can be
recovered from a file system journal depends on the type and
amount of activity in the file system, as well as file system
implementation details. With file systems such as Ext3fs that
can journal both data and metadata, the amount of recoverable
MACtimes can be disappointingly small. On the other hand,
systems with little activity can have records that go back more
than an entire day. In such cases, reading a file system
journal can be like watching a tree grow, one ring at a time.
Category:
- Security
