Author: Peter Seebach
machine; it runs Linux binaries on a NetBSD kernel. Linux emulation
let you run plenty of useful programs that won’t run natively
under NetBSD, such as Sun’s 1.4 Java Runtime Environment and JDK.
Setting up Linux emulation requires a kernel option
(COMPAT_LINUX) in the NetBSD kernel, and some local
files. The kernel option is specified in the default GENERIC configuration;
if you aren’t using that, just make sure you have a line reading “options
COMPAT_LINUX” in your kernel configuration file. As for the
Linux programs, most are dynamically linked and thus need copies
of Linux shared libraries, which are installed under
/emul/linux.
The easiest way to set up the necessary libraries and support
files is to use the packages found in pkgsrc
to install the various binaries correctly. The key package is called
suse_base; there are a number of others. You can install them all
at once by installing the suse_linux package, which gives you all
the most common libraries, including X Window support. As of this
writing, this package is based on SUSE Linux version 7.3. This isn’t
an install of SUSE; it’s just unpacking a handful of shared libraries and
support files.
You may need to mount the /proc filesystem, which gives
filesystem access to a number of system data structures and process-specific
data structures. If you already have it in your fstab, add the “linux”
option to the flags field. If not, a sample line would look like
this:
/proc /proc procfs rw,linux
0 0
You need to create the /proc directory first, with
owner root and mode 755 privileges.
Once you’ve got these steps done, most Linux binaries
should just run out of the box. When you start a Linux executable,
the NetBSD kernel flags it as needing Linux emulation. This changes
the behavior of some system calls to provide behavior compatible
with Linux kernel behavior.
Some packages may give strange error messages in Linux emulation
mode; for instance, StarOffice 7 gives hundreds of repetitions of
“file_image_pagein: Function not implemented” during startup, but
it works fine anyway. Packages such as Sun’s JRE and JDK work
without complaint, as does the Linux build of Netscape. To find
programs that won’t work you have to look a little farther afield.
Video games are more likely to have problems, as are some programs
that use system calls that are harder to emulate; a Linux build of
a program such as WINE, for instance, is very unlikely to work
without effort and may not work at all.
The Linux emulation code is actively maintained by the NetBSD project
staff, so support is improving. Things that were unstable or unreliable
a while back may work better
today. Also, sometimes an apparent problem with emulation is
actually a problem with the program running under emulation; for
instance, until Sun fixed a bug where Sun’s Java runtime would hang
in multithreaded programs, I thought I was seeing an emulation
problem, but it was actually a bug in the Java runtime.
How does it work?
The NetBSD kernel has a number of execsw structures
that describe different types of executable programs. One feature
of the execsw structure is a struct object, which is attached to the process and which
emul
contains various tables used to translate system calls, error codes,
and other such communication between systems. When trying to
execute a binary, the kernel iterates through the available
execsw
structures, testing each to see whether it thinks it can handle
the executable. For instance, the Linux emulator checks the file for an
ELF header that claims to be a Linux binary.
If the kernel finds one that can execute the file it
uses it. If execsw structure has a struct associated with it, then that emulation layer is used
emul
for that process.
Let’s use an example to see how this works. If you try to start
a Linux ELF binary, the NetBSD execsw structure’s
probe routine looks at the file, concludes it can’t handle it, and
rejects it. The exec system call then tries other
structures (perhaps other emulators, or the default “interpreter” structure
used with shell scripts), until the Linux emulation execsw
structure’s probe routine confirms that the one being tested can
run the file.
The corresponding struct emul is then attached to the
process, which is run using the Linux emulator’s code from that
point on. The emulator translates system calls, errno
values, and anything else used in the API between the kernel and
user code.
Filesystem magic
One obvious problem is that, since most programs are dynamically
linked, they need to be able to find the shared C library, which
means you need to have a shared C library loaded. Linux and BSD
systems have very different libraries, so there needs to be an
additional shared library for the Linux executables.
To facilitate loading the “right” versions of various system
files, the kernel intercepts all file lookups from emulated
binaries. They are first looked up in a “shadow root” — a directory
tree in /emul/linux containing files that differ
between Linux and BSD systems. If the file in question is not
there, the kernel looks in the regular filesystem tree. Thus,
libc.so is loaded from the shadow root, but user home
directories are found in the regular filesystem tree. Looking in
the /emul/linux tree to see which files are loaded
there helps a lot in understanding the Linux emulation code.
This system isn’t quite flawless — for instance, symbolic links
don’t use the shadow root — but in practice, it works fine.
Linux and NetBSD systems use somewhat different formats for the
/proc filesystem. NetBSD’s mount_procfs command takes
an optional -o linux option to direct it to use more
Linux-compatible formats and enable a couple of Linux features
normally not present in NetBSD’s /proc filesystem (for instance, a file
called “stat” which holds data similar to, but differently formatted than,
the file called “status” that NetBSD normally uses). Not all programs need this, but many do.
Programs that don’t work
Not every application works in emulation mode. The most common
problem is that the system may lack a shared library that a
program needs. The default install from pkgsrc installs the most
commonly used libraries, but you could end up needing a library
that’s not installed. Getting the replacement library isn’t always
easy. One useful strategy is to install the program on a Linux
box (ideally, the same distro and version
from which you’re getting all your shared
libraries), and execute the ldd command to get a list
of libraries it’s linking with. Once you have such a list, you can
copy the files over.
One thing to watch out for: Some libraries may have multiple
versions that are intended to work on different systems. A rendering
library that uses direct rendering support on a Linux system is
unlikely to work under the very different memory architecture of
a NetBSD system. If you have both hardware and software versions
of a rendering library, the software one is more likely to work,
but it may be quite a bit slower.
Only rarely will a program actually cause the operating system
to crash. More likely is for a program to fail, possibly with a
confusing error message. However, I’ve seen one program (a video
game) actually hang a NetBSD system, so make sure you try a program
a couple of times before you run it on a heavily loaded system
doing other work. Sync your disks and stop any important or
resource-intensive activities before testing a new application for
the first time.
A more subtle kind of error comes from programs whose install
scripts start with #!/bin/sh, but which are actually
bash scripts. This used to be more common than it is
now, but it still happens often enough to be annoying. Merely
having the right /bin/sh in the /emul/linux
“shadow root” isn’t enough; the script may be run
without the Linux emulation flag when spawned from other programs, since it apparently
runs just fine as a standard shell script. One workaround is to
temporarily replace /bin/sh with a link to some version
of bash. Another is to modify the script’s #! line
to refer to something other than /bin/sh, such
as /bin/bash.
When a program isn’t working, another useful technique is to
run it under ktrace to see what output it produces.
This command logs all the system calls the program makes and writes
them to a file called, by default, ktrace.out. You can review the
chain of system calls with kdump. You might see output
like this (taken from the command ls /foo):
4653 ls CALL __stat13(0x80510c0,0xbfbff4e0) 4653 ls NAMI "/foo" 4653 ls RET __stat13 -1 errno 2 No such file or directory
What this tells you is that namei (a kernel
internal function that converts names to inodes) tried to look up
“/foo” and got “No such file or directory.” If a Linux program
aborted and a look through kdump reveals that it
couldn’t find some file, check a Linux system to see whether that
file is part of a standard installation, and if so, copy it over
into the /emul/linux tree.
Linux device drivers won’t work at all under NetBSD. Maybe
someday someone will write code to make them work, but for now,
there’s just no way to use device drivers or other kernel modules
from Linux systems on a BSD system.
Nominal performance
The majority of real-world applications exhibit normal performance
in Linux emulation mode. People who have gotten used to PC emulators
on a Mac, such as VirtualPC, often expect emulation to imply slow
operation. Not so. While it’s difficult to isolate emulation cost
from other OS overhead or differences in the performance of C
libraries, in practice, if a program doesn’t rely heavily on direct
rendering and is usable under Linux, it’ll be just as usable under
Linux emulation on comparable hardware. StarOffice 7 is slick and
responsive. Netscape 7 runs as reliably as a native browser, and
runs Flash animations and the like quite nicely. You probably
wouldn’t want to use Linux emulation code for a high-performance
project, but for getting access to a specific application or module,
it’s pretty useful.
Linux executables can be run in emulation mode on Alpha, ARM,
i386, m68k, and PowerPC systems. Both a.out and ELF executables
are supported. If you use the pkgsrc installer, it’s pretty easy
to set up all the important files, and the kernel options you need
are included in the GENERIC kernel, so it Linux emulation should run
out of the box. You do need root privileges to configure a system
to run Linux binaries, but once the emulator and support files
are set up, the applications work fine for ordinary users.
Linux emulation code lets you run most standard applications without
a great deal of effort, and could help provide a standardized
environment for users who need to run different OSs, or use
proprietary code. It won’t do everything, but it does enough to
be a practical and useful tool.
Peter Seebach has been running Linux binaries on NetBSD for a few
years, and no one has arrested him yet, so it’s probably allowed.
Category:
- BSD
