By Steven H. VanderLeest, Software Engineering Technical Lead at The Boeing Company
Introduction
From the early days of Linux, I was a fan of this innovative, open-source Operating System (OS). I appreciated it as a hobbyist, helping me run Linux at home. I appreciated it as an educator, helping my computer engineering students walk with Linux through OS concepts. However, as a professional working in the safety-critical domain of aerospace, I wondered: could Linux fly?
My Pre-flight Taxi with Linux
My journey with Linux had its roots in the 1980s before Linus Torvalds introduced his new OS to the world in 1991. During my undergraduate degree in the 1980s, my engineering program had some labs equipped with the relatively recent IBM Personal Computer (PC). The machines were amazing, but my ability to command their power was somewhat limited by the OS, which was the Microsoft Disk Operating System (MS-DOS). When I reached my third year, I gained access to a Sun Workstation running SunOS, a variant of Unix. I quickly learned to appreciate the rich menagerie of shell commands, the power of combining them with redirection such as pipes, and the aesthetics of the fledgling X-Windows GUI.
I first heard about Linux in graduate school in the early 1990s at the University of Illinois at Urbana-Champaign. My doctoral thesis was on Input/Output (I/O) performance, especially on multiprocessor systems. My research analyzed and quantified I/O performance on OSs such as SunOS, SGI IRIX, DEC OSF/1, HP-UX, and Linux. One key finding of my research was that I/O performance could be impacted by the interference caused by unrelated transactions contending for shared resources within a multi-processor system. The magnitude of the impact was heavily dependent not only on the computing hardware architecture but also on the architecture of the OS. Interference could even occur on a uni-processor where independent processes had I/O tasks clustered in time.
As an educator, I applied Linux in my teaching. After finishing my Ph.D., I returned to my alma mater, Calvin College (now University), to take a position as a professor of engineering, teaching computer engineering topics. Linux provided a rich learning environment where my students could look under the hood while learning about operating systems. The transparency of open-source code made an ideal environment for learning and innovation. I also wanted to share my love for working at the interface between computer hardware and software. Studying the Linux kernel provided key insights into how the OS manages the hardware on behalf of applications. The overall system’s performance will improve if the OS is reasonably tuned to take advantage of the hardware architecture.
As a hobbyist, I used Linux at home. I set it up on any extra desktop or laptop I could get my hands on. The whole family got involved when I set up MythTV, an open-source streaming media system, and installed it on a spare Linux desktop system along with an expansion card to capture and record live television. We were asynchronously watching programs and never missing an episode well before any of our friends or neighbors followed suit with ReplayTV or TiVo.
As an engineering professional, I found opportunities to bolster my work with Linux. The challenge was that my employers often required MS Windows as the standard a bureaucratic IT department imposed. Nevertheless, I discovered ways to use Linux by dual-booting or a LiveCD approach and eventually run Linux in a virtual machine using hypervisors like VirtualBox. Like its Unix forebears, Linux was much more stable and reliable than Windows. Even if an application program went astray, I got a segmentation fault warning at most, and the other processes continued. Windows was prone to the Blue Screen of Death, bringing the system to a halt much too often. While it might be distressing to lose your work when this happened, losing a few minutes of labor (or hours if you didn’t save often) was a minor albeit annoying inconvenience. I couldn’t expect higher reliability since that wasn’t a use case for office desktop systems. I quickly realized that Windows doesn’t apply to safety-critical systems.
I also would not expect an operating system designed for an office desktop/laptop to work for embedded systems where the available main memory and secondary storage are limited. Embedded computing platforms are all around us but hidden inside our vehicles, more sophisticated consumer electronics, and smart devices. Windows might not work in these use cases, but Linux could! I started using Linux on embedded development boards when chip manufacturers such as Freescale (later NXP), Intel, Texas Instruments, and others began providing a Linux Board Support Package. The chip makers found this approach was the most effective way to get developers up and running quickly on their new hardware.
Taking Flight with Linux
Within safety-critical domains such as aerospace, Linux provides the foundation for multiple software development environments that run on desktops and laptops. As we move toward distributed development, Linux is a ubiquitous cloud guest OS.
For embedded, safety-critical applications, Linux is less common than a Real-Time Operating System (RTOS). However, a group of Linux developers has been slowly improving real-time performance since the 1990s. Attention coalesced into the PREEMPT-RT patch since 2004, with key parts of the patch making their way to the mainline kernel code. Today, almost all PREEMPT-RT functionality is mainlined but must be enabled through kernel configuration parameters. As for the safety-critical need, in the early 2010s, several research groups examined Linux as a foundation for an Integrated Modular Avionics (IMA) system. I led one of these efforts as the Principal Investigator for a Small Business Innovation Research (SBIR) contract with the US Defense Advanced Research Projects Agency (DARPA). We developed a proof-of-concept safety-critical system that combined the Xen hypervisor with Linux as a guest OS, to provide ARINC 653 partitioning, a key standard related to IMA.
Over the past decade, multiple private endeavors have applied Linux in aeronautical and astronautical computing systems, even platforms with modest safety criticality, though only a few of these efforts have been publicized. Demonstrating that software is reliable enough for flight is ambitious. I work for Boeing, one of the aerospace companies tackling that challenge. The next section provides an overview of the four key characteristics necessary to put aircraft using Linux into the air.
Developing Software for Aerospace is Challenging
For use in avionics (an electronic computing platform used on an aircraft), the software must be fast, deterministic, embedded, and assured.
Fast
For use in avionics, Linux must be fast. The Linux developer community is already heavily focused on speed, constantly innovating kernel performance improvements.
The aerospace industry can largely leverage the Linux community effort toward high performance. There may be a few specialized devices where drivers must be further optimized. However, those devices will almost always follow the existing design patterns and take advantage of community innovations, such as io_uring. Another example of an area that might need more attention is boot time. For aerospace, certain fault-tolerance techniques require a fast boot-up (or in-air re-boot) time. In these cases, the system must be operational in only a few seconds or even less.
Deterministic
For use in avionics, Linux must be deterministic. Remember the action thriller series 24? Jack Bauer (played by Kiefer Sutherland) would introduce the series with a voice-over claiming “events occur in real-time”. The audience understood that we were watching as if it were airing live. This commonly understood definition of real-time is not quite the same idea as a real-time computing system. For an RTOS, real-time means that the response to critical events will occur within a deterministic amount of time, even in the worst case. Most computing systems- hardware and software- are tuned to optimize the average response time. Most users and actions enjoy a rapid response, but sometimes at the expense of a slow response for certain users or certain actions. A deterministic system is not necessarily fast — it simply means that we can bound, with confidence, the maximum for critical response times. We want a guaranteed maximum response time in a real-time system, even in the worst case. If we were grading responses like students, we don’t care if the best score was an A+ or the average score was a C. We care that the worst score is still a passing grade in real-time systems. Let’s say the system must always respond within 50 milliseconds, or something bad happens. Over a series of tests, perhaps you find that the fastest response is 12 milliseconds, the average is 27 milliseconds, and the worst is 42 milliseconds. For determinism, we only care that the worst response is still under the requirement (in this example, it appears to be meeting our needs).
The aerospace industry can leverage the Linux community’s effort toward determinism. The PREEMPT_RT patches developed over the last 20 years have largely been mainlined, but must still be configured to enable them. Deterministic boot time has received less attention than deterministic response time, but both are important for aerospace applications.
Embedded
For use in avionics, Linux must be embedded. Embedded use cases are constrained with limited size, weight, and power. The most widely deployed embedded instance of Linux is probably the Android OS, used on the largest number of smartphones around the globe today. The vast majority of the billions of embedded devices that make our digital world run smoothly are not this visible — they are under the hood in your car, behind the panel of your home thermostat, and in many other behind-the-scenes locations.
Many industries, including the aerospace industry, continue to turn to Linux for embedded systems. Chip manufacturers continue to support Linux, often the first OS for which they provide starter software development kits. Developers from across the open-source community continue to develop drivers for new devices.
Assured
Regulatory agencies often oversee safety-critical systems to ensure the software is correct to a high confidence level. Because public safety is at stake, the agencies generally have the authority to enforce standards before a product can be released. For use in avionics, Linux must be assured. For avionics software in civilian aircraft, the authority to approve flight certification is specific to a geographic region. For example, in the United States, it is the Federal Aviation Administration (FAA); in most of Europe, it is the European Union Aviation Safety Agency (EASA).
The details of safety standards vary across industries such as nuclear, automotive, medical, aeronautical, rail, and others. However, the same basic concepts are found in all of them, such as expert peer review or formal means of verification and validation to show the software is suited to purpose. Most have two aspects: ensuring the software is reliable (it does the things we want) and safe (it does not do things we do not want).
A key standard for avionics software is DO-178C, which describes software development life cycle processes and objectives that must be met. DO-178C defines five software levels. The lowest is level E, where a software bug has no impact on the safety of the crew or passengers. An example might be the passenger entertainment system. The highest is level A, where a software bug could have catastrophic results. An example might be the flight control software that responds to pilot commands.
The aerospace industry can leverage much less from the Linux community regarding assurance than the other criteria stated earlier. On the one hand, Linux has been extensively field-tested, so it has a strong product history. Due to the crowd-sourcing nature of open source, Linux likely has more expert peer reviews than any other existing software. Assurance of Linux also benefits from the reasonably large number of tests available within several test frameworks. On the other hand, Linux was not designed expressly for aerospace, nor even for safety-critical use cases in general. The design has been much more iterative and ad-hoc, making it more challenging to demonstrate the correct design to software safety regulatory authorities.
Conclusion
Linux is already being used in flight-certified systems at level D. Aerospace companies like Boeing are now poised to use Linux more broadly and at higher levels of assurance, with groups like ELISA leading the effort. ELISA is the Enabling Linux In Safety Applications project under the Linux Foundation. Its mission is to make it easier for companies to build and certify Linux-based safety-critical applications. ELISA recently formed a new working group focused on Aerospace, which will tackle some of the challenges outlined above. We are just getting this group started and welcome new members!
I have crawled, walked, and run with Linux. Now it is time to fly!
